pandas - Better way to compare all items in a dataframe and replace similar items with fuzzy matching python -


i'm wondering if there's better way compare items in dataframe column each other , replace items if have high fuzzy set matching score. ended using combinations, feeling memory intensive , inefficient. code below.

to clarify: central question here not fuzzy matching aspect, aspect of comparing items in list each other , replacing items match.

newl = list(true_df2.name.unique())

def remove_duplicate_names(newl, name, origdf, namesave):     """     function removes duplicate names. replaces longer names shorter names     takes in (1) newl: list of unique names, generic words have been stripped out.     (2)name: name of dataframe column     (3)origdf: original dataframe being rewritten     (4)namesave: name of saved matchedwords file: e.g, 'save1'. created (4) because file      takes long time run.      returns dataframe       """     if isinstance(newl, pd.dataframe):         newl = list(newl[name].unique())      if isinstance(newl, list):           cnl = list(combinations(newl, 2))         matchword = []         in cnl:             fp = fuzz.partial_ratio(i[0], i[1])             if len(i[0]) > 3 , len(i[1]) > 3:                  if not i[0] == i[1]:                     #if i[0] or i[1] == 'york university':                     #    continue                      #i can edit these conditions make matches more or less strict                     #higher values mean more strict                     #using more criteria 'and' means more strict                     if fp >= 98:                         shortstr = min(i, key=len)                         longstr = max(i,key=len)                         matchword.append((shortstr, longstr))         pair in matchword:         #replace in each longstring spot, shorter string         print 'pair', pair         print origdf[name][origdf[name].str.contains(pair[1])]         #origdf[name][origdf[name].str.contains(pair[1])] = pair[0].strip()         origdf.ix[origdf[name].str.contains(pair[1]), 'name'] = pair[0]     return origdf


Comments

Post a Comment

Popular posts from this blog

android - Get AccessToken using signpost OAuth without opening a browser (Two legged Oauth) -

i using signpost oauth access data magento server. have read various tutorials on same , reach point open browser user can enter credentials. however, per requirement have automate part. hence, user should not browser page. have set-up done on server side ( magento ), hit url , returned call page. same through program in android. i have tried below, consumer = new commonshttpoauthconsumer(key, secret); provider = new commonshttpoauthprovider(oauth_init_url,access_token_url, authorize_url); try { provider.retrieverequesttoken(consumer, oauth.out_of_band); log.d("tokens" , consumer.gettoken() + " -- " + consumer.gettokensecret()); } and request tokens. dont know how bypass next step. tried directly accessing accesstoken (stupid of me) provider.retrieveaccesstoken(consumer, "mycallback://callback"); no luck, ends in oauth.signpost.exception.oauthnotauthorizedexception: authorization failed (server replied 401). ...
Read more

org.mockito.exceptions.misusing.InvalidUseOfMatchersException: mockito -

getting error org.mockito.exceptions.misusing.invaliduseofmatchersexception: invalid use of argument matchers! 1 matchers expected, 2 recorded. exception may occur if matchers combined raw values: //incorrect: somemethod(anyobject(), "raw string"); when using matchers, arguments have provided matchers. example: //correct: somemethod(anyobject(), eq("string matcher")); at when( getprogramservice .callservice(any(getprogramtype.class))) .thenreturn(jaxbresponse); please explain error , possible resolution the code posted looks fine; don't see problem it. code before or above causing exception. key here: invalid use of argument matchers! 1 matchers expected, 2 recorded. any , used above, doesn't return "special kind of null" or "special kind of getprogramtype"; don't exist. instead, returns null , side effect puts matcher on stack. whe...
Read more

google shop client API returns 400 bad request error while adding an item -

i'm working on solution add items in google shopping market java application. following tutorial real mess because of missing classes of laste release, after setting 1.5-beta maven managed make base example typecheck correctly classes. still can make simple task of adding 1 item. instead of posting whole code i'm posting part creates product feed. private product createproduct(string id) { product product = new product(); product.title = "red wool sweater"; product.content = new content( "text", "comfortable , soft, sweater keep warm on " + "cold winter nights. red , blue stripes."); product.appcontrol = new appcontrol(); product.appcontrol.addrequireddestination("productads"); product.externalid = id; product.lang = "it"; product.country = "it"; product.condition = "nuovo"; product.price = new price("eu...
Read more