hadoop - Unable to use bucket map join in hive 0.12 -


i trying out hive optimization features , encountered such problem: cannot use bucket map join in hive 0.12. after setting tried below, only 1 hashtable file generated , join turn out map join.

i have 2 tables both in rcfile format , both bucktized 10 bucket, created , populated follows(origin data generated tpc-h):

hive> create table lsm (l_orderkey int, l_partkey int, l_suppkey int, l_linenumber int, l_quantity double, l_extendedprice double, l_discount double, l_tax double, l_returnflag string, l_linestatus string, l_shipdate string, l_commitdate string, l_receiptdate string, l_shipstruct string, l_shipmode string, l_comment string) clustered (l_orderkey) 10 buckets stored rcfile; hive> create table osm (o_orderkey int, o_custkey int) clustered (o_orderkey) 10 buckets stored rcfile; hive> set hive.enforce.bucketing=true; hive> insert overwrite table lsm select * orili; hive> insert overwrite table osm select o_orderkey, o_custkey orior;

and can query both table’s data normally, , lsm 790mb, osm 11mb, both 10 bucket files, want try bucket map join:

hive> set hive.auto.convert.join=true;  hive> set hive.optimize.bucketmapjoin=true; hive> set hive.enforce.bucketmapjoin=true; hive> set hive.auto.convert.join.noconditionaltask=true; hive> set hive.auto.convert.join.noconditionaltask.size=1000000000000000; hive> set hive.input.format=org.apache.hadoop.hive.ql.io.bucketizedhiveinputformat;

and query follows:

hive> select /*+ mapjoin(osm) */ osm.o_orderkey, lsm.* osm join lsm on osm.o_orderkey = lsm.l_orderkey;

this query generate 1 hashtable of osm , fall map join, confused it. have hint set enable bucket map join feature, or there problems in query ?

short version:

set hive> set hive.ignore.mapjoin.hint=false; make bucket map join work expected. means 10 small tables's bucket files build hash table , hash join corresponding big file's buckets.

a longer version:

dive hive 0.12 code , find hive.ignore.mapjoin.hint in hiveconf.java , set true default, means /*+ mapjoin */ hint ignored deliberately. since there 2 phase of optimization in hive, logical optimization , physical optimization, both rule based optimizations.

  • logical optimization

    in logical optimization, mapjoin optimization followed bucketmapjoin optimization, bucketmapjoin optimization take mapjoin operator tree input , convert bucketmapjoin, hinted query first transformed mapjoin , bucketmapjoin. therefore, hint disabled logical optimisation nothing join optimisation on join tree.

  • physical optimization

    in physical optimisation, hive.auto.convert.join tested , mapjoinresolver used , convert reduce join mapjoin. no further bucketmapjoin optimization rules in phase. that's why mapjoin in question.


Comments

Post a Comment

Popular posts from this blog

android - Get AccessToken using signpost OAuth without opening a browser (Two legged Oauth) -

i using signpost oauth access data magento server. have read various tutorials on same , reach point open browser user can enter credentials. however, per requirement have automate part. hence, user should not browser page. have set-up done on server side ( magento ), hit url , returned call page. same through program in android. i have tried below, consumer = new commonshttpoauthconsumer(key, secret); provider = new commonshttpoauthprovider(oauth_init_url,access_token_url, authorize_url); try { provider.retrieverequesttoken(consumer, oauth.out_of_band); log.d("tokens" , consumer.gettoken() + " -- " + consumer.gettokensecret()); } and request tokens. dont know how bypass next step. tried directly accessing accesstoken (stupid of me) provider.retrieveaccesstoken(consumer, "mycallback://callback"); no luck, ends in oauth.signpost.exception.oauthnotauthorizedexception: authorization failed (server replied 401). ...
Read more

org.mockito.exceptions.misusing.InvalidUseOfMatchersException: mockito -

getting error org.mockito.exceptions.misusing.invaliduseofmatchersexception: invalid use of argument matchers! 1 matchers expected, 2 recorded. exception may occur if matchers combined raw values: //incorrect: somemethod(anyobject(), "raw string"); when using matchers, arguments have provided matchers. example: //correct: somemethod(anyobject(), eq("string matcher")); at when( getprogramservice .callservice(any(getprogramtype.class))) .thenreturn(jaxbresponse); please explain error , possible resolution the code posted looks fine; don't see problem it. code before or above causing exception. key here: invalid use of argument matchers! 1 matchers expected, 2 recorded. any , used above, doesn't return "special kind of null" or "special kind of getprogramtype"; don't exist. instead, returns null , side effect puts matcher on stack. whe...
Read more

google shop client API returns 400 bad request error while adding an item -

i'm working on solution add items in google shopping market java application. following tutorial real mess because of missing classes of laste release, after setting 1.5-beta maven managed make base example typecheck correctly classes. still can make simple task of adding 1 item. instead of posting whole code i'm posting part creates product feed. private product createproduct(string id) { product product = new product(); product.title = "red wool sweater"; product.content = new content( "text", "comfortable , soft, sweater keep warm on " + "cold winter nights. red , blue stripes."); product.appcontrol = new appcontrol(); product.appcontrol.addrequireddestination("productads"); product.externalid = id; product.lang = "it"; product.country = "it"; product.condition = "nuovo"; product.price = new price("eu...
Read more