r - How to set up cluster slave nodes (on Windows) -


i need run thousands* of models on 15 machines (each of 4 cores), windows. started learn parallel, snow , snowfall packages , read bunch of intro's, focus on setup of master. there little information on how set worker (slave) nodes on windows. information contradictory: some sock cluster practically easiest way go, others claim sock cluster setup complicated on windows (sshd setup) , best way go mpi.

so, easiest way install slave nodes on windows? mpi, pvm, sock or nws? my, possibly naive ideas (listed priority):

  1. to use 4 cores on slave nodes (required).
  2. ideally, need r packages , slave r script or r function listen on port , wait tasks master.
  3. ideally, nodes can added/removed dynamically cluster.
  4. ideally, slaves connect master - wouldn't have list slaves ip's in configuration of master.

only 1 100% required, 2-4 "would good". naive request?

i sorry have not been able figure out available docs , tutorials. grateful if point me out right source.

* note each of thousands of models take @ least 7 minutes, there won't big communication overhead.

it's shame how these apis (like parallel/snow/snowfall) complex work with, lots of docs not need... have found api simple , goes straight ideas sketched!! redis , doredis r package (as recommended here). very simple tutorial present! modified bit , got this:

the workers need r, doredis package , script:

require(doredis)     redisworker('jobs', '10.0.0.7') # ip of server

the master needs redis server running (installed the experimental windows binaries windows), , r code:

require(doredis) registerdoredis('jobs') foreach(j=1:10,.combine=sum,.multicombine=true) %dopar%     ... # whatever need run removequeue('jobs')

adding/removing workers dynamic, no need specify ips @ master, automatic "load balanancing", simple , no need tons of docs! solution fulfills requirements , more - stated in ?registerdoredis:

the doredis parallel end tolerates faults among worker processes , automatically resubmits failed tasks.

i don't know how complex using parallel/snow/snowfall socks/mpi/pvm/nws, if possible @ all, guess complex...

the disadvantages of using redis found:


Comments

Post a Comment

Popular posts from this blog

android - Get AccessToken using signpost OAuth without opening a browser (Two legged Oauth) -

i using signpost oauth access data magento server. have read various tutorials on same , reach point open browser user can enter credentials. however, per requirement have automate part. hence, user should not browser page. have set-up done on server side ( magento ), hit url , returned call page. same through program in android. i have tried below, consumer = new commonshttpoauthconsumer(key, secret); provider = new commonshttpoauthprovider(oauth_init_url,access_token_url, authorize_url); try { provider.retrieverequesttoken(consumer, oauth.out_of_band); log.d("tokens" , consumer.gettoken() + " -- " + consumer.gettokensecret()); } and request tokens. dont know how bypass next step. tried directly accessing accesstoken (stupid of me) provider.retrieveaccesstoken(consumer, "mycallback://callback"); no luck, ends in oauth.signpost.exception.oauthnotauthorizedexception: authorization failed (server replied 401). ...
Read more

org.mockito.exceptions.misusing.InvalidUseOfMatchersException: mockito -

getting error org.mockito.exceptions.misusing.invaliduseofmatchersexception: invalid use of argument matchers! 1 matchers expected, 2 recorded. exception may occur if matchers combined raw values: //incorrect: somemethod(anyobject(), "raw string"); when using matchers, arguments have provided matchers. example: //correct: somemethod(anyobject(), eq("string matcher")); at when( getprogramservice .callservice(any(getprogramtype.class))) .thenreturn(jaxbresponse); please explain error , possible resolution the code posted looks fine; don't see problem it. code before or above causing exception. key here: invalid use of argument matchers! 1 matchers expected, 2 recorded. any , used above, doesn't return "special kind of null" or "special kind of getprogramtype"; don't exist. instead, returns null , side effect puts matcher on stack. whe...
Read more

google shop client API returns 400 bad request error while adding an item -

i'm working on solution add items in google shopping market java application. following tutorial real mess because of missing classes of laste release, after setting 1.5-beta maven managed make base example typecheck correctly classes. still can make simple task of adding 1 item. instead of posting whole code i'm posting part creates product feed. private product createproduct(string id) { product product = new product(); product.title = "red wool sweater"; product.content = new content( "text", "comfortable , soft, sweater keep warm on " + "cold winter nights. red , blue stripes."); product.appcontrol = new appcontrol(); product.appcontrol.addrequireddestination("productads"); product.externalid = id; product.lang = "it"; product.country = "it"; product.condition = "nuovo"; product.price = new price("eu...
Read more