php - Get specific content with CURL on all links in a page (like a spider) -


i'm coding little app start url , on links in specific page. next, go on links , scrape contents show specific content (numbers 10 or more char). code retrieve blank page, wrong?

//i  $url = 'http://xxx.xxx'; $str = file_get_contents($url); $original_file = file_get_contents($url); $stripped_file = strip_tags($original_file, "<a>"); preg_match_all("/<a(?:[^>]*)href=\"([^\"]*)\"(?:[^>]*)>(?:[^<]*)<\/a>/is", $stripped_file, $matches); $links = $matches[1]; //print_r($links);  //f  //f $count = count($links); for($i=0;$i<=$count;$i++) {   //i   $curl_handle=curl_init();   curl_setopt($curl_handle, curlopt_url,$links[$i]);   curl_setopt($curl_handle, curlopt_connecttimeout, 2);   curl_setopt($curl_handle, curlopt_returntransfer, 1);   curl_setopt($curl_handle, curlopt_useragent, 'mozilla/5.0 (windows; u; windows nt 5.1; rv:1.7.3) gecko/20041001 firefox/0.10.1');   $query = curl_exec($curl_handle);   curl_close($curl_handle);   preg_match_all('/\b3\d+/', $query, $matches2);   $numbers = $matches2[0];     $count = 0;   foreach($numbers $value) {     if(strlen((string)$value) >= 10) echo '<br><br>[' . $count++ . "]" . $value;    }   //f    } //f

issue#1: html can have urls following picking links /home/test.php without base http://www.example.com/. before requesting curl, print on screen or browser , check is.

<a href="/home/test.php">link</a>

issue#2: 2 seconds curlopt_connecttimeout can prove less you. try increasing value.

curl_setopt($curl_handle, curlopt_connecttimeout, 10);

if problem still persists, please show sample page link. , sample internal link blank response.


Comments

Post a Comment

Popular posts from this blog

android - Get AccessToken using signpost OAuth without opening a browser (Two legged Oauth) -

i using signpost oauth access data magento server. have read various tutorials on same , reach point open browser user can enter credentials. however, per requirement have automate part. hence, user should not browser page. have set-up done on server side ( magento ), hit url , returned call page. same through program in android. i have tried below, consumer = new commonshttpoauthconsumer(key, secret); provider = new commonshttpoauthprovider(oauth_init_url,access_token_url, authorize_url); try { provider.retrieverequesttoken(consumer, oauth.out_of_band); log.d("tokens" , consumer.gettoken() + " -- " + consumer.gettokensecret()); } and request tokens. dont know how bypass next step. tried directly accessing accesstoken (stupid of me) provider.retrieveaccesstoken(consumer, "mycallback://callback"); no luck, ends in oauth.signpost.exception.oauthnotauthorizedexception: authorization failed (server replied 401). ...
Read more

org.mockito.exceptions.misusing.InvalidUseOfMatchersException: mockito -

getting error org.mockito.exceptions.misusing.invaliduseofmatchersexception: invalid use of argument matchers! 1 matchers expected, 2 recorded. exception may occur if matchers combined raw values: //incorrect: somemethod(anyobject(), "raw string"); when using matchers, arguments have provided matchers. example: //correct: somemethod(anyobject(), eq("string matcher")); at when( getprogramservice .callservice(any(getprogramtype.class))) .thenreturn(jaxbresponse); please explain error , possible resolution the code posted looks fine; don't see problem it. code before or above causing exception. key here: invalid use of argument matchers! 1 matchers expected, 2 recorded. any , used above, doesn't return "special kind of null" or "special kind of getprogramtype"; don't exist. instead, returns null , side effect puts matcher on stack. whe...
Read more

google shop client API returns 400 bad request error while adding an item -

i'm working on solution add items in google shopping market java application. following tutorial real mess because of missing classes of laste release, after setting 1.5-beta maven managed make base example typecheck correctly classes. still can make simple task of adding 1 item. instead of posting whole code i'm posting part creates product feed. private product createproduct(string id) { product product = new product(); product.title = "red wool sweater"; product.content = new content( "text", "comfortable , soft, sweater keep warm on " + "cold winter nights. red , blue stripes."); product.appcontrol = new appcontrol(); product.appcontrol.addrequireddestination("productads"); product.externalid = id; product.lang = "it"; product.country = "it"; product.condition = "nuovo"; product.price = new price("eu...
Read more