0

i had made a screen scraping module which works very fine but with certain limitations.now i want to remove those boundations,but i got so unpredictable and different error. Before anything goes in ur mind let me wat is actually hapening. Initially i used screen scraping to retrieve result for a set of keyword(search content) google's all search engine like co.in/co.uk/nl/de/com.

But now i had to scrape the logic for multiple search engine and multiple keywords in a loop.

Lets check out this with an example:

keyword     se            company         rank
telephony google.co.in    airtel          01
telephony google.co.in    bsnl            04
telephony google.co.in    aircel          06
telephony google.co.in    idea            03
mobile op google.co.uk    airtel          09
mobile op google.co.uk    bsnl            04

and so.. for more than 6 keywords and all shown search engines and for all company.

Initially i was retreiving it for one keyword,se and all company.but now i have to make a list of all keywords,se,company. Simply i used loops to do that.But i faced these errors:

  1. memory allocated 343322111 bytes overflowed(...[to remove this i used ini_set('memory') func]
  2. after sum request google used capcha. To remove capcha i used sleep, or usleep() but it not solving purpose.atlast ERROR: connection reset. I cant use 30sec or more in usleep func.it will take hours to retreive info.My code search data for 5pages of google, that means 50responses.Lib using simple_html_dom.php

It works fine for 1page page but not for greater than 3pages.What should i do/use??

Aakash Sahai
  • 3,935
  • 7
  • 27
  • 41
  • did you try setting time limit to 0 ? "set_time_limit(0);". for documentation : http://php.net/manual/en/function.set-time-limit.php (default is 30 seconds I guess) – frail Apr 01 '11 at 12:09
  • yes it is used.because of that only i am able to execute it for one page.because there may be cases when there is limited bandwidth. – Aakash Sahai Apr 01 '11 at 12:11

4 Answers4

2

The Captcha is Googles way to say you that they found you using it commercially, and want you to use their paid service from now on http://code.google.com/intl/en/apis/customsearch/v1/overview.html

As for the memory problem, we can't help you without some code overview. (But to conserve some at least split out the keywords instead of keeping complete pages or DOM parse trees around.)

mario
  • 144,265
  • 20
  • 237
  • 291
  • how can isend such a big code here...memory is not a issue not...as it was solved by ini_set() func. but now prob is how can i tell google or how to pause code excution for a specific time and connection also didnt get reset?? – Aakash Sahai Apr 01 '11 at 12:17
  • Your connection won't get reset or stalled if you use the advised JSON API. – mario Apr 01 '11 at 12:18
  • usleep working but dont know wats happenign...that my code is not completing.i got two problems.either connection reset/or capcha.wat to do now – Aakash Sahai Apr 01 '11 at 12:19
  • The solution to your problem is to stop screen scraping, and follow the link in above answer. – mario Apr 01 '11 at 12:21
0

It doesn't matter what time limit you're gonna use - it won't solve your problem. What you need to do is either use their API, which is inconsistent with the real results you see, or sign up for 100 proxies and iterate thru them all in a round robin format. You can easily scrape Google 24/7 with 100 proxies or so, and it just costs $100. Make sure you clear cookies after every request, and set a good user agent (nothing dumb that makes Google think you're a bad bot).

I rather do that than pay for their API which gives you X amount of calls and wastes your money. Yes, I know technically it's against their TOS, but it seems like what you're doing is harmless.

Henley
  • 21,258
  • 32
  • 119
  • 207
0

sleep() function with &num=100 in query solves the problem. Using &num=100 reduces the number of request to google 10times. and between every request i used 5 sec delay which google seems to be a valid,genuine,human request.

Aakash Sahai
  • 3,935
  • 7
  • 27
  • 41
0

Instead of selecting the first 5 pages of 10 results select 1 page of 50 results!

Make sure you use a typical user-agent so you don't look like a bot. To make yourself look less suspicious also follow some of the result links occasionally using Google's redirect URL, like a real user would.

You can also rent proxies, but the above techniques should be sufficient for most cases.

hoju
  • 28,392
  • 37
  • 134
  • 178