1

I am trying to write a Python 3 function that works like the following:

links = google_search('"done" "store"')
for l in links: print(l)

The output should be a very long list, something like:

www.cnn.com/articles/done_with_sotres.html
www.something.org/whatever?p=bla
.
.
.

I found this suggestion, Google Search from a Python App, but I seem to get only 4 urls in "hits", and I am unsure how to get the rest. any suggestion will be highly appreciated!

EDIT: stupid me! I didn't put the implementation. anyway, it's the one described in the link:

def search(search_string):
  query = urllib.parse.urlencode({'q': search_string})
  url = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&%s' % query
  search_response = urllib.request.urlopen(url)
  search_results = search_response.read().decode("utf8")
  results = json.loads(search_results)
  data = results['responseData']
  print('Total results: %s' % data['cursor']['estimatedResultCount'])
  hits = data['results']
  print('Top %d hits:' % len(hits))
  for h in hits: print(' ', h['url'])
  print('For more results, see %s' % data['cursor']['moreResultsUrl'])
  return hits
Community
  • 1
  • 1
UriCS
  • 175
  • 2
  • 11
  • It would seem that the problem is in your `google_search` method, but you didn't provide its implementation. It makes it impossible to answer your question with any kind of certainty. – maksimov Oct 30 '14 at 00:51
  • my bad. I added it. tanks! – UriCS Oct 30 '14 at 01:47

1 Answers1

5

The maximum number of results that can be obtained from this api is 8 results for each query.

You get it by adding a "&rsz=large" to the url:

url = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&%s&rsz=large' % query

There is another useful argument "start=" which let you move in the result set. So basically you may loop on you can ask 1st block of 8 results, second block and so on(start=1, start=8 and so on).

url = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=large&%s&start=%d' % (query, i)

In any case please note that this api is deprecated (https://developers.google.com/web-search/docs/)

aberna
  • 5,594
  • 2
  • 28
  • 33