4

I am currently trying to get a list of Google Search results via Python.

Many different package have stopped working or have been deprecated when google changed the html layout several years ago e.g. pygoogle, xgoogle

Searching for "Hiking Trails Los Angeles" on Google, how do I return the top 10 results, ideally with url, title and description or other attributes that are available?

Goofball
  • 735
  • 2
  • 9
  • 27

1 Answers1

7

Yh, google-search-api has been deprecated, hence the pygoogle which was a wrapper for the Google search api. At the top of the search api page, there's a warning, along with:

We encourage you to investigate the Custom Search API, which may provide an alternative solution.

But using this custom search api to search the whole web isn't pretty straightforward. Here I found 2 detailed guides (SO answers):

  1. Programmatically searching google in Python using custom search
  • 1st step: get Google API key.
  • 2nd step: setup Custom Search Engine so that you can search the entire web.
  • 3rd step: install Google API client for Python.
  • 4th step (bonus): do the search.

So, after setting this up, you can follow the code samples from few places:

  1. What are the alternatives now that the Google web search API has been deprecated?

Yes, Google Custom Search has now replaced the old Search API, but you can still use Google Custom Search to search the entire web, although the steps are not obvious from the Custom Search setup.

To create a Google Custom Search engine that searches the entire web:

  1. From the Google Custom Search homepage ( http://www.google.com/cse/ ), click Create a Custom Search Engine.
  2. Type a name and description for your search engine.
  3. Under Define your search engine, in the Sites to Search box, enter at least one valid URL (For now, just put www.anyurl.com to get past this screen. More on this later ).
  4. Select the CSE edition you want and accept the Terms of Service, then click Next. Select the layout option you want, and then click Next.
  5. Click any of the links under the Next steps section to navigate to your Control panel.
  6. In the left-hand menu, under Control Panel, click Basics.
  7. In the Search Preferences section, select Search the entire web but emphasize included sites.
  8. Click Save Changes.
  9. In the left-hand menu, under Control Panel, click Sites.
  10. Delete the site you entered during the initial setup process.

Google Custom Search is not free all the way i.e. Pricing:

  • Custom Search Engine (free) For CSE users, the API provides 100 search queries per day for free. If you need more, you may sign up for billing in the API Console. Additional requests cost $5 per 1000 queries, up to 10k queries per day.
  • Google Site Search (paid). For detailed information on GSS usage limits and quotas, please check GSS pricing options.
Community
  • 1
  • 1
Nabeel Ahmed
  • 18,328
  • 4
  • 58
  • 63