2


My goal is to create a small sript that find all the result of a google search but in "raw".
I don't speak english very well so i prefer to give an exemple to show you what i would like :
I Type : elephant
The script return
www.elephant.com
www.bluelephant.com
www.ebay.com/elephant
.....

I was thinking about urllib.request but the return value will not be usable to that ! I found some tutorials but not adapted at all to my wish ! Like i told you my goal is to have an .txt file as output witch contains alls the website who match with my query !
Thanks all

Matthieu Mollard
  • 45
  • 1
  • 2
  • 11

4 Answers4

4

One simple way is to make a request to google search, then parse the html result. You can use some Python libraries such us Beautiful Soup to parse the html content easily, finally get the url link you need.

Hui-Yu Lee
  • 909
  • 8
  • 20
2

These seem to change often, so hopefully this answer remains useful for a little while...

First, you'll need to create a Google Custom Search, by visiting their site or following the instructions provided here https://developers.google.com/custom-search/docs/tutorial/creatingcse.

This will provide you with both

  • Custom Search Engine ID
  • API Key

credentials which are needed to use the service.

In your python script, you'll want to import the following package:

from googleapiclient.discovery import build

which will enable you to create a build object:

service = build("customsearch", developerKey=my_api_key)

According to the docs, this constructs a resource for interacting with the API.

When you want to return search results, call execute() on service's cse().list() method:

res = service.cse().list(q=my_search_keyword, cx=my_cse_id, **kwargs).execute()

to return a list of search results, where each result is a dictionary object. The i'th result's URL can be accessed with the "link" key:

ithresult = res[i]['link']

Note that you can only return 10 results in a single call, so make use of the start keyword argument in .list(), and consider embedding this call in a loop to generate several links at a time.

You should be able to find plenty of SO answers about saving your search results to a text file.

N.B. One more thing that confused me at first - presumably you'll want to search the entire web, and not just a single site. But when creating your CSE you will be asked to specify a single site, or list of sites, to search. Don't worry, just enter any old thing, you can delete it later. Even Google endorses this hack:

Convert a search engine to search the entire web: On the Custom Search home page, click the search engine you want. Click Setup, and then click the Basics tab. Select Search the entire web but emphasize included sites. In the Sites to search section, delete the site you entered during the initial setup process.

9th Dimension
  • 305
  • 1
  • 5
  • 16
  • Thanks a lot for this answer, I'll work on it !! Thank you again – Matthieu Mollard Jun 11 '16 at 09:17
  • The main reason I put this answer here is because when I did this, I needed to scrape images from google image search. I couldn't work out how to get the image URLs by parsing the Google Image search query result page (but then again I don't know anything about web or networking). – 9th Dimension Aug 30 '16 at 13:05
1

I just add 2 points to "9th Dimension" answer.

  1. Use this guide to find your Custom Search Engine ID
  2. A small modification should be made in the second line of the code: as the following, the "version" should be added as an argument

service = build('customsearch','v1',developerKey= my_api_key)

payam.1991
  • 71
  • 1
  • 3
0

You have 2 options - using API or make a request like a browser does and then parse HTML.

  • First option is rather tricky to set up and is limited - 100 free queries/day, then 1000 for $5.

  • Second option is easier but it violates Google's ToS.

Community
  • 1
  • 1
Daerdemandt
  • 2,281
  • 18
  • 19