0

I would like to write a program that loops through about 7000 entries of a certain keyword and then searches Google for that keyword. Then for each keyword, I would like to return the number of search results. Finally, I would like to populate a pandas dataframe like structure with keyword1, no_results1, keyword2, no_results2, ..., keywordn, no_resultsn.

I am proficient with python and use of web scraping tools like BeautifulSoup and selenium. However, I am also aware that it is not necessarily legal or conforming to Google's terms and conditions to use their search engine willy nilly.

Therefore, I looked into the google JSON API which appears to be somewhat suited to my usage. However, the self-proclaimed goal of this API is as follows:

Google Custom Search enables you to create a search engine for your website, your blog, or a collection of websites. You can configure your engine to search both web pages and images. You can fine-tune the ranking, add your own promotions and customize the look and feel of the search results. You can monetize the search by connecting your engine to your Google AdSense account.

Which is not quite what I need. Is it possible/legal to use google custom search features without creating one's own website?

user32882
  • 5,094
  • 5
  • 43
  • 82
  • If you think this question should be closed, would you please indicate why so that I can improve it. Thanks – user32882 Jan 10 '19 at 09:27
  • Each question in SO should have one question in it, not two. Even so, the two questions you have asked are far too broad to provide any meaningful answer without guiding you through the entire process. – Ethan Field Jan 10 '19 at 09:45
  • 1
    It's a REST API, so the answer to your question is "Yes, you can use it without creating a website". If you need to know more than that, search "How to poll REST API with Python". – Ethan Field Jan 10 '19 at 09:48
  • @Ethan Field... doing some reading now. Have you ever used it without creating a website? – user32882 Jan 10 '19 at 09:58
  • I've never used this API before, but I have used Python to interact with REST APIs before. There is also a question right here on SO for it: https://stackoverflow.com/questions/17301938/making-a-request-to-a-restful-api-using-python – Ethan Field Jan 10 '19 at 10:03
  • Actually I found out that custom search returns inferior results than a regular google search https://stackoverflow.com/questions/20182667/google-api-returning-different-results-than-website. This kind of makes it obsolete for my use case – user32882 Jan 10 '19 at 10:46
  • Are you planning on using the googleapiclient module to accomplish the keyword searching? – Life is complex Jan 11 '19 at 18:43
  • How is googleapiclient different from the Custom Search Engine? – user32882 Jan 11 '19 at 19:29
  • Here is the difference: https://support.google.com/customsearch/answer/70392?hl=en – Life is complex Jan 20 '19 at 18:31

0 Answers0