-1

I would like to develop a system in python to get the position and number of the results of keywords in google.

I tried using the Google API, however, I found that with the use of custom search engine and I have different results with fewer pages compared to "official" google. I've tried a variety of ready-made modules (eg: xgoogle, pattern, etc.).

I tried scraping but I do not think is the right way.

My question is: what is the best approach to achieve the results required by me without scraping? It is not possible to have them?

RoverDar
  • 441
  • 2
  • 12
  • 32
  • 1
    Google is very clear that no scraping is allowed in their TOS. And since they do not provide other means to get what you need then you cannot get it without breaking TOS. – Tymoteusz Paul Sep 23 '13 at 13:30
  • People keep quoting the Google TOS to such questions, who tells them that the guy asking accepted the TOS ? You do not need to accept the TOS to access Google, it's public without account or registration. Check here for a detailed answer on the topic: http://stackoverflow.com/questions/22657548/is-it-ok-to-scrape-data-from-google-results/22703153#22703153 – John Mar 22 '17 at 14:31
  • @John advising people to break the law is generally a bad idea. The TOS is a form of contract, where you accept the contract by not immediately disconnecting the socket if you do not agree to the terms. – Tritium21 Jul 08 '17 at 09:30
  • @Tritium21 Accusing people of breaking the law or advertising to do that is more than just a bad idea as well. Google is not a country, I am not living in Google either. Their TOS is not the law ... You do not accept a TOS automatically without even looking at it, in no country in this world. So if you choose to accept the Google TOS that's fine but please do not white-knight force others to do the same. Google never accepted my TOS by the way, on my website I also have a TOS that disallows automated access. Funny fact: Google, Bing, Yahoo, Yandex and all the others give a damn about it – John Jul 08 '17 at 22:27
  • @john Google has a tos - which is a contract you agree to by using the service. if you break the tos, you are breaking the contract, which means you are in violation of the law, and are liable to be sued. I am flagging your comments as offensive because they are potentially encouraging illegal activities. – Tritium21 Jul 09 '17 at 02:40
  • Well that's not how law works but I end this conversation now, ignorance as a developer is a real dead end. – John Jul 10 '17 at 09:16

1 Answers1

0

Always use web services APIs. If a site does not have one, and you are not the site's owner, then that is an indication that they do not want you to use automatic tools to fetch their data.

Additionally, fetching a keyword rank is dubious at best. Google adjusts its rank on many factors, including your search history, your location, your locale, and any number of factors that are trade secrets. Using the API will be the most generic results, even if they don't match the ones you get when you search from Firefox or Chrome.

TL;DR Your best bet are the methods you already don't like.

Tritium21
  • 2,845
  • 18
  • 27
  • The API is too limited. You are right about the locale. The country, language and Google domain all play a significant role. For a "general ranking" I would use US/US/google.com. The API is kinda useless and can only be used for small volume, I would not recommend using it for many scenarios. – John Mar 22 '17 at 14:34