Make program running a web site for different inputs

Question

This is a general question, and I don't expect any specific solution here. I'm just very curious about the approach.

Let us say I have txt-file with many keywords. There are so many keywords that I can't make a seperate google-search for each one of them. That would take too long time. My goal is to make a program which takes every keyword in the file, makes a google-search for that keyword, then saves the result in a file (e.g. a line at the result page), then proceeds to the next keyword and does the exact same procedure.

I wonder if this is possible at all? What kind of programming language should I use? what would be the approach?

The first thing is that massive automated script is not allowed from nearly all sites for example google https://code.google.com/apis/customsearch/v1/overview.html doesn't allow more than 100 request per day per user from a computer program — Xavier Combelle, Dec 22 '11 at 15:39

score 2 · Answer 1 · edited May 23 '17 at 12:27

2

Here is one way of doing it.

Create a java(or any other language) program which will read the file with keywords.
For each keyword read from the file, Use Google Search APIs to search (See example here).
Save the the results returned from API in a file.

edited May 23 '17 at 12:27

Community

1
1

answered Dec 22 '11 at 13:46

Santosh

17,667
4
54
79

Zain Khan · Answer 2 · 2011-12-23T06:16:13.087

1

You are looking for something like bowser automation I suppose. You can look for Selenium and choose python as your tool.

edited Dec 23 '11 at 06:16

answered Dec 22 '11 at 13:42

Zain Khan

3,753
3
31
54

score 1 · Accepted Answer · edited Dec 22 '11 at 14:01

1

Sounds like a job for a scripting language. You could go with wget in a shell script, perl/python etc, or you could look into the api google provides.

edited Dec 22 '11 at 14:01

Michael Blaustein

901
6
24

answered Dec 22 '11 at 13:45

Thomas

5,074
1
16
12

score 1 · Answer 4 · answered Dec 22 '11 at 13:50

Yes, this would be possible.

Pretty much any programming language you like would do this. I will assume, from your question, that you are new to programming, so I might recommend VB.NET, as the syntax is pretty simple.

You can form your own URLs using parameters, like http://www.google.com/search?as_q=nintendo+wii&as_sitesearch=.com. A Google search for Google Search URL parameters might be a good place to start.

score 1 · Answer 5 · answered Dec 22 '11 at 15:46

1

Taking account the limitation of google api that is possible. I think the best tool for it is python.

answered Dec 22 '11 at 15:46

Xavier Combelle

10,968
5
28
52

Make program running a web site for different inputs

5 Answers5