Is it possible to use another web-site search results in mine?

Question

I have to develop a web app that shows certain places on a map, and those places can be found on another websites map already... is it possible that, from my app, I call that other website search form and gather the results?

Until they catch you and start blocking your requests. – Hot Licks May 17 '13 at 19:20 — Hot Licks, May 17 '13 at 19:20

score 5 · Answer 1 · edited May 23 '17 at 10:25

5

I would recommend to use a standard API for that:

YQL:

The Yahoo! Query Language is an expressive SQL-like language that lets you query, filter, and join data across Web services.
Bing API: typical usage.
How can you search Google Programmatically Java API

edited May 23 '17 at 10:25

Community

1
1

answered May 17 '13 at 17:29

0x90

39,472
36
165
245

Martin Perry · Accepted Answer · 2013-05-17T17:37:45.553

Its possible... but legality is another question.

If search engine on "another" web page using GET, then its simplier (of course, with POST can it be done too). Just create URL, load its content within your app and then write parser for resulting HTML. There is big probability, that results will be formated in stable way (table, divitis), so its easier. Just write simple regex for result HTML (or other type) and get results from that.

Problem with this "brute force" data mining from search result is, that if they change their page, your app "goes to hell". In my application, I solve that using extarnal config file including URL where you change only search query and external regexp.

Example:

URL: http://www.example.com/search.php?query=[SEARCHED_DATA]

Text [SEARCHED_DATA] is within app replaced with whateever you are searching.

If page is changed, you can easily commint app update without need of rebuilding it.

Some pages offering search APIs, which are better way. But from your question, I assume, that this it not the case.

Please read the terms and conditions of the website you are trying to scrape before doing this. Most sites explicitly forbid doing this. — Mahesh Guruswamy, May 17 '13 at 17:32
It is for a code test, I mean, a technical test I was asked to do for a course I am attending. Thanks — diminuta, May 17 '13 at 18:17

score 3 · Answer 3 · answered May 17 '13 at 17:28

3

Your best bet is to try to integrate with web sites which offer up some sort of API access to their data. Trying to scrape HTML from a 3rd party site is asking for trouble.

answered May 17 '13 at 17:28

Mahesh Guruswamy

769
5
15

3

Both legal and technical. – Michael Todd May 17 '13 at 17:28
2

If legal issues are solved, you still have the fact that they can change their site or layout or URL format at any time without warning. That will break your site. – Lee Meador May 17 '13 at 17:30
@LeeMeador, I am not recommending doing screen scraping...in fact I am recommending against it. – Mahesh Guruswamy May 17 '13 at 17:32
@MaheshGuruswamy I agree with you. Just giving a reason why you are right. – Lee Meador May 17 '13 at 17:41
I was looking for an API in their website to do this, but couldn't find it... but it's just for a code test I was asked to do... so it's not a commercial webapp I'm developing or something... – diminuta May 17 '13 at 18:18

Is it possible to use another web-site search results in mine?

3 Answers3