-1

hi i am trying to build a simple application using grails wherein i need to crawl 3 websites to get data abt the price off the book.And after getting those details when i select to buy it has to redirect to tht selected site.example refer the link http://www.mydiscountbay.com/ I am stuck i dont know hw to implement a simple crawler in grails.pls guide me with a sample code or tutorial on hw to implement it

thanks in advance

pradeep
  • 1
  • 2

2 Answers2

0

Implementing crawler has nothing to do with grails, there are some opensource java crawlers that you may be able to use or customize as per your need. Front end part would be like a normal grails web app.

Sudhir N
  • 4,008
  • 1
  • 22
  • 32
  • I would say write your questions here, that way other's will be able to help you as well. – Sudhir N Apr 30 '12 at 04:48
  • sure no probs,k i heard tht the simplest way to get the contents of an HTML page is to call getText() on the URL.hw do i implement this.I mean where to add the code and stuff. – pradeep Apr 30 '12 at 04:53
  • You do that in service - do you already know grails, or you have absolutely no idea ? – Sudhir N Apr 30 '12 at 08:33
  • i am a beginner to grails. So could you please post a simple code snippet to do a http get call in grails. Thanking you in advance – pradeep Apr 30 '12 at 09:58
0

Using something like URL#getText() will not get you very far with webs that have redirections, cookies, etc.

For anything even a little bit involved, use commons HttpClient, or the groovy HttpBuilder.

http://hc.apache.org/httpcomponents-client-ga/index.html

http://groovy.codehaus.org/HTTP+Builder

To parse the response and extract content, use XmlSlurper, eg: Using XmlSlurper: How to select sub-elements while iterating over a GPathResult

Community
  • 1
  • 1
Luis Muñiz
  • 4,649
  • 1
  • 27
  • 43