7

I have used 3 languages for Web Scraping - Ruby, PHP and Python and honestly none of them seems to perfect for the task.

Ruby has an excellent mechanize and XML parsing library but the spreadsheet support is very poor.

PHP has excellent spreadsheet and HTML parsing library but it does not have an equivalent of WWW:Mechanize.

Python has a very poor Mechanize library. I had many problems with it and still unable to solve them. Its spreadsheet library also is more or less decent since it unable to create XLSX files.

Is there anything which is just perfect for webscraping.

PS: I am working on windows platform.

Shubham
  • 21,300
  • 18
  • 66
  • 89
  • 3
    Excel format support is sort of orthogonal to web scraping... – Josh Lee Aug 12 '10 at 13:21
  • 2
    I think 'perfect' is pretty subjective, however, have you tried .NET? I think its XML parsing library is pretty strong and you can tie into MSOffice pretty easily as well for Spreadsheet support. – Tommy Aug 12 '10 at 13:22
  • @Tommy: Does it have an equivalent of Mechanize? I had used it with WebBrowserControl but found it inefficient. – Shubham Aug 12 '10 at 13:25
  • I;ve never used that particular function, but here could be a start for you to begin to look into finding similar functionality. http://stackoverflow.com/questions/122614/is-there-a-net-equivalent-of-perls-lwp-wwwmechanize – Tommy Aug 12 '10 at 13:41
  • What, exactly, do you want to scrape from the web? Everything? XML data? What you want from the web can completely change what you want to do. – Wayne Werner Aug 12 '10 at 13:51
  • @Wayne - I need just plain simple data from various sites. Some sites require login while some to post forms. – Shubham Aug 12 '10 at 13:56
  • Ruby is my favourite - especially the Mechanize gem (I'm told the Johnson gem can be used to interpret the javascript on pages too, where necessary). Why are you using Excel spreadsheets though? Surely data scraped from the web will be perfect for a database of some sort? When I *need* to get data into Excel I just generate a CSV and open with Excel. – JP. Aug 12 '10 at 13:59
  • @JP - Well just need some basic formatting of cells - alignment, hyperlinks etc.. – Shubham Aug 12 '10 at 14:09
  • Mechanize for python is **very poor**?? I have used it lots and I didn't know that. – John La Rooy Aug 13 '10 at 00:56

4 Answers4

2

Check Python + Scrappy, it is pretty good:

http://scrapy.org/

juanjux
  • 621
  • 6
  • 15
1

Why not just use the XML Spreadsheet format? It's super simple to create, and it would probably be trivial with any type of class-based system.

Also, for Python have you tried BeautifulSoup for parsing? Urllib+BeautifulSoup makes a pretty powerful combo.

Wayne Werner
  • 49,299
  • 29
  • 200
  • 290
1

Short answer is no.

The problem is that HTML is a large family of formats - and only the more recent variants are consistent (and XML based). If you're going to use PHP then I would recommend using the DOM parser as this can handle a lot of html which does not qualify as well-formed XML.

Reading between the lines of your post - you seem to be:

1) capturing content from the web with a requirement for complex interaction management

2) parsing the data into a consistent machine readable format

3) writing the data to a spreadsheet

Which is certainly 3 seperate problems - if no one language meets all 3 requirements then why not use the best tool for the job and just worry about an suitable interim format/medium for the data?

C.

symcbean
  • 47,736
  • 6
  • 59
  • 94
0

Python + Beautiful Soup for web scraping and since you are on windows, you could use win32com for Excel automation to generate your xlsx files.

Martin
  • 5,954
  • 5
  • 30
  • 46