5

I am currently writing an Android app that, among other things, uses text information from websites which I do not own. In addition, some of the pages require authentification.

For some pages I have been able to log in and retrieve the html code using BasicNameValuePairs and an HTTPClient with its associated objects.

Unfortunately, these methods retrieve the webpage source without running any javascript functions that a browser (Android Webview even) would normally run. I need the text that some of these scripts are retrieving.

I've done my research, but everything I've found is guesswork & extremely confusing. I'm okay with ignoring pages that require login for now. Also, I am willing to post any code that may be useful for constructing a solution; It is an independent project.

Any concrete solutions for scraping the html result from javascript calls? An example would be absolutely top-notch.

bhekman
  • 3,227
  • 21
  • 24
  • Note: Many threads I've seen suggest looking into "Rhino", but I haven't found a way to make it work yet. Rhino: http://www.mozilla.org/rhino/tutorial.html – bhekman Jun 19 '12 at 01:11

2 Answers2

8

Final Success:

Other Things I Tried:

  • HttpClient provided by Android
    • Cannot run javascript
  • HtmlUnit
    • 4 hours, no success. Also huge, added 12 mb to my apk.
  • SL4A
    • Finally compiled. Used THIS guide to set-up. Abandoned as overkill for a simple rhino jar.

Things That Might Work:

  • Selenium

Further results will be posted. Others results will be added if posted.

Note: many of the options listed above reference each other. I think rhino is included in both sl4a and htmlunit. Also, I think htmlunit contains selenium.

bhekman
  • 3,227
  • 21
  • 24
4

The aforementioned solutions are very slow and restrict you to 1 url (well, not really, but I dare you to scrape 10 urls with Rhino while your user is impatiently waiting for results).

An alternative is to use a cloud scraping solution. You get the benefit of not wasting phone bandwidth on downloading content you won't use.

Try this solution: Bobik Java SDK

It gives you the ability to scrape up to hundreds of sites in a matter of seconds

Yevgeniy
  • 1,313
  • 2
  • 13
  • 26
  • Well my solution is working well enough for it's application, but I'll give you the accept since others seem to have found your answer useful. Thanks & kudos. – bhekman Jun 26 '12 at 21:46
  • Thanks. By the way, I wrote a detailed article about the advantages of using Bobik at http://zscraper.wordpress.com/2012/07/03/a-comparison-shopping-android-app-without-backend/ – Yevgeniy Jul 24 '12 at 19:58
  • usebobik.com is just an advertisement center nothing out there. – Nasz Njoka Sr. May 24 '15 at 19:46