2

I'm trying to read in the HTML from a webpage and parse information from it using a URLConnection in Java. It works, but the page only loads part of the content, the rest is loaded as the user scrolls down the page. Is there any way for a Java program to trigger this? My program doesn't actually open the webpage in a browser, just a connection to the page. If it's relevant, I can add the URL I'm accessing.

I've been trying to find the answer, and found a few similar topics on here, most of them without answers. However, I eventually made my way to this topic, which sounds like what I need, but I looked at the URLs of the calls being made and they're not always the same, so I can't just type them into the program. I looked at the topic it was supposedly a duplicate of, but that didn't seem to apply to my problem either, unless I misunderstood something. Is there any way to find these URLs each time the program runs, or any way to trick the connection into thinking I'm scrolling down the page? Or can I make a general "request" or "POST" as I've seen in some related topics, that will automatically call the appropriate URL (An explanation of a "POST" would be appreciated as well)?

Community
  • 1
  • 1
Tim
  • 95
  • 3
  • 7
  • 2
    _POST_ - use Google. As for the rest: it's called lazy loading. If you want to extract the rest of the content you either have to emulate what the javascript in the source document is doing (which probably means reverse-engineering the script), or you have to provide an environment in which you can execute the javascript. Neither of these options is easy. Does the web site have an API you could use? –  Apr 04 '14 at 01:36
  • @Mike No, as far as I can find the website doesn't have an API. Should I even bother then? This information isn't critical, I can still get enough, it'd just be nice to have access to more. – Tim Apr 04 '14 at 01:41

0 Answers0