0

I’m trying to get data from a website but it takes a second to load (JavaScript table) so I get inconsistent results (sometimes more of the webpage sometimes less) depending on the connection speed.

I’m using MATLAB and have the following code:

% DEFINE URL
URL = 'https://www.wunderground.com/history/daily/us/il/chicago/KORD';
jUrl = java.net.URL(URL);

% CONNECTION
conn = jUrl.openConnection() 
conn.setRequestProperty{<<needed details here>>};
conn.connect()

% OUTPUT
output = java.io.ByteArrayOutputStream()
output.flush()

% INPUT
ins = conn.getInputStream();
org.apache.commons.io.IOUtils.copy(ins,output);

output.close();
ins.close();

Is there a way to implement a pause to wait for the javascript on the page to fully load?

Thanks.

Brian Parme
  • 1
  • 1
  • 1
    Is what you posted the full Java code for reading the file? Because for Java code it looks incomplete. Also Java uses InputStream to read data, that code looks more like sending data. – Robert Sep 14 '22 at 19:32
  • There is certainly a way to make this work. It seems like the Matlab `pause` command is the direct answer to your question. But there are probably other/better options to explore. To echo @Robert 's comment, can you check the reference code you posted, to make sure that it works as intended? (Except for the delay issue of course.) Note: 4 spaces at the beginning of a line formats the code as code, for much better readability. – Pursuit Sep 14 '22 at 21:00
  • Thanks. I've updated the post with the full code and formatted. I've tried the pause command but it doesn't seem anywhere I place it to make a difference. – Brian Parme Sep 15 '22 at 03:13
  • It may have to do with calling a method doing something like this : myDate = java.util.Date; javaMethod('setMonth',myDate,3) – John BG Sep 15 '22 at 04:44
  • Java with `URL` and `openConnection` does not interpret nor execute JavaScript included on the page you load, thus this can not be the problem. The only thing you could add is to test the status/response code if it is 200 for HTTP OK: https://stackoverflow.com/a/6467922/150978 – Robert Sep 15 '22 at 07:22
  • I had a line in there for a while getting the code and I do get 200 right after conn.connect – Brian Parme Sep 15 '22 at 13:57

0 Answers0