I'm trying to put together a Jsoup app that pulls four factors data from the following site:
https://www.nba.com/stats/teams/four-factors
I ran into a problem recently where Jsoup was pulling blanks and after some searching I found out that it's because the page loads the data through JSON which has a delay, so when Jsoup connects and goes to extract the data there's nothing there.
My question is, is there a way to delay Jsoup from pulling the data until the page is properly loaded?
My code is below. Any thoughts or suggestions are much appreciated:
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
import org.jsoup.nodes.Element;
public class Comparison {
public void loadTeams() {
try {
ArrayList<Team> NBA = new ArrayList<Team>(30);
Document nba = Jsoup.connect("https://www.nba.com/stats/teams/four-factors").get();
Elements team = nba.getElementsByAttributeValue("field", "TEAM_NAME");
Elements shooting = nba.getElementsByAttributeValue("field" , "EFG_PCT");
Elements turnovers = nba.getElementsByAttributeValue("field" , "TM_TOV_PCT");
Elements rebounds = nba.getElementsByAttributeValue("field" , "OREB_PCT");
Elements freethrows = nba.getElementsByAttributeValue("field" , "FTA_PCTS");
for (int i = 0; i < team.size(); i++) {
String name = team.get(i).text();
double shootingScore = Double.parseDouble(shooting.get(i).text());
double turnoversScore = Double.parseDouble(turnovers.get(i).text());
double reboundsScore = Double.parseDouble(rebounds.get(i).text());
double freethrowsScore = Double.parseDouble(freethrows.get(i).text());
NBA.add(new Team(name, shootingScore, turnoversScore, reboundsScore, freethrowsScore));
}
//for testing//
for(int i=0; i<NBA.size(); i++) {
System.out.print(NBA.getClass());
}
}
catch (IOException e)
{
System.out.println(e.getMessage());
}
}
}