0

I'm trying to read the text from https://mtgjson.com/api/v5/AllPrintings.json. I have tried with this code:

url = new URL("https://mtgjson.com/api/v5/AllPrintings.json");
conn = (HttpsURLConnection) url.openConnection();

BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream())); // error here

String inputLine;
StringBuffer content = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
    content.append(inputLine);
}
System.out.println(content);

I keep getting IOException with the BufferedReader (conn.getInputStream()). The text from the url does not contain a new line character. How can I read this data?

(Edit)
I'm using Java 1.8 with Apache NetBeans 16. I'm sticking with 1.8 so I can also use Eclipse Neon3.

Error:

java.io.IOException: Server returned HTTP response code: 403 for URL: https://mtgjson.com/api/v5/AllPrintings.json
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
    at tests.MtgJson.main(MtgJson.java:44)

I've also been trying ProcessBuilder with curl and it's giving better results but curl stops after about a minute. Curl continues if I terminate the program inside Netbeans but doesn't always finish creating the file contents. I shouldn't have to stop my program for curl to continue. Is there something I'm missing for curl to work?

String command = "curl --keepalive-time 5 https://mtgjson.com/api/v5/AllPrintings.json";
ProcessBuilder pb = new ProcessBuilder(command.split(" "));
pb.redirectOutput(new File("AllPrintings.json"));
Process process = pb.start();
// use while() or process.waitfor();
while(process.isAlive())
    Thread.sleep(1000);
process.destroy();

Answer (since I can't post one):

String command = "curl https://mtgjson.com/api/v5/AllPrintings.json";
ProcessBuilder pb = new ProcessBuilder(command.split(" "));

pb.inheritIO(); // keep the program from hanging
pb.redirectOutput(new File("AllPrintings.json"));

Process process = pb.start();
process.waitFor(); // waiting for the process to terminate.

The complete file is created without hanging then the program will close. Curl outputs info to the console and must be consumed (found here).

Daniel Vaughn
  • 13
  • 2
  • 7
  • 2
    What do you want to do with that huge lump of data? Just print it to the screen? You probably can't load it into memory as a whole as long as you don't modify the JVM settings for heap size, but depending on your needs, maybe you don't need to... – cyberbrain Feb 12 '23 at 21:32
  • 3
    To start with, don't use a line-oriented method when there are no lines in the input. You can just use read() on the InputStream, in convenient buffer-loads. I'd also try and avoid building the entire 300M-char string (=600M bytes as UTF-16). – undefined symbol Feb 12 '23 at 21:33
  • @cyberbrain I want to save it as a file then parse the file for an offline database. I would like to get my application to do this so the user doesn't have to download it themselves. I already have a way to parse the file so I need a way to get the file. – Daniel Vaughn Feb 13 '23 at 03:39
  • https://stackoverflow.com/a/25045224/21197077 this might be helpful – gneginskiy Feb 13 '23 at 10:06
  • FYI: you're not doing yourself many favors by staying on a 9 year old Java version that has reached its EOL last year in order to stay on an IDE release version from 7 years ago. Eclipse is free and JDK installations are free. Get the latest Eclipse and a recent JDK. – Joachim Sauer Feb 14 '23 at 08:53

1 Answers1

0

There is no need to use byte->character conversion with BufferedReader just to make a copy. Instead copy the content directly to a file using Java NIO Files.copy, and then use the output file for any further processing:

Path file = Path.of("big.json"); 
// Older JDK use Paths.get("filename")
Files.copy(conn.getInputStream(), file);
System.out.println("Saved "+Files.size(file)+" bytes to "+file);

Which should print:

Saved 313144388 bytes to big.json
DuncG
  • 12,137
  • 2
  • 21
  • 33
  • Sorry, I should of said the version of java I'm using so I will edit my question (Java 1.8 with Apache NetBeans 16). `of()` isn't a method I can use. I was probably too focused on trying to keep it short. Though I can use `Paths.get(URI)` or `Paths.get(String, String...)`. Unfortunately I am still getting an IOException with `Files.copy()`. The similarity between the errors is `conn.getInputStream()`. – Daniel Vaughn Feb 14 '23 at 02:50
  • It's a http 403 response. Not sure why I would get that if I can view the data in a browser. – Daniel Vaughn Feb 14 '23 at 07:22
  • Add the exception details to the question. – DuncG Feb 14 '23 at 07:59
  • Note that I've run above code on JDK19. – DuncG Feb 15 '23 at 08:27