Trying to download URL contetnt into .txt file using jsoup

Question

I am trying to download the content of pages like this one and writing it into a .txt file for later use.

doc = Jsoup.connect(link).userAgent("Mozilla").get();
String cityInfo = doc.html();            

int index = cityInfo.indexOf("},"); // keeps just the first object as it has the highest score.
String cityInfo1 = cityInfo.substring(index+1) + "}]}"; // gets the object in the correct format as some characters are not selected when downloading
bw1.write(cityInfo1); //saves json object into text file

I keep getting this error, and if I use the ignoreContentType(true) method it just gets rid of the error and my text file remains empty.

"Exception in thread "main" 
org.jsoup.UnsupportedMimeTypeException: Unhandled content type. 
Must be text/*, application/xml, or application/xhtml+xml.
Mimetype=application/json, URL=http://transport.opendata.ch/v1/locations?query=Aarau"

I want to download the whole page only as text and handle it later. What would you suggest? — BlueWookie, May 12 '16 at 10:39
Check out the JSON Format: https://en.wikipedia.org/wiki/JSON — Tim, May 12 '16 at 10:42
This should answer your question: http://stackoverflow.com/questions/7467568/parsing-json-from-url — Tim, May 12 '16 at 10:45
I didn't get if the first part reads the page's URL or the page's content — BlueWookie, May 12 '16 at 10:59
It reads the content. Then, in the second part, the content is parsed and can be used. — Tim, May 12 '16 at 11:08
Thanks man! It worked fine! Should I delete my question now? since there is not an answer to that.. — BlueWookie, May 12 '16 at 21:14
You might as well leave it, it might be helpful to someone else :) — Tim, May 13 '16 at 06:34

score 0 · Answer 1 · answered May 12 '16 at 10:21

0

Add the ignoreContentType(true):

doc = Jsoup.connect(link).ignoreContentType(true).userAgent("Mozilla").get();

answered May 12 '16 at 10:21

Lee

738
3
13

I tried but still nothing gets written in my text file. I get this instead: "Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=405, URL=http://transport.opendata.ch/v1/locations?query=Aarau" – BlueWookie May 12 '16 at 10:27
Ah, then that's the inverse - the 405 is indicating that the server doesn't understand what you are supplying or isn't willing to return a type that it thinks you are asking for. The solution is to be more specific about the content type you supply and request. – Lee May 12 '16 at 10:32
So should I download it as just text, JSON or another format type? – BlueWookie May 12 '16 at 10:40

Trying to download URL contetnt into .txt file using jsoup

1 Answers1