Reading large data from an inputstream some data missing at the end

Question

Java(TM) SE Runtime Environment (build 1.7.0_45-b18)

Hello,

I am using the httpUrlConnection to retrieve a json string from a webservice. Then I get the inputStream from the connection

jsonString = readJSONInputStream(mHttpUrlconnection.getInputStream());

I then use the following function to read the inputstream to get the JSON.

private String readJSONInputStream(final InputStream inputStream) {
    Reader reader = null;

    try {
         final int SIZE = 16024;

         char[] buffer = new char[SIZE];
         int bytesRead = 0;
         int read = 0;

         reader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"), SIZE);

         String line = "";
         String jsonString = "";

         while((line = reader.readLine()) != null) {
            jsonString += line;
         }

         /* Success */
         return jsonString;
    }
    catch(IndexOutOfBoundsException ex) {
        log.log(Level.SEVERE, "UnsupportedEncodingexception: " + ex.getMessage());
    }
    catch(IOException ex) {
        log.log(Level.SEVERE, "IOException: " + ex.getMessage());
    }
    finally {
        /* close resources */
        try {
            reader.close();
            inputStream.close();
        }
        catch(IOException ex) {
            log.log(Level.SEVERE, "IOException: " + ex.getMessage());
        }
    }

    return null;
}

However, if the json is small say 600 bytes then everything is ok. But I have some JSON that is about 15000 bytes in size so I set the maximum size to 16024.

However, the JSON it only reads about about ~6511 and just cuts off.

If the JSON is small there is no problem < 1000 bytes. But for the larger JSON it only read about half of it.

I the data is there as I have tested this in a browswer using the http request plugin.

Am I doing anything wrong here. Anything I should check.

Many thanks for any suggestions,

How do you know how large the content should be in your program if you have never read it successfully? Could there be an encoding issue at hand here? I think you should post code which can be dumped into any IDE and run for testing. — Tim Biegeleisen, Nov 19 '15 at 05:25
Our webservice is where the json is. So our app will retrieve this json from our webservice. So we know what it contains. — ant2009, Nov 19 '15 at 05:32
I'm limited in my capacity to help because I don't even have the URL. However, have you compared the JSON you _are_ able to download against what you cannot download? Use an online diff checker and let us know what is missing. — Tim Biegeleisen, Nov 19 '15 at 05:37
I can't give the url as it our internal company's data. However, I have already compared what is on the webservice and what I have downloaded. Just ~half is always downloaded. The JSON is valid so there is nothing wrong with that. Our smaller JSON data is ok. I thought it had something to do with me reading from the inputstream. — ant2009, Nov 19 '15 at 05:56
Does _every_ JSON file have the same cutoff experience? Or is this particular to this single file? — Tim Biegeleisen, Nov 19 '15 at 06:00
You have declared byte array and not using it effectively. Sample code: out = new FileOutputStream(args[1]);//// or you can hard code the filename // Read data into buffer and then write to the output file byte[] buffer = new byte[8192]; int bytesRead; while ((bytesRead = in.read(buffer)) != -1) { out.write(buffer, 0, bytesRead); }. Related question: http://stackoverflow.com/questions/31390793/cannot-download-file-from-url-in-java/33252453#33252453 — Ravindra babu, Nov 19 '15 at 06:12
I have other JSON's that work fine. But they are all less than 1000 bytes in length. I have just problem with this one, as it is about 15000 bytes in size. — ant2009, Nov 19 '15 at 06:12
Byte array code handles more than 3 MB file properly without any issues. — Ravindra babu, Nov 19 '15 at 06:15
I have tested that code snippet and everything works fine and the json that gets written to the file contains the valid JSON. However, I need to write this to a String and return that. — ant2009, Nov 19 '15 at 06:56
Since when does `IndexOutOfBoundsException` mean 'unsupported encoding'? And where in your code is it even possible for `IndexOutOfBoundsException` to be thrown? When you get an exception, print *its* message, not some arbitrary sequence of characters of your own. And what exactly is `buffer` for? — user207421, Nov 19 '15 at 09:04

score 0 · Accepted Answer · answered Nov 20 '15 at 08:38

0

Problem resolved. Due to the logger not displaying all the information. Not really a problem afterall.

answered Nov 20 '15 at 08:38

ant2009

27,094
154
411
609

Reading large data from an inputstream some data missing at the end

1 Answers1