Byte array to file in java without overwriting,

Question

The code below gets a byte array from an HTTP request and saves it in bytes[], the final data will be saved in message[].

I check to see if it contains a header by converting it to a String[], if I do, I read some information from the header then cut it off by saving the bytes after the header to message[].

I then try to output message[] to file using FileOutputStream and it works slightly, but only saves 10KB of information,one iteration of the while loop, (seems to be overwriting), and if I set the FileOutputStream(file, true) to append the information, it works... once, then the file is just added on to the next time I run it, which isn't what I want. How do I write to the same file with multiple chunks of bytes through each iteration, but still overwrite the file in completeness if I run the program again?

byte bytes[] = new byte[(10*1024)];
            while (dis.read(bytes) > 0)
            {
                //Set all the bytes to the message
                byte message[] = bytes;
                String string = new String(bytes, "UTF-8");

                //Does bytes contain header?
                if (string.contains("\r\n\r\n")){
                    String theByteString[] = string.split("\r\n\r\n");
                    String theHeader = theByteString[0];
                    String[] lmTemp = theHeader.split("Last-Modified: ");
                    String[] lm = lmTemp[1].split("\r\n");
                    String lastModified = lm[0];
                    //Cut off the header and save the rest of the data after it
                    message = theByteString[1].getBytes("UTF-8");

                    //cache
                    hm.put(url, lastModified);
                }

                //Output message[] to file.
                File f = new File(hostName + path);
                f.getParentFile().mkdirs(); 
                f.createNewFile();
                try (FileOutputStream fos = new FileOutputStream(f)) {
                    fos.write(message);
                } catch (IOException ioe) {
                    ioe.printStackTrace();
                }

            }
        }

Jon Skeet · Answer 1 · 2016-10-14T21:09:06.573

You're opening a new FileOutputStream on each iteration of the loop. Don't do that. Open it outside the loop, then loop and write as you are doing, then close at the end of the loop. (If you use a try-with-resources statement with your while loop inside it, that'll be fine.)

That's only part of the problem though - you're also doing everything else on each iteration of the loop, including checking for headers. That's going to be a real problem if the byte array you read contains part of the set of headers, or indeed part of the header separator.

Additionally, as noted by EJP, you're ignoring the return value of read apart from using it to tell whether or not you're done. You should always use the return value of read to know how much of the byte array is actually usable data.

Fundamentally, you either need to read the whole response into a byte array to start with - which is easy to do, but potentially inefficient in memory - or accept the fact that you're dealing with a stream, and write more complex code to detect the end of the headers.

Better though, IMO, would be to use an HTTP library which already understands all this header processing, so that you don't need to do it yourself. Unless you're writing a low-level HTTP library yourself, you shouldn't be dealing with low-level HTTP details, you should rely on a good library.

Updated with your suggestions. I'm just waiting to hear back from EJP on how to set the string. — john stamos, Oct 15 '16 at 00:42

score 1 · Answer 2 · answered Oct 14 '16 at 21:04

1

Open the file ahead of the loop.

NB you need to store the result of read() in a variable, and pass that variable to new String() as the length. Otherwise you are converting junk in the buffer beyond what was actually read.

answered Oct 14 '16 at 21:04

user207421

305,947
44
307
483

To my new string of bytes? How would I set the string length? – john stamos Oct 15 '16 at 00:27
@johnstamos: You'd call the string constructor taking it: `new String(bytes, 0, length, StandardCharsets.UTF_8)`. – Jon Skeet Oct 15 '16 at 05:02
It is only part of gthe answer and not very usefull, since the processing of headers on every read from inputStream also could result in junk being written to file. – Krzysztof Cichocki Oct 15 '16 at 12:12

score 0 · Answer 3 · edited May 23 '17 at 12:16

0

There is an issue with reading the data - you read only part of the response (because at that moment not all data was transfered to you yet) - so obviusly you write only that part.

check this answer for how to read full data from the InputStream:

Convert InputStream to byte array in Java

edited May 23 '17 at 12:16

Community

1
1

answered Oct 14 '16 at 20:57

Krzysztof Cichocki

6,294
1
16
32

`while (dis.read(bytes) > 0)` looks like it's reading all the data to me. It's just reading it in several chunks, and reopening the file for output on each iteration. – Jon Skeet Oct 14 '16 at 20:59
But looking again, there is more to it, I agree. – Jon Skeet Oct 14 '16 at 21:00
@Jon Skeet ok :) sory for the -1, but as you removed your post I can't undo it, I'll upvote next time we met here if you write something usefull - I'am sure you'll do :). Cheers! – Krzysztof Cichocki Oct 14 '16 at 21:02
I deleted it to give a more fuller response. Going to undelete soon. (Done now...) – Jon Skeet Oct 14 '16 at 21:03

Byte array to file in java without overwriting,

3 Answers3