How to get a byte array from FileInputStream without OutOfMemory error

Question

I have a FileInputStream which has 200MB of data. I have to retrieve the bytes from the input stream.

I'm using the below code to convert InputStream into byte array.

private byte[] convertStreamToByteArray(InputStream inputStream) {
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    try {
        int i;
        while ((i = inputStream.read()) > 0) {
            bos.write(i);
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
    return bos.toByteArray();
}

I'm getting OutOfMemory exception while coverting such a large data to a byte array.

Kindly let me know any possible solutions to convert InputStream to byte array.

Why would you need to load 200 MBs in RAM? Any android cell using this app would collapse. — Luiggi Mendoza, Apr 05 '13 at 04:22
The out of memory error is because you are trying to hold everything in memory at once. Changing the _way_ you are trying to convert it to a byte array isn't going to help; the problem is the idea of converting to a byte array. — Ted Hopp, Apr 05 '13 at 04:24
Actually there is a scenario where I need to send large data to the server. I'm having a solution to handle that part. Kindly let me know if you have any solutions for the above query. Thanks. — macOsX, Apr 05 '13 at 04:26
You can read a chunk of the file, send it to server and repeat the process until there is nothing more to read. Also, when you read an `InputStream`, you should use a `byte[1024*X] buffer`, where X should have a value of 1, 2, 4 or 8. AFAIK 1024*4 is one of the fastest. — Luiggi Mendoza, Apr 05 '13 at 04:26
If you need to send a large file to the server (and your customers are just going to love the data charges for transmitting 200MB!), just write it to the server connection on the fly as you are reading it. — Ted Hopp, Apr 05 '13 at 04:27
@Ted Hopp: Thanks for your suggestion. Is there any way to write such a huge data into a file without converting it to a byte array? — macOsX, Apr 05 '13 at 04:31
@thenna.mail for your question for TedHopp, no, there's no way to do that. It would be better if you post your functional requirement along with your try in order to get better solution proposals. — Luiggi Mendoza, Apr 05 '13 at 04:33
Why the heck would you need to upload 200Mb data (file) from a device to a Server. Is it a video file or something? If your end user is on slower network, how long will it take for this to complete? — midhunhk, Apr 05 '13 at 04:43
please try [MyAnswer][1] to read and write data from file and cache memory. [1]: http://stackoverflow.com/questions/9451572/saving-a-json-file-to-internal-memory/10226083#10226083 — Anand Phadke, Apr 05 '13 at 04:48
Take a look at [the answer by Ramsay Domloge](http://stackoverflow.com/a/15831636/535871). It shows how to write an input stream to an output stream without first converting it to a byte array. If your output stream is the socket to the server, then the file won't have to be loaded all at once into memory. — Ted Hopp, Apr 05 '13 at 14:25

score 10 · Answer 1 · answered Apr 05 '13 at 10:31

10

Why do you want to hold the 200MB file in memory? What are you going to to with the byte array?

If you are going to write it to an OutputStream, get the OutputStream ready first, then read the InputStream a chunk at a time, writing the chunk to the OutputStream as you go. You'll never store more than the chunk in memory.

eg:

     public static void pipe(InputStream is, OutputStream os) throws IOException {

        int read = -1;
        byte[] buf = new byte[1024];

        try {
            while( (read = is.read(buf)) != -1) {
                os.write(buf, 0, read);
            }
        }
        finally {
            is.close();
            os.close();
        }
    }

This code will take two streams and pipe one to the other.

answered Apr 05 '13 at 10:31

Ramsay Domloge

695
3
11

nice code, but you are still saving all the 200MB in the memory. you should add os.flush() call inside the while loop. – eladyanai Jul 27 '15 at 08:38
"you are still saving all the 200MB in the memory". Not strictly true - that depends on the underlying implementation of OutputStream. ByteArrayOutputStream will certainly buffer it all in memory (and calling flush() will do nothing), but FileOutputStream will manage its own internal buffering and should be trusted to do as it sees fit. Unnecessarily calling flush() would be second-guessing the implementation and might break any performance gains from internal buffering. – Ramsay Domloge Jul 28 '15 at 10:34

score 2 · Answer 2 · answered Apr 05 '13 at 04:37

Android application has limited Heap Memory and which depend on devices. Currently most of the new devices has 64 but it could be more or less depend on Manufacturer. I have seen device come come with 128 MB heap Memory.

So what this really mean?

Its simply means that regardless of available physical memory your application is not allowed to grow more then allocated heap size.

From Android API level 11 you can request for additional memory by using manifest tag android:largeHeap="true" which will be double your heap size. That simply means if your devices has 64 you will get 128 and in case of 128 you will get 256. But this will not work for lower API version.

I am not exactly sure what is your requirement, but if you planning to send over HTTP then read file send data and read again. You can follow the same procedure for file IO also. Just to make sure not to use memory more then available heap size. Just to be extra cautious make sure you leave some room for application execution.

score 1 · Answer 3 · answered Apr 05 '13 at 04:37

1

Your problem is not about how to convert InputStream to byte array but that the array is to big to fit in memory. You don't have much choice but to find a way to process bytes from InputStream in smaller blocks.

answered Apr 05 '13 at 04:37

Evgeniy Dorofeev

133,369
30
199
275

score 0 · Answer 4 · answered Apr 05 '13 at 04:28

0

You'll probably need to massively increase the heap size. Try running your Java virtual machine with the -Xms384m -Xmx384m flag (which specifies a starting and maximum heap size of 384 megabytes, unless I'm wrong). See this for an old version of the available options: depending on the specific virtual machine and platform you may need to do some digging around, but -Xms and -Xmx should get you over that hump.

Now, you probably really SHOULDN'T read it into a byte array, but if that's your application, then...

answered Apr 05 '13 at 04:28

Femi

64,273
8
118
148

Do you have an android cell? Do you know how much available RAM it has after turning it on? – Luiggi Mendoza Apr 05 '13 at 04:28
This is Android, which runs the Dalvik VM, not a JVM. The user doesn't get to set startup options like heap size. – Ted Hopp Apr 05 '13 at 04:29
Ah..missed the tag: my mistake. Yeah, that's never going to work for him: Android apps generally have heap sizes limited to less than 64 MB, even at the high end. `android:largeHeap="true"` might get him partway there on very very recent devices (as covered http://stackoverflow.com/questions/5350465/android-heap-size-on-different-phones-devices-and-os-versions) but in general that's just really not possible. – Femi Apr 05 '13 at 04:32

score -2 · Answer 5 · answered Apr 05 '13 at 05:00

-2

try this code

private byte[] convertStreamToByteArray(InputStream inputStream) {
ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream();
                    int readByte = 0;
        byte[] buffer = new byte[2024];

                    while(true)
                    {
                        readByte = inputStream.read(buffer);
                        if(readByte == -1)
                        {
                            break;
                        }
                        byteOutStream.write(buffer);
                    }
                    inputStream.close();
                    byteOutStream.flush();
                    byteOutStream.close();
                    byte[] byteArray= byteOutStream.toByteArray();
                    return byteArray;
}

try to read chunk of data from InputStream .

answered Apr 05 '13 at 05:00

Anand Phadke

531
3
28

2

This is going to be absolutely no help with the out-of-memory error. If anything, it makes the problem a tiny bit worse by using a 2K buffer. – Ted Hopp Apr 05 '13 at 05:54
-1 for OuyOfMemory exception in your code. your code should include 'byteOutStream.flush();' inside the while loop or you want to load 200MB to the heap – eladyanai Jul 27 '15 at 08:37
There is absolutely no need to call flush(). As [user_half] says, this will actually be slightly worse. On top of that, you are also writing the *complete* buffer to the output stream, even if it's only partially filled. So you are going to *corrupt* the stream. You should call byteOutStream.write(buffer, 0, readByte); Lastly, readByte isn't actually a byte, it is an int that represents the number of bytes read, so most people call it 'read'. Your name suggests that it is the byte that was read, rather than the count. – Ramsay Domloge Oct 27 '16 at 22:49

How to get a byte array from FileInputStream without OutOfMemory error

5 Answers5