0

I'm having problems downloading files with my company's application, it crashes when I download files over 50mb.

I'm getting the file as an array of native bytes from a WS and then I convert it into an array of Object Byte to serialize it before converting these bytes into the file to download.

This works perfectly with small files but when you are downloading files over 50mb the conversion from bytes[] to Byte[] fails.

This was the code I had:

private RespuestaDescargarDocumento rellenarRespuestaDescarga(byte[] contenido, String nombreDocumento){

        Byte[] bytes = ArrayUtils.toObject(contenido);

        RespuestaDescargarDocumento respuesta = new RespuestaDescargarDocumento();
        respuesta.setContenido(bytes);
        respuesta.setNombreDocumento(nombreDocumento);

        return respuesta;
    }

I take a look what toObject() does and I realised that it is doing a new Byte() every iteration of this loop so if I'm downloading a file of for example 52mb it does 53167398 iterations and for every iteration is creating a new Byte.

public static Byte[] toObject(byte[] array) {
        if (array == null) {
            return null;
        } else if (array.length == 0) {
            return EMPTY_BYTE_OBJECT_ARRAY;
        }
        final Byte[] result = new Byte[array.length];
        for (int i = 0; i < array.length; i++) {
            result[i] = new Byte(array[i]);
        }
        return result;
    }

I think this is ridiculous so I looked the way of doing this conversion without this method and found this thread:

how to convert byte[] to Byte[], and the other way around?

And I tried the best answer way like this:

private RespuestaDescargarDocumento rellenarRespuestaDescarga(byte[] contenido, String nombreDocumento) {

        //Byte[] bytes = ArrayUtils.toObject(contenido);
        Byte[] bytes = byteToByte(contenido);

        RespuestaDescargarDocumento respuesta = new RespuestaDescargarDocumento();
        respuesta.setContenido(bytes);
        respuesta.setNombreDocumento(nombreDocumento);

        return respuesta;
    }

    private Byte[] byteToByte(byte[] array) {
        if (array == null || array.length == 0) {
            return ArrayUtils.EMPTY_BYTE_OBJECT_ARRAY;
        }
        Byte[] bytes;
        bytes = new Byte[array.length];
        for (int i = 0; i < array.length; i++) {
            bytes[i] = array[i];
        }
        return bytes;
    }

And now I can download perfectly that file of 52mb so that worked, or that was I thought, when I tried to download a bigger file of 150mb it crashed again. This time the loop is doing 159551619 iterations.

I'm getting the following error:

java.lang.OutOfMemoryError: Java heap space

In this line:

bytes = new Byte[array.length];

So it doesn't even get the for loop of my function, is like Java can't create an array of this size (159551619). But I found that the maximum size for an array in Java is 2147483647, so it should still work with this.

Am I doing something wrong?

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
Ise92
  • 127
  • 1
  • 1
  • 12
  • 2
    You're running out of memory, either allocate more memory to your application, or don't try to retain all this data in memory. Also, why do you want to to use `Byte[]`? Why can't you just use `byte[]`? You normally should avoid Object wrappers of primitives: they consume a lot more memory (eg instead of a single byte per byte, you'll need 4-8 bytes (depending on 32 or 64 bit) for each byte in the array + the size of the `Byte` instance itself. – Mark Rotteveel Oct 31 '18 at 11:43
  • 2
    What do you get from converting byte[] to Byte[]? Why don't you write the bytes directly to the file as is? To avoid memory overflow it would be best to avoid storing the whole file in memory. Byte streams are better. Read one byte block and write this block to disk. Then read the next byte block and so on. – vanje Oct 31 '18 at 11:47
  • 1
    Your second attempt does exactly the same as the first. The `new Byte` is just not visible because it happens automatically due to auto boxing. Get rid of the conversion altogether it is useless, slow and memory intensive. – Henry Oct 31 '18 at 12:30
  • I don't really know the reason about this conversion because it's legacy code but you're right, this conversion is useless. I found that after that there's another conversion from Byte[] to byte[], so this makes this more stupid. I removed that conversion and used the byte[] array that I get from the WS to make de file directly and it's working well for any kind of files (included the one with 150mb). Thanks you very much. – Ise92 Oct 31 '18 at 15:51

0 Answers0