5

I have been trying to solve a memory issue in a Java program where we are loading an entire file into memory, base64 encoding it and then using it as a form parameter in a post request. This is cause OOME due to the extremely large file size.

I am working on a solution where I am able to stream the file through a base64 encoder, into the request body of an Http Post request. One of the common patterns I have noticed in all of the popular encoding libraries( Guava, java.util.Base64, android.util.Base64 and org.apache.batik.util ) is that if the library supports encoding with Streams, the Encoding is always done through an OutputStream and the Decoding is always done through an InputStream.

I am having trouble finding/determining the reasoning behind these decisions. Given that so many of these popular and well-written libraries align with this api design, I assume that there is a reason for this. It doesn't seem very difficult to adapt one of these decoders to become an InputStream or accept an InputStream, but I am wondering if there is a valid architectural reason these encoders are designed this way.

Why do common libraries do Base64 encoding through an OuputStream and Base64 decoding through an InputStream?

Examples to back up my claims:

java.util.Base64
 - Base64.Decoder.wrap(InputStream stream)
 - Base64.Encoder.wrap(OutputStream stream)

android.util.Base64
 - Base64InputStream  // An InputStream that does Base64 decoding on the data read through it.
 - Base64OutputStream // An OutputStream that does Base64 encoding

google.common.io.BaseEncoding
 - decodingStream(Reader reader)
 - encodingStream(Writer writer)

org.apache.batik.util
 - Base64DecodeStream implements InputStream
 - Base64EncodeStream implements OutputStream

M. Wallace
  • 53
  • 5
  • For the exact reason you already stated: "This is cause OOME due to the extremely large file size." If you stream the data then there is no need to keep all of it in memory at once, attempting to do so will lead to out of memory exceptions. In short, don't try and do it in RAM. – Elliott Frisch Feb 07 '20 at 21:36
  • Of course, with [HTTP POST request allowing binary data](https://www.w3schools.com/tags/ref_httpmethods.asp), you may wonder what the point of the base 64 encoding / decoding is anyway. It's an interesting question, I struggled with it too in my early days, but once you get it, you get it, riding a bicycle style. – Maarten Bodewes Feb 08 '20 at 01:20
  • I don't fully understand the question. You don't need that capability in your situation; you are just wrapping the `OutputStream` of the POST request with a wrapper, right? If you don't need it, why do you suppose many other people do? – erickson Feb 08 '20 at 01:50
  • Unfortunately, I have no control over the API at the other end which specifies the file must be base64 encoded as a parameter, using application/x-www-form-urlencoded content type. RE: wrapping the `OutputStream` of the POST request. For whatever reason, the very old http client abstraction we have in our project doesn't expose a way to wrap the `OutputStream`, but it _does_ supply a way to provide an `InputStream`. Trying to cater to this API was what let me to this question. @MaartenBodewes answer was exactly what I was looking for to confirm it is the incorrect approach. Thanks! – M. Wallace Feb 08 '20 at 06:58
  • Ah, now I get where you're coming from. I guess in that case you can use `PipedInputStream` to reverse the stream and still use an `OutputStream` and wrap the base 64 encoder around that. Note that this is presuming multiple threads (!) because it if ever blocks, you're kinda screwed. You may need to implement a more specific non-blocking `InputStream` for your application if you have to remain in a single thread. Of course the `PipedInputStream` **does** buffer. – Maarten Bodewes Feb 08 '20 at 10:27
  • 1
    Your old colleagues had an `InputStream` to a certain source and asked themselves, "now how can we send that over an HTTP connection?". It is a logical but wrong solution to use the given `InputStream` for that (because of the handling of blocking calls mainly). In Java 9, there is the (convenience) method [`transferTo`](https://docs.oracle.com/javase/9/docs/api/java/io/InputStream.html#transferTo-java.io.OutputStream-). It is the logical counterpart to `PipedInputStream` which takes out the burden of buffering / looping from the programmer to connect the two. – Maarten Bodewes Feb 08 '20 at 11:39
  • Thanks for the background; that makes more sense. – erickson Feb 10 '20 at 16:36

1 Answers1

5

Well, yes, you can reverse it, but this makes the most sense. Base64 is used to make binary data - generated or operated on by the application - compatible with a text based outside environment. So the base 64 encoded data is always required on the outside and the decoded binary data is required on the inside.

An application generally doesn't perform any operations on the base 64 encoded data itself; it is just needed to communicate binary data with another application when a text interface is required or expected.


If you want to export your binary data to the outside, naturally you would use an output stream. If that data needs to be encoded in base 64, you make sure you send the data to an output stream that encodes to base 64.

If you want to import your binary data from the outside then you would use an input stream. If that data is encoded in base 64 then you first need to decode it, so you make sure you decode it before treating it as a binary stream.


Lets create a bit of a picture. Say you have an application that operates in a textual oriented environment but operates on binary data. The important part is the direction of the arrows from the context of the application on the left.

Then you get for the input (read calls):

{APPLICATION} <- (binary data decoding) <- (base64 decoding) <- (file input stream) <- [BASE 64 ENCODED FILE]

for this you naturally use input streams.

So let's look at the output (write calls):

{APPLICATION} -> (binary data encoding) -> (base64 encoding) -> (file output stream) -> [BASE 64 ENCODED FILE]

for this you naturally use output streams.

These stream can be connected to each other by chaining them together, i.e. using one stream as parent of the other stream.


Here is an example in Java. Note that creating the binary encoder/decoder in the data class itself is a bit ugly; generally you would use another class for that - I hope it suffices for demonstration purposes.

import static java.nio.charset.StandardCharsets.UTF_8;

import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Base64;

public class BinaryHandlingApplication {

    /**
     * A data class that encodes to binary output, e.g. to interact with an application in another language.
     * 
     * Binary format: [32 bit int element string size][UTF-8 element string][32 bit element count]
     * The integers are signed, big endian values.
     * The UTF-8 string should not contain a BOM.
     * Note that this class doesn't know anything about files or base 64 encoding.
     */
    public static class DataClass {
        private String element;
        private int elementCount;

        public DataClass(String element) {
            this.element = element;
            this.elementCount = 1;
        }

        public String getElement() {
            return element;
        }

        public void setElementCount(int count) {
            this.elementCount = count;
        }

        public int getElementCount() {
            return elementCount;
        }

        public String toString() {
            return String.format("%s count is %d", element, elementCount);
        }

        public void save(OutputStream out) throws IOException {

            DataOutputStream dataOutputStream = new DataOutputStream(out);

            // so here we have a chain of:
            // a dataoutputstream on a base 64 encoding stream on a fileoutputstream 


            byte[] utf8EncodedString = element.getBytes(UTF_8);
            dataOutputStream.writeInt(utf8EncodedString.length);
            dataOutputStream.write(utf8EncodedString);

            dataOutputStream.writeInt(elementCount);
        }

        public void load(InputStream in) throws IOException {
            DataInputStream dataInputStream = new DataInputStream(in);

            // so here we have a chain of:
            // a datainputstream on a base 64 decoding stream on a fileinputstream 

            int utf8EncodedStringSize = dataInputStream.readInt();
            byte[] utf8EncodedString = new byte[utf8EncodedStringSize];
            dataInputStream.readFully(utf8EncodedString);
            this.element = new String(utf8EncodedString, UTF_8);

            this.elementCount = dataInputStream.readInt();
        }

    }

    /**
     * Create the a base 64 output stream to a file; the file is the text oriented
     * environment.
     */
    private static OutputStream createBase64OutputStreamToFile(String filename) throws FileNotFoundException {
        FileOutputStream textOutputStream = new FileOutputStream(filename);
        return Base64.getUrlEncoder().wrap(textOutputStream);
    }

    /**
     * Create the a base 64 input stream from a file; the file is the text oriented
     * environment.
     */
    private static InputStream createBase64InputStreamFromFile(String filename) throws FileNotFoundException {
        FileInputStream textInputStream = new FileInputStream(filename);
        return Base64.getUrlDecoder().wrap(textInputStream);
    }

    public static void main(String[] args) throws IOException {
        // this text file acts as the text oriented environment for which we need to encode
        String filename = "apples.txt";

        // create the initial class
        DataClass instance = new DataClass("them apples");
        System.out.println(instance);

        // perform some operation on the data
        int newElementCount = instance.getElementCount() + 2;
        instance.setElementCount(newElementCount);

        // write it away
        try (OutputStream out = createBase64OutputStreamToFile(filename)) {
            instance.save(out);
        }

        // read it into another instance, who cares
        DataClass changedInstance = new DataClass("Uh yeah, forgot no-parameter constructor");
        try (InputStream in = createBase64InputStreamFromFile(filename)) {
            changedInstance.load(in);
        }
        System.out.println(changedInstance);
    }
}

Especially note the chaining of the streams and of course the absence of any buffers whatsoever. I've used URL-safe base 64 (in case you want to use HTTP GET instead).


In your case, of course, you could generate a HTTP POST request using an URL and directly encode to the retrieved OutputStream stream by wrapping it. That way no base 64 encoded data needs to be (extensively) buffered. See examples on how to get to the OutputStream here.

Remember, if you need to buffer, you're doing it wrong.

As mentioned in the comments, HTTP POST doesn't need base 64 encoding but whatever, now you know how you can encode base 64 directly to a HTTP connection.


java.util.Base64 specific note: Although base 64 is text, the base64 stream generates / consumes bytes; it simply assumes ASCII encoding (this can be fun for UTF-16 text). Personally I think this is a terrible design decision; they should have wrapped a Reader and Writer instead, even if that slows down encoding slightly.

To their defense, the various base 64 standards and RFC also get this wrong.

Maarten Bodewes
  • 90,524
  • 13
  • 150
  • 263
  • The sentence 'However, why wouldn't you use an `OutputStream` for output and an `InputStream` for input?' would be improved with a reason as to _why_ those are true (being input vs. output) in the contexts of decoding and encoding. I know an attempt is made at this afterwards in one direction, but it comes across a rewording ('encode' and 'retrieving') rather than justification. (I agree entirely with the sentiment of this answer for the record.) – BeUndead Feb 07 '20 at 22:19
  • This answer feels unsatisfying to me. Why is encoding considered to be output and decoding considered to be input? What would the fallback/downside be of providing something like a `Base64EncodingFilterInputStream` for example. – M. Wallace Feb 07 '20 at 22:46
  • 1
    Provided example. The downside of such an encoding input stream is that you would end up with base 64 *inside your application*. Why would you need that? You cannot use it for chaining streams as shown in the demo. – Maarten Bodewes Feb 08 '20 at 01:03
  • @MaartenBodewes: Thanks for updating (and going above and beyond). Reads much better now. Upped :). – BeUndead Feb 08 '20 at 04:13
  • @MaartenBodewes Wonderful response that really resonates with me, thank you. The advice to not use buffering has me wondering what you mean, sorry this has been my first year with Java after DotNet for almost 20. Currently getting burned because of my i/o stream nativity in a P8 project that makes heavy use of file/binary i/o. I thought buffering was a good thing, at least with regards to http and to Java i/o Streams? – Stephen Patten Sep 25 '20 at 13:12
  • Nowadays the OS will do the buffering anyway. Only use a buffered stream if the performance is lacklustre, but note that it might not help much. Buffering before writing to a stream makes no sense either, generally the TCP protocol will do the buffering (the stream needs to be separated into IP packets after all). – Maarten Bodewes Sep 26 '20 at 15:26