Calculating size of Post Content-Length for Base64 encoded file

Question

I am trying to upload some large files from an Android device to a .Net Web Service. This web service has been set up so that it accepts these files as a POST parameter and that the files have to be sent as a Base64 encoded string.

I have been able to use this library by Christian d'Heureuse to convert the file to a Base64 String, calculate the size of the string in bytes and send it up previously, however the method I used before involved loading the whole file into memory which was causing out of memory errors when dealing with large files, which was not unexpected.

I have been trying to convert the file into Base64 in chunks and stream this data through the connection (using the data output stream object) as it is being converted, so the whole file does not need to be loaded into memory in one go, however I can't seem to accurately work out the size of the Content-Length for the request before converting the file - I usually seem to be about 10 bytes out - frustratingly, it does occasionally work!

I have also found that some of the time when this does work the server returns the following error message "Invalid size for a Base64 char array". I believe this to be an issue with padding characters, however I can't see a problem with my code works this out, some advice on this issue would be much appreciated!

This is the code the generates the request and streams the data:

    try
{


    HttpURLConnection connection = null;

    DataOutputStream outputStream = null;

    DataInputStream inputStream = null;

    //This is the path to the file
    String pathToOurFile = Environment
            .getExternalStorageDirectory().getPath()
            + "/path/to/the/file.zip";

    String urlServer = "https://www.someserver.com/somewebservice/";

    int bytesRead, bytesAvailable, bufferSize;

    byte[] buffer;

    int maxBufferSize = 456;


    //The parameters of the POST request - File Data is the file in question as a Base64 String
    String params = "Username=foo&Password=bar&FileData=";

    File sizeCheck = new File(pathToOurFile);

    Integer zipSize = (int) sizeCheck.length();

    Integer paddingRequired = ((zipSize * 8) / 6) % 3;

    Integer base64ZipSize = ((zipSize * 8) / 6)
            + ((zipSize * 8) / 6) % 3;

    Integer paramLength = params.getBytes().length;

    //Code to work out the number of lines required, assuming we create a new
    //line every 76 characters - this is used t work out the number of
    //extra bytes required for new line characters
    Integer numberOfLines = base64ZipSize / 76;

    Log.i(TAG, "numberOfLines: " + numberOfLines);

    Integer newLineLength = System.getProperty("line.separator")
            .getBytes().length;

    //This works out the total length of the Content
    Integer totalLength = paramLength + base64ZipSize
            + (numberOfLines * newLineLength) + paddingRequired;

    Log.i(TAG, "total Length: " + totalLength);

    FileInputStream fileInputStream = new FileInputStream(new File(
            pathToOurFile));

    URL url = new URL(urlServer);

    connection = (HttpURLConnection) url.openConnection();

    connection.setDoInput(true);

    connection.setDoOutput(true);

    connection.setUseCaches(false);

    connection.setRequestMethod("POST");

    connection.setRequestProperty("Connection", "Keep-Alive");

    connection.setRequestProperty("Content-Type",
            "application/x-www-form-urlencoded;");

    connection.setRequestProperty("Content-Length", ""
            + totalLength); // number of bytes

    outputStream = new DataOutputStream(
            connection.getOutputStream());

    //Write out the parameters to the data output stream
    outputStream.writeBytes(params);

    bytesAvailable = fileInputStream.available();

    bufferSize = Math.min(bytesAvailable, maxBufferSize);

    buffer = new byte[bufferSize];

    bytesRead = fileInputStream.read(buffer, 0, bufferSize);

    Integer totalSent = paramLength;

    Integer enLen = 0;


    //Convert the file to Base64 and Stream the result to the
    //Data output stream
    while (bytesRead > 0)
    {
        String convetedBase64 = Base64Coder.encodeLines(buffer);
        convetedBase64 = convetedBase64.replace("=", "");

        if (totalSent >= (totalLength - 616))
        {
            Log.i(TAG, "about to send last chunk of data");
            convetedBase64 = convetedBase64.substring(0,
                    convetedBase64.length() - 1);
        }

        Log.i(TAG,
                "next data chunk to send: "
                        + convetedBase64.getBytes().length);
        Log.i(TAG, "'" + convetedBase64 + "'");

        enLen = enLen + convetedBase64.length();
        outputStream.writeBytes(convetedBase64);

        totalSent = totalSent + convetedBase64.getBytes().length;

        Log.i(TAG, "total sent " + totalSent);
        Log.i(TAG, "actual size: " + outputStream.size());

        bytesAvailable = fileInputStream.available();
        bufferSize = Math.min(bytesAvailable, maxBufferSize);
        buffer = new byte[bufferSize];
        bytesRead = fileInputStream.read(buffer, 0, bufferSize); // read
                                                                    // into
                                                                    // the
                                                                    // buffer
    }

    Log.i(TAG, "enLen: " + enLen);
    Log.i(TAG, "paddingRequired: " + paddingRequired);

    for (int x = 0; x < paddingRequired; x++)
    {
        outputStream.writeBytes("=");
    }

    InputStream is2 = connection.getInputStream();
    String output = IOUtils.toString(is2);
    Log.i(TAG, "Got server response: " + output);
    fileInputStream.close();
    outputStream.flush();
    outputStream.close();
}
catch (Exception ex)
{
    Log.e(TAG, "caught an exception:" + ex.getMessage());
}

I would be very appreciative if anyone could point out any errors in my code that could be causing this, or suggest a better way of converting and uploading the file.

Answer 1

Soooo... I did manage to find a few solutions to this problem, just in case anyone stumbles on this question I'll leave them here:

The first was to write the data out to a temporary file, so I could get the size after conversion in bytes - seemed like a good idea at first, however was inefficient and after discovering other methods seemed silly.

The other method is to not specify a content length (I didn't know you could do this!). Unfortunately Android still tried to allocate enough memory for the upload, which caused problems.

If you specify the connection to use ChunkedStreamingMode Android then plays nice and buffers the upload, saving on ram used (apart from a strange bug on 2.2).

The code for this is as such:

httppost.setDoInput(true);
httppost.setDoOutput(true);
httppost.setRequestMethod("POST");
httppost.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
httppost.setChunkedStreamingMode(0); //This specifies the size of the chunks - 0 uses the system default

DataOutputStream dos = new DataOutputStream(httppost.getOutputStream());
dos.writeBytes(content); //This code write out the rest of the post body first, for example username or password

try
{
    BufferedInputStream dis = new BufferedInputStream(new FileInputStream("path/to/some/file"), 2048);

    byte[] in = new byte[512];

    while (dis.read(in)) > 0)
    {
        dos.write(Base64.encode(in, Base64.URL_SAFE)); //This writes out the Base64 data
                                                       //I used the web safe base64 to save URL encoding it again
                                                       //However your server must support this
        dos.write(newLine); //this write out a newline character
    }

dos.flush();
dis.close();
dos.close();

I hope this helps!!!