Android httpclient file upload data corruption and timeout issues

Question

i'm having issues with uploading images in android.

i'm using apache httpmime 4.1 lib the code is like this:

MultipartEntity reqEntity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);

reqEntity.addPart("image", new FileBody(new File(AndorraApplication.getPhotosPath() + "/" + entity.getFileName()), "image/jpeg"));
resp = NetworkUtils.sendHttpRequestMultipart(EXPORT_PHOTOS_URI, reqEntity);

NetworkUtils class:

public class NetworkUtils {
    public static final int REGISTRATION_TIMEOUT = 3 * 1000; 
    public static final int WAIT_TIMEOUT = 5 * 1000;

    public static HttpResponse sendHttpRequestMultipart(String uri, MultipartEntity entity) {
        HttpClient mHttpClient = new DefaultHttpClient();
        final HttpParams params = mHttpClient.getParams();
        HttpConnectionParams.setConnectionTimeout(params, REGISTRATION_TIMEOUT);
        HttpConnectionParams.setSoTimeout(params, WAIT_TIMEOUT);
        ConnManagerParams.setTimeout(params, WAIT_TIMEOUT);

        HttpPost post = new HttpPost(uri);
        post.addHeader(entity.getContentType());
        post.setEntity(entity);
        HttpResponse resp = mHttpClient.execute(post);
    }
}

sometimes everything works fine, but sometimes (especially on a slow connection) the image is uploaded very corrupted. the example is here: http://pixelbirthcloud.com/574_orig.jpg

it doesn't throw any exceptions. the lenght of the uploaded file is same as the original one.. tried to change mime type to application/octet-stream or remove it at all. tries playing with timeouts. still same result. end users upload corrupted images almost all the time (although i managed to get bronem images just 2 times only).. the image size was 2.5 megs at first, but then i reduced it to 500-700 kb. didn't fix the problem though.

haven't tried to change the apache's library.. maybe it is the problem.. but as far as i read the net, nobody was experiencing this with httpmime library.

what can it be? i'm completely lost now :(

the other issue is that timeouts sometimes not working.

like when it comes to this line: HttpResponse resp = mHttpClient.execute(post); and i disable 3g connection it just waits for like 17-20 minutes instead of 3 or 5 seconds.. and only then throws exception. tried different methods. like this:

        HttpParams params = new BasicHttpParams();
        HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
        HttpProtocolParams.setContentCharset(params, HTTP.UTF_8);
        HttpProtocolParams.setUseExpectContinue(params, false);  
        HttpConnectionParams.setConnectionTimeout(params, 10000);
        HttpConnectionParams.setSoTimeout(params, 10000);
        ConnManagerParams.setMaxTotalConnections(params, 5);
        ConnManagerParams.setTimeout(params, 30000);

        SchemeRegistry registry = new SchemeRegistry();
        registry.register(new Scheme("http",PlainSocketFactory.getSocketFactory(), 80));
        registry.register(new Scheme("https",PlainSocketFactory.getSocketFactory(), 80));
        ThreadSafeClientConnManager manager = new ThreadSafeClientConnManager(params, registry);
        HttpClient httpclient = new DefaultHttpClient(manager, params);

but still doesn't work :)

Answer 1

See My Code of Image Uploader and it worked great for me This class Uploads a file to the server plus in the end read the XML reply also. Filter the code as per your requirement.. It worked pretty smooth for me

package com.classifieds;

import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;

import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;

import javax.xml.parsers.ParserConfigurationException;
import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;

import org.xml.sax.Attributes;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.DefaultHandler;


import android.util.Log;

public class Uploader 
{

    private String Tag = "UPLOADER";
    private String urlString ;//= "YOUR_ONLINE_PHP";
    HttpURLConnection conn;
    String exsistingFileName ;

    private void uploadImageData(String existingFileName , String urlString)
    {
        String lineEnd = "\r\n";
        String twoHyphens = "--";
        String boundary = "*****";
        try {
            // ------------------ CLIENT REQUEST

            Log.e(Tag, "Inside second Method");

            FileInputStream fileInputStream = new FileInputStream(new File(
                    exsistingFileName));

            // open a URL connection to the Servlet

            URL url = new URL(urlString);

            // Open a HTTP connection to the URL

            conn = (HttpURLConnection) url.openConnection();

            // Allow Inputs
            conn.setDoInput(true);

            // Allow Outputs
            conn.setDoOutput(true);

            // Don't use a cached copy.
            conn.setUseCaches(false);

            // Use a post method.
            conn.setRequestMethod("POST");

            conn.setRequestProperty("Connection", "Keep-Alive");

            conn.setRequestProperty("Content-Type",
                    "multipart/form-data;boundary=" + boundary);

            DataOutputStream dos = new DataOutputStream(conn.getOutputStream());

            dos.writeBytes(twoHyphens + boundary + lineEnd);
            dos
                    .writeBytes("Content-Disposition: post-data; name=uploadedfile;filename="
                            + exsistingFileName + "" + lineEnd);
            dos.writeBytes(lineEnd);

            Log.v(Tag, "Headers are written");

            // create a buffer of maximum size

            int bytesAvailable = fileInputStream.available();
            int maxBufferSize = 1000;
            // int bufferSize = Math.min(bytesAvailable, maxBufferSize);
            byte[] buffer = new byte[bytesAvailable];

            // read file and write it into form...

            int bytesRead = fileInputStream.read(buffer, 0, bytesAvailable);

            while (bytesRead > 0) {
                dos.write(buffer, 0, bytesAvailable);
                bytesAvailable = fileInputStream.available();
                bytesAvailable = Math.min(bytesAvailable, maxBufferSize);
                bytesRead = fileInputStream.read(buffer, 0, bytesAvailable);
            }

            // send multipart form data necesssary after file data...

            dos.writeBytes(lineEnd);
            dos.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);

            // close streams
            Log.v(Tag, "File is written");
            fileInputStream.close();
            dos.flush();
            dos.close();

        } catch (MalformedURLException ex) {
            Log.e(Tag, "error: " + ex.getMessage(), ex);
        }

        catch (IOException ioe) {
            Log.e(Tag, "error: " + ioe.getMessage(), ioe);
        }


        SAXParserFactory spf = SAXParserFactory.newInstance();
        SAXParser sp = null;
        try {
            sp = spf.newSAXParser();
        } catch (ParserConfigurationException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (SAXException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        // Get the XMLReader of the SAXParser we created.
        XMLReader xr = null;
        try {
            xr = sp.getXMLReader();
        } catch (SAXException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        // Create a new ContentHandler and apply it to the XML-Reader
        MyExampleHandler1 myExampleHandler = new MyExampleHandler1();
        xr.setContentHandler(myExampleHandler);

        // Parse the xml-data from our URL. 
        try {
            xr.parse(new InputSource(conn.getInputStream()));
        //xr.parse(new InputSource(new java.io.FileInputStream(new java.io.File("login.xml")))); 
        } catch (MalformedURLException e) {
            Log.d("Net Disconnected", "NetDisconeeted");
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            Log.d("Net Disconnected", "NetDisconeeted");
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (SAXException e) {
            Log.d("Net Disconnected", "NetDisconeeted");
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        // Parsing has finished.

    }

    public Uploader(String existingFileName, boolean isImageUploading , String urlString ) {

        this.exsistingFileName = existingFileName;
        this.urlString = urlString;

    }

    class MyExampleHandler1 extends DefaultHandler
    {
    // ===========================================================
    // Methods
    // ===========================================================

    @Override
    public void startDocument() throws SAXException {

    }

    @Override
    public void endDocument() throws SAXException {
        // Nothing to do
    }

    @Override
    public void startElement(String namespaceURI, String localName,
              String qName, Attributes atts) throws SAXException {


    }

    /** Gets be called on closing tags like:
     * </tag> */
    @Override
    public void endElement(String namespaceURI, String localName, String qName)
              throws SAXException {


    }

    /** Gets be called on the following structure:
     * <tag>characters</tag> */
    @Override
    public void characters(char ch[], int start, int length) {

     }
    }

}

Answer 2

I had the same corruption problem on 80% of my uploaded files. Emulator didn't fail to upload though. Corrupted files were 1k larger that original ones. Then I set the buffer of the output stream to 1 byte and it began to work with no problem. Finally I let it be 8 bytes and haven't had no more corruption problems. A buffer of about 80 or 50, I can't remember, also failed. Don't understand what the problem is but I'm happy it is working this way. This page was so inspiring thanks.

Answer 3

ok. spent 2 days on testing this issue and found out following:
when using tcpdump on android it comes that the data wasn't corrupt, BUT the tcp packet size was 1516, which is very strange, cause the normal ethernet packet size is 1500 and everything more than 1516 is way too big.
i manually changed the MTU to 576 (i believe it is the standard for ppp, which 3G actually is) and it works perfect! 150 of 150 images were uploaded normally!

this doesn't solve the actual problem, though, cause it's impossible to change mtu on non-rooted devices i believe, and you have to change it every time you reboot the device (or every time you bring up the interface - not sure about it, cause couldn't find the way of getting MTU value via ifconfig). but at least i know where the problem is.
setting http chunk size to a smaller value (tried 300 bytes) didn't affect it (i believe it's because the http headers are way too big themselves)... so... so nothing =)

will try to post it to android-developer group on google, but their moderation is way too slow... we'll see...