Android AWS s3 SDK download throws "Socket is closed" exception or terminates early

Question

I'm trying to download an object from S3 using the AWS Android SDK 1.0.4 or 1.0.3.

This is my code:

AmazonS3Client client = getConnection(userCredentials);
S3Object obj = client.getObject(workspaceName, objectName);
ObjectMetadata meta = obj.getObjectMetadata();
long size = meta.getContentLength();
logger.info("S3 object length: "+size);
InputStream is = new BufferedInputStream(obj.getObjectContent());
byte[] fragmentBytes = IOUtils.toByteArray(is);
logger.info("Read bytes: "+ fragmentBytes.length);

This sometimes, rarely, works. Most of the time either an "java.net.SocketException: Socket is closed" is thrown or no error is reported but only part of the object is read.

Note, that the BufferedInputStream should not be necessary for IOUtils.toByteArray(...) and it makes no difference if it is there or not.

The problem does not seem to occur when stepping through the code line by line in a debugger.

This happens on my Android 2.3.3 device and 3.2 device. It happens over WiFi and 3G.

Any ideas?

ps> The objects are about 250k big

score 5 · Accepted Answer · edited Feb 01 '13 at 07:48

5

The issue is that the client is being Garbage Collected. To fix it, you need to keep an explicit reference to the S3Client.

The flow of the bug is as follows:

The program gets an S3 client
The program gets a S3 object from the client.
The program gets an inputStream from the s3Object, which gets it through the client.
Garbage collection runs, and since there is no reference to the client, it gets Garbage Collected, randomly closing your inputStream.

IMO, this is a very bad bug by Amazon. Why the S3 object input stream does not have a reference back to the client is a mystery to me.

Here's the thread on the AWS forums: https://forums.aws.amazon.com/thread.jspa?threadID=83326

edited Feb 01 '13 at 07:48

Filip Spiridonov

34,332
4
27
30

answered Feb 01 '13 at 07:23

Lexi Viripaeff

66
1
1

From the thread linked above it is unclear if we need to have reference on appropriate s3Object as well. – om-nom-nom Sep 22 '13 at 22:33
It's been a while, but from memory I think it turned out in the end that exactly this was the problem. – Carsten Jun 07 '16 at 22:32

score 2 · Answer 2 · answered Nov 28 '12 at 18:41

You probably already solved this but I could not find the correct answer and ended up figuring it out by myself, so here goes the solution:

The getObjectContent methods documentation is very clear about closing the InputStream:

S3ObjectInputStream com.amazonaws.services.s3.model.S3Object.getObjectContent()

Gets an input stream containing the contents of this object. Callers should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3.

You are making the same mistake as I was of using the InputStream (S3Object) as if closure was transparent.

I am guessing you need to get an image from S3 to be used on Android, so here goes an example method that returns a Bitmap. It really applies to any kind of input.

private Bitmap getFromStore(String fileName) {
    AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY));

    S3Object o = client.getObject(BUCKET, fileName);
    InputStream is = o.getObjectContent();

    Bitmap image = null;

    try {   
        // Use the inputStream and close it after.
        image = BitmapFactory.decodeStream(is);
    } finally {
        try {
            is.close();
        } catch (IOException e) {
            Log.w(TAG, "Error trying to close image stream. ", e);
        }
    }

    return image;
}

Just in case, dont forget to execute this on a different thread then the UI.

Good luck,

Thanks, but I don't think this was the problem in my case. Looking at my code it properly closes the input stream and the error can occur even in the first download (before any stream should be closed). I have a workaround for my problem, but it is very crude: Catch the exception when it happens and restart downloading the file. Eventually it will succeed. — Carsten, Nov 30 '12 at 01:32

score 1 · Answer 3 · edited May 23 '17 at 12:13

1

Well, it seems you are not alone. The accepted in the post answer talks of your problem.

How to write an S3 object to a file?

I would try looping the content like the answer in the question above. Buffered is good, but....

You could also try not using the IOUtil, after all there are other options in java.

edited May 23 '17 at 12:13

Community

1
1

answered Dec 22 '11 at 05:17

Ted Johnson

4,315
3
29
31

Thanks, but the linked post is about a slightly different problem and its solution does not help in my case. IOUtil more or less do the same thing as the code in the answer. The only difference is the buffer size and the Thread Interrupt handling. Thread interrupt is not a problem in my case and the buffer size (1k vs 4k) should not make a difference. – Carsten Dec 22 '11 at 05:33

score 0 · Answer 4 · edited Jun 06 '16 at 18:51

no need to keep re-initializing s3.

In your onCreate, make the call to initialize s3Object and s3Client.

Then use the call in your asynctask. This way, your s3Client will keep the same data and never close the socket connection with s3 when doing the while read.

S3Client s3Client;
S3Object s3Object;

onCreate() {
     s3Client = new AmazonS3Client(new BasicSessionCredentials(
                  Constants.ACCESS_KEY_ID, Constants.SECRET_KEY, Constants.TOKEN
                ));
     object = new S3Object();
}

doinbackground() {
     object = s3Client.getObject(new GetObjectRequest(Constants.getBucket(), id +".png"));
}

/****** add this in your onCreate ******/
AmazonS3Client client = getConnection(userCredentials);

Android AWS s3 SDK download throws "Socket is closed" exception or terminates early

4 Answers4