How do I read big images in Java with limited heap space on AWS Lambda?

Question

Although this doesn't have anything to do with my problem, I would like to give some context.

My setup is as follows : An Amazon S3 bucket that stores images and another S3 bucket that stores the thumbnail of the images. The goal is whenever a new image is uploaded to the first bucket, a Lambda function is triggered, reads the image from the first bucket, creates a thumbnail and stores it in the second bucket. The issue that I'm facing is that this procedure works perfectly when the images are small in size but when the image size gets bigger, I run out of Java heap space when reading the image.

Note: I CANNOT increase the Java heap size of the AWS Lambda function, it is at the max permissible amount already.

The code I use is as follows:

        AmazonS3 s3Client = new AmazonS3Client();
        S3Object s3Object = s3Client.getObject(new GetObjectRequest(srcBucket, srcKey));
        InputStream objectData = s3Object.getObjectContent();

        System.out.println("Reading source image");
        BufferedImage srcImage = ImageIO.read(objectData);

The ImageIO.read() throws the following error when trying to read in a 200 MB image:

        Java heap space: java.lang.OutOfMemoryError java.lang.OutOfMemoryError: Java heap space 
        at java.awt.image.DataBufferByte.<init>(DataBufferByte.java:92) 
        at java.awt.image.ComponentSampleModel.createDataBuffer(ComponentSampleModel.java:445) 
        at java.awt.image.Raster.createWritableRaster(Raster.java:941) 
        at javax.imageio.ImageTypeSpecifier.createBufferedImage(ImageTypeSpecifier.java:1074) 
        at javax.imageio.ImageReader.getDestination(ImageReader.java:2892) 
        at com.sun.imageio.plugins.jpeg.JPEGImageReader.readInternal(JPEGImageReader.java:1071) 
        at com.sun.imageio.plugins.jpeg.JPEGImageReader.read(JPEGImageReader.java:1039) 
        at javax.imageio.ImageIO.read(ImageIO.java:1448) 
        at javax.imageio.ImageIO.read(ImageIO.java:1352)

Now, as I understand, using BufferedImage is bad since it uncompresses the image. So, my question is: Is there any other way to read the image other than ImageIO.read()? Or read in an image into some other format perhaps?

Another thought that crossed my mind was to read parts of image at a time by using rectangular dimensions, but I wouldn't know the dimensions of the complete image without reading the image so I wasn't sure how to go about this.

Any help is much appreciated!

This is a tricky problem. AWS Lambda does allow 512 MB of temporary local disk storage. You'd have to stream the image to disk, then find a way to incrementally resize it. I'm not sure there are any Java libraries out there to do this. Python may have some options with PIL. — aglassman, May 05 '16 at 21:26
@aglassman Thanks for your reply! Lambda is really restrictive isn't it? :( I've tried doing this in nodeJS and that was running out of memory too. I will take a look at the Python option as well if this doesn't work out — sidd, May 05 '16 at 21:38
Yeah it can be restrictive, lambda I think is more for mass data collection, and data transformation / manipulation. I wouldn't have thought to use it to process large images. There may be some better options out there. If you're using it specifically for its S3 triggers, you could maybe have it send a message to a beefier backend server that would then pull the object by ID or something. — aglassman, May 05 '16 at 21:45
@BrianKent Thanks for getting back! I think I am able to use ImageIO.createImageInputStream. Code I used was "ImageInputStream iis = ImageIO.createImageInputStream(objectData); System.out.println("10 bits = " + iis.readBits(10));" And the output was "10 bits = 1023" --> Just printing something to make sure it was able to read the original image. With that in mind, what would you suggest I do next? — sidd, May 05 '16 at 22:06
See http://stackoverflow.com/questions/5874593/loading-large-images-as-thumbnails-without-memory-issues-in-java — Brian Kent, May 06 '16 at 01:39

How do I read big images in Java with limited heap space on AWS Lambda?

0 Answers0