1

I wrote a java program, where I am uploading a 1GB text file into S3 bucket(uploading using Lambda from one S3 bucket to another). But I am getting "java.lang.OutOfMemoryError: Java heap space" . I have changed the heap memory to 2048 mb but still it's not working .

Here's the code :

public void s3Upload() throws UnsupportedEncodingException {
        try{
        final AmazonS3 s3Client =AmazonS3ClientBuilder.standard().
                                 withRegion(Regions.US_EAST_1).build();

        System.out.println(" Creating file transfer ");

        final S3Object s3Object = s3Client.getObject("my01bucket", "S3text.txt");
        final S3ObjectInputStream s3ObjectInputStream = s3Object.getObjectContent();

        final long contentLength = s3ObjectInputStream.toString().length();

        final ObjectMetadata objectMetadata = new ObjectMetadata();
        objectMetadata.setContentType("text/plain ; charset=utf-8");
        objectMetadata.setContentLength(contentLength);

        System.out.println("Uploading file....");

        final TransferManager tm = TransferManagerBuilder.standard().withS3Client(s3Client).
                                   withMultipartUploadThreshold((long)5*1024*1024).build();
        final Transfer upload = tm.upload(String.valueOf(System.getenv("S3_BUCKET_URL")),
                    "S3text1.txt", s3ObjectInputStream , new ObjectMetadata());
        upload.waitForCompletion();
        tm.shutdownNow();

        } catch (InterruptedException | AmazonClientException e) {
            e.printStackTrace();
        }
    }

Note: used Lambda cause that's where the business logic will take place. Uploading from one bucket to another , just to check the file uploading functionality into S3 using java.

Manvendra Pratap Singh
  • 11
  • 1
  • This seems related: [**Java Heap Space is insufficient to upload files on AWS S3**](https://stackoverflow.com/questions/54379555/java-heap-space-is-insufficient-to-upload-files-on-aws-s3), even though you're already setting content length. – Andrew Henle Oct 19 '22 at 19:48
  • (1) `s3ObjectInputStream.toString().length()` isn't doing what you think it is -- and that's a good thing, because otherwise it would load the entire file into memory. (2) You're passing an empty `ObjectMetadata` to `TransferManager.upload()`, so it will try to cache the entire file in memory. – kdgregory Oct 19 '22 at 20:02
  • To fix, get the object metadata from `s3Object` and either pass directly to the upload or create a new instance with just the values you want. – kdgregory Oct 19 '22 at 20:05
  • 2
    But if all you're doing is copying an object from one bucket to another, it would be _far_ better to use `AmazonS3.copy()`, because that happens entirely within S3. – kdgregory Oct 19 '22 at 20:06
  • @kdgregory I know I have an option to copy but here j am trying to workout upload just to check the functionality. – Manvendra Pratap Singh Oct 20 '22 at 02:24
  • @kdgregory so you're saying instead of taking Metadata from s3ObjectInputStream I should get it from S3Object? – Manvendra Pratap Singh Oct 20 '22 at 02:33

0 Answers0