0

We will be receiving daily file from consumer which is 45 MB size . We have a requirement to break this 45 file into small chunk of file based on a configurable number of rows.

Is there any AWS service available on top of S3 which can do this work

This process should be automated one and no manual intervention should be there. Need to achieve it using Java (Preference).Any other language is also fine.

sushant shekhar
  • 51
  • 4
  • 1
    Why not do it using lambda function itself? – Marcin Dec 02 '22 at 06:18
  • Hi Marcin , is there any such AWS service available to do this?If not finally we will go for writing lamda only. But exploring if there is already some service avialable. – sushant shekhar Dec 02 '22 at 06:29
  • Probably Glue Jobs could do it. – Marcin Dec 02 '22 at 06:59
  • 1
    See https://stackoverflow.com/a/56149026/10354667 for a way to do this using AWS CLI and the ```split``` command, but a Lambda function would likely be the easiest and cost effective way. – chamal Dec 02 '22 at 07:02

1 Answers1

0

There is no "split my file into small chunks" service in AWS. You would need compute to perform this operation, such as an Amazon EC2 instance, an AWS Lambda function, or an AWS Fargate container.

If "automated one and no manual intervention" means "do this when the file is uploaded" then the most appropriate would be an AWS Lambda function since S3 can trigger the function upon upload.

The Lambda function will be provided with the Bucket and Key of the S3 object that triggered the function. You will need to write to code that downloads the file, splits it into multiple files and uploads them to S3. You might be able to do it in-memory depending on file size. There are plenty of examples online of how to use S3 from an AWS Lambda function.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470