0

Is it possible to take a file or files, create a zip version of that file and then create an inputstream version of the new zip without storing the whole zip in memory or on a file system?

I am ideally looking for an input->output->input stream where the data is only ever buffered or piped.

The reason, I am trying to take some objects stored in an aws bucket and zip them using an aws lambda function, but lambda has memory limits and can't always store a full zip.

I have been reading about piped input/outputstreams and still don't fully understand how (if at all) my goal is achievable.

private final String BUCKET = "aws-bucket-name";
private final int BUFFER = 2048;

public zip( String[] keys ) {
    AmazonS3 s3Client = new AmazonS3Client( new EnvironmentVariableCredentialsProvider() );
    for( String inFilename : keys ) {
        String outFilename = "output/" + inFilename;
        S3Object object = s3Client.getObject( new GetObjectRequest( BUCKET, inFilename ) );
        InputStream objectData = object.getObjectContent();
        ObjectMetadata omd = new ObjectMetadata();
        BufferedInputStream origin = new BufferedInputStream( objectData, BUFFER );

    /* need to take above file(s) create a zip stream to them use as input stream for putObject method
     * FileOutputStream dest = new FileOutputStream( outFilename );
     * ZipOutputStream out = new ZipOutputStream( new BufferedOutputStream( dest ) );
     * byte data[] = new byte[ BUFFER ];
     * ZipEntry entry = new ZipEntry( output );
     * out.putNextEntry( entry );
     * int count;
     * while ( ( count = origin.read( data, 0, BUFFER ) ) != -1 )
     * {
     *     out.write( data, 0, count );
     * }
     */

        s3Client.putObject( BUCKET, outFilename, origin, omd );
    // close streams
    }
}
APW
  • 420
  • 4
  • 15
  • Yes, store the ZIP output as a file and then open the file for input. Hard to see what the mystery is. – user207421 Aug 22 '16 at 09:42
  • I have no where to store a zip file hence my need to use some kind of buffered or piped stream – APW Aug 22 '16 at 09:55
  • You have no file system? Really? – user207421 Aug 22 '16 at 10:34
  • Correct it is using aws' lambda service, it is designed for quick low memory functions, I can use and access aws services like buckets and databases, but to put something in a bucket I need to supply an inputstream. The zip is going to end up in the bucket, I just need to figure out a way of getting it there. – APW Aug 22 '16 at 10:53
  • You have a 500MB `/tmp` space on AWS Lambda. – Mark B Aug 22 '16 at 13:44
  • the tmp folder is fine when the output is smaller than 500MB, but doesn't help when the file is larger, which could easily happen. – APW Aug 22 '16 at 15:15
  • From the general lack of enthusiasm for this question I assume that it is not possible to run some kind of continuous stream. – APW Aug 23 '16 at 08:38
  • The question should be tagged with s3 and maybe aws-api, not aws-lambda since the fact it is running on Lambda isn't relevant to the issue at hand. There is a lack of enthusiasm because you asked a vague "is it possible" question and stated you are having trouble understanding things, and then just dumped all your code. If you want a specific answer then specify exactly what you want your code to do, any error messages you are receiving, and what line of code is failing. – Mark B Aug 23 '16 at 16:11
  • Your question looks like a duplicate of this one: http://stackoverflow.com/questions/11715979/best-way-to-pipe-inputstream-to-outputstream – Mark B Aug 23 '16 at 16:11
  • Thank you for your input and the link, I spent 2 days searching before posting and oddly that never came up in my searches. – APW Aug 24 '16 at 09:55

0 Answers0