3

Let's say I have 50 objects (15Mb each) stored in Google Cloud Storage. Now I need to create a zip archive containing all of them and store the resultant file back at GCS. How can I do that from within an appengine java app?

Fábio Uechi
  • 807
  • 7
  • 25
  • You can use ZipOutputStream. [This answer](http://stackoverflow.com/a/13772805/624900) has most of what you need to get started. – jterrace Jul 26 '13 at 15:34
  • Wrote the method above based on your recommendation. Just don't know how to check data integrity yet. – Fábio Uechi Jul 27 '13 at 21:09
  • You should post that as an answer instead of in the question if it works. What do you mean by integrity? – jterrace Jul 27 '13 at 21:41

1 Answers1

2

I wrote the method below which seems to be working fine.

public static void zipFiles(final GcsFilename targetZipFile,
        final GcsFilename... filesToZip) throws IOException {

    Preconditions.checkArgument(targetZipFile != null);
    Preconditions.checkArgument(filesToZip != null);
    Preconditions.checkArgument(filesToZip.length > 0);

    final int fetchSize = 4 * 1024 * 1024;
    final int readSize = 2 * 1024 * 1024;
    GcsOutputChannel outputChannel = null;
    ZipOutputStream zip = null;
    try {
        GcsFileOptions options = new GcsFileOptions.Builder().mimeType(MediaType.ZIP.toString()).build();
        outputChannel = GCS_SERVICE.createOrReplace(targetZipFile, options);
        zip = new ZipOutputStream(Channels.newOutputStream(outputChannel));
        GcsInputChannel readChannel = null;
        for (GcsFilename file : filesToZip) {
            try {
                final GcsFileMetadata meta = GCS_SERVICE.getMetadata(file);
                if (meta == null) {
                    LOGGER.warning(file.toString() + " NOT FOUND. Skipping.");
                    continue;
                }
                //int fileSize = (int) meta.getLength();
                //  LOGGER.fine("adding " + file.toString());
                ZipEntry entry = new ZipEntry(file.getObjectName());
                zip.putNextEntry(entry);
                readChannel = GCS_SERVICE.openPrefetchingReadChannel(file, 0, fetchSize);
                final ByteBuffer buffer = ByteBuffer.allocate(readSize);
                int bytesRead = 0;
                while (bytesRead >= 0) {
                    bytesRead = readChannel.read(buffer);
                    buffer.flip();
                    zip.write(buffer.array(), buffer.position(), buffer.limit());
                    buffer.rewind();
                    buffer.limit(buffer.capacity());
                }       

            } finally {
                zip.closeEntry();
                readChannel.close();
            }
        }
    } finally {
        zip.flush();
        zip.close();
        outputChannel.close();
    }
}
Fábio Uechi
  • 807
  • 7
  • 25