13

I have 3GB of size data file to upload into GCP Cloud Storage. I tried with the example in GCP Uploading objects tutorial. But when I'm trying to upload I got the following error.

java.lang.OutOfMemoryError: Required array size too large

I tried as follows,

BlobId blobId = BlobId.of(gcpBucketName, "ft/"+file.getName());
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setContentType("text/plain").build();
Blob blob = storage.get().create(blobInfo, Files.readAllBytes(Paths.get(file.getAbsolutePath())));
return blob.exists();

How may I fix this? Are there any possible way to upload large files using GCP Cloud storage java client?

codebot
  • 2,540
  • 3
  • 38
  • 89

2 Answers2

14

Version of storage:

  <artifactId>google-cloud-storage</artifactId>
  <version>1.63.0</version>

Preparation: 

            BlobId blobId = BlobId.of(BUCKET_NAME, date.format(BASIC_ISO_DATE) + "/" + prefix + "/" + file.getName());
            BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setContentType("application/gzip").build();
            uploadToStorage(storage, file, blobInfo);

Main method:

private void uploadToStorage(Storage storage, File uploadFrom, BlobInfo blobInfo) throws IOException {
    // For small files:
    if (uploadFrom.length() < 1_000_000) {
        byte[] bytes = Files.readAllBytes(uploadFrom.toPath());
        storage.create(blobInfo, bytes);
        return;
    }

    // For big files:
    // When content is not available or large (1MB or more) it is recommended to write it in chunks via the blob's channel writer.
    try (WriteChannel writer = storage.writer(blobInfo)) {

        byte[] buffer = new byte[10_240];
        try (InputStream input = Files.newInputStream(uploadFrom.toPath())) {
            int limit;
            while ((limit = input.read(buffer)) >= 0) {
                writer.write(ByteBuffer.wrap(buffer, 0, limit));
            }
        }

    }
}
Alexey Alexeenka
  • 891
  • 12
  • 15
  • Examples in GC Storage GIT https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-examples/src/main/java/com/google/cloud/examples/storage/snippets/StorageSnippets.java – Olga Aug 07 '19 at 06:52
  • Thanks for the hint. This solution is exakt what i'm looking for. – Ben Dec 09 '20 at 18:21
2

This is happening because the array that is returned by Files.readAllBytes has a bigger size than the maximum allowed.

A workaround you can do is to divide the file into multiple arrays of bytes, upload them as separated files to the bucket and join them using the gsutil compose command.

mgoya
  • 512
  • 3
  • 12