How to copy file from k8 pod to GCP storage programmatically?

Question

I am deploying my springboot application image to GCP.My springboot application processes data and creates one file.I want to copy this file to GCP Storage.Can we achieve this from Kubernetes pod to GCP Storage?

Answer 1

Yes you can, you need to install gsutil in your springboot application image then mount your storage token in that container as configmap or secret.

In the end you can run gsutil like this:

gsutil cp source_file gs://my-bucket

Answer 2

You can use GCSFUSE to mount GCS bucket to Kubernetes Pod as a volume.

Here is an example of Container Lifecycle Hooks to mount and unmount GCS bucket.

#...

securityContext:
  privileged: true
  capabilities:
    add:
      - SYS_ADMIN
lifecycle:
  postStart:
    exec:
      command: ["gcsfuse", "-o", "nonempty,allow_other", "bucket-name", "/usr/share/nginx/bucket-data"]
  preStop:
    exec:
      command: ["fusermount", "-u", "/usr/share/nginx/bucket-data"]

#...

Reference: MOUNTING GOOGLE CLOUD STORAGE BUCKET TO KUBERNETES POD

And to copy the file from your Pod to GCS bucket, use Fabric8 Kubernetes Client, which is described in detail here.

Example code:

public class DownloadFileFromPod {
    public static void main(String[] args) {
        try (KubernetesClient client = new KubernetesClientBuilder().build()) {
            // Path Where to copy file to cloud storage
            Path downloadToPath = new File("path_of_gcs").toPath();
            // Using Kubernetes Client to copy file from pod
            client.pods()
                    .inNamespace("default")                         // <- Namespace of pod
                    .withName("quarkus-84dc4885b-tsck6")            // <- Name of pod
                    .file("/deployments/quarkus-1.0.0-runner.jar")  // <- Path of file inside pod
                    .copy(downloadToPath);                            // <- path where to copy downloaded file
        }
    }
}

You can also use the Cloud Storage client libraries to programmatically achieve this.

Here is an example for uploading files using java

import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

public class UploadObject {
  public static void uploadObject(
      String projectId, String bucketName, String objectName, String filePath) throws IOException {
    // The ID of your GCP project
    // String projectId = "your-project-id";

    // The ID of your GCS bucket
    // String bucketName = "your-unique-bucket-name";

    // The ID of your GCS object
    // String objectName = "your-object-name";

    // The path to your file to upload
    // String filePath = "path/to/your/file"

    Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
    BlobId blobId = BlobId.of(bucketName, objectName);
    BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();

    // Optional: set a generation-match precondition to avoid potential race
    // conditions and data corruptions. The request returns a 412 error if the
    // preconditions are not met.
    Storage.BlobTargetOption precondition;
    if (storage.get(bucketName, objectName) == null) {
      // For a target object that does not yet exist, set the DoesNotExist precondition.
      // This will cause the request to fail if the object is created before the request runs.
      precondition = Storage.BlobTargetOption.doesNotExist();
    } else {
      // If the destination already exists in your bucket, instead set a generation-match
      // precondition. This will cause the request to fail if the existing object's generation
      // changes before the request runs.
      precondition =
          Storage.BlobTargetOption.generationMatch(
              storage.get(bucketName, objectName).getGeneration());
    }
    storage.create(blobInfo, Files.readAllBytes(Paths.get(filePath)), precondition);

    System.out.println(
        "File " + filePath + " uploaded to bucket " + bucketName + " as " + objectName);
  }
}

Also check this thread with python examples for more details.