Inter project file transfer in firebase using cloud trigger functions using golang

Question

I am trying to transfer files between the two projects. I managed to create a service account and gave read access to the storage in the source project and write access to the storage in the destination project. I created a storage trigger function in the source project which will read the file when created. Now my requirement is to copy the same file to the destination project.

I have this code copied from online examples, which can copy files between bucket in the same project. I want to achieve the same between two different projects. Is this possible? TIA.

package p

import (
    "context"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "log"
    "time"

    "cloud.google.com/go/functions/metadata"
    "cloud.google.com/go/storage"
)

var (
    storageClient     *storage.Client
    sourceBucket      = "testSource"
    destinationBucket = "testDestination"
)

type Event struct {
    Kind                    string                 `json:"kind"`
    ID                      string                 `json:"id"`
    SelfLink                string                 `json:"selfLink"`
    Name                    string                 `json:"name"`
    Bucket                  string                 `json:"bucket"`
    Generation              string                 `json:"generation"`
    Metageneration          string                 `json:"metageneration"`
    ContentType             string                 `json:"contentType"`
    TimeCreated             time.Time              `json:"timeCreated"`
    Updated                 time.Time              `json:"updated"`
    TemporaryHold           bool                   `json:"temporaryHold"`
    EventBasedHold          bool                   `json:"eventBasedHold"`
    RetentionExpirationTime time.Time              `json:"retentionExpirationTime"`
    StorageClass            string                 `json:"storageClass"`
    TimeStorageClassUpdated time.Time              `json:"timeStorageClassUpdated"`
    Size                    string                 `json:"size"`
    MD5Hash                 string                 `json:"md5Hash"`
    MediaLink               string                 `json:"mediaLink"`
    ContentEncoding         string                 `json:"contentEncoding"`
    ContentDisposition      string                 `json:"contentDisposition"`
    CacheControl            string                 `json:"cacheControl"`
    Metadata                map[string]interface{} `json:"metadata"`
    CRC32C                  string                 `json:"crc32c"`
    ComponentCount          int                    `json:"componentCount"`
    Etag                    string                 `json:"etag"`
    CustomerEncryption      struct {
        EncryptionAlgorithm string `json:"encryptionAlgorithm"`
        KeySha256           string `json:"keySha256"`
    }
    KMSKeyName    string `json:"kmsKeyName"`
    ResourceState string `json:"resourceState"`
}

func init() {
    var err error

    storageClient, err = storage.NewClient(context.Background())
    if err != nil {
        log.Fatalf("storage.NewClient: %v", err)
    }
}

func FileTransfer(ctx context.Context, e Event) error {
    meta, err := metadata.FromContext(ctx)
    if err != nil {
        return fmt.Errorf("metadata.FromContext: %v", err)
    }

    //Retry Logic - Ignore events that are too old
    expiration := meta.Timestamp.Add(10 * time.Second)
    if time.Now().After(expiration) {
        log.Printf("event timeout: halting retries for expired event '%q'", meta.EventID)
        return nil
    }

    byt, _ := json.Marshal(&e)
    fmt.Println("Event Data: ", string(byt))

    docData, err := read(storageClient, e.Bucket, e.Name)
    if err != nil {
        fmt.Println("Document read failed")
        return fmt.Errorf("Document read failed")
    }

    fmt.Println("Document Data Read success: ", string(docData))

    err = copyToBucket(storageClient, destinationBucket, e.Bucket, e.Name)
    if err != nil {
        fmt.Println("Document copy failed")
        return fmt.Errorf("Document copy failed")
    }

    return nil
}

func read(client *storage.Client, bucket, object string) ([]byte, error) {
    ctx := context.Background()
    // [START download_file]
    rc, err := client.Bucket(bucket).Object(object).NewReader(ctx)
    if err != nil {
        return nil, err
    }
    defer rc.Close()

    data, err := ioutil.ReadAll(rc)
    if err != nil {
        return nil, err
    }
    return data, nil
    // [END download_file]
}

func move(client *storage.Client, bucket, object string) error {
    ctx := context.Background()
    // [START move_file]
    dstName := object + "-rename"

    src := client.Bucket(bucket).Object(object)
    dst := client.Bucket(bucket).Object(dstName)

    if _, err := dst.CopierFrom(src).Run(ctx); err != nil {
        return err
    }
    if err := src.Delete(ctx); err != nil {
        return err
    }
    // [END move_file]
    return nil
}

func copyToBucket(client *storage.Client, dstBucket, srcBucket, srcObject string) error {
    ctx := context.Background()
    // [START copy_file]
    dstObject := srcObject
    src := client.Bucket(srcBucket).Object(srcObject)
    dst := client.Bucket(dstBucket).Object(dstObject)

    if _, err := dst.CopierFrom(src).Run(ctx); err != nil {
        return err
    }
    // [END copy_file]
    return nil
}

Do not copy Cloud Storage Object data in your function (download/upload/delete). Cloud Storage can copy for you. https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite — John Hanley, Dec 06 '19 at 16:06

Luiz Ferraz · Accepted Answer · 2019-12-06T17:36:13.977

0

If you added all the permissions properly this code should work.

GCS buckets are not bound to projects as a child resource in their path, only for billing and deletion. Accessing them from the same project or from different projects require no change at the code.

edited Dec 06 '19 at 17:36

answered Dec 05 '19 at 17:55

Luiz Ferraz

1,427
8
13

Buckets are bound to projects. The name must be globally unique. – John Hanley Dec 06 '19 at 16:00
1

Suggestion: edit your answer with the additional information and then delete your comment. I will then delete my comments also. This will make for a better answer. – John Hanley Dec 06 '19 at 17:21

score 0 · Answer 2 · answered Dec 06 '19 at 12:43

There no difficulties for achieving this. Grant the service account of your function (the default one, or this one that you explicitly set at deployment) with the role Storage Object Creator. I recommend you to grant the service account on only the bucket and not at the project level for keeping the narrowest autorisations' scope.

I also recommend you to set in environment variable the source and the destination bucket for gaining in agility.

In addition, your move function could reuse the copyToBucket and then perform the delete. This for reducing the code duplication. Even if you have to add a parameter for the file destination name.

Final word about your file read. Be careful to the size of the file, not to be too big for your function allocated memory. And for now, you do nothing with. Useful to get it?

The destination project is kind of a militarized zone, so the service account (which has source read and destination write access) that runs the cloud function, must be created in the destination project. Due to this, I was not able to assign the service account to the cloud function. — Hardy, Dec 24 '19 at 09:23

score 0 · Answer 3 · answered Jan 10 '20 at 08:32

0

You have to Add the cloud function service account of the project A in the project B and grant the Storage permissions.

Follow the indications in this answer from other post. https://stackoverflow.com/a/35558464/6003934

answered Jan 10 '20 at 08:32

Juancki

1,793
1
14
21

If you have any errors when deploying or calling the Google Cloud Function, please share the error trace. – Juancki Jan 10 '20 at 08:37

Inter project file transfer in firebase using cloud trigger functions using golang

3 Answers3