0

I have the following situation. I'm using laravel and google cloud storage so basically, I have my own custom filesystem attached to google cloud storage. My code looks like this - 

try{
    $disk = Storage::disk('gcs');

                $disk->put('/service_provider/transports/tech_passports', $file1);
                $disk->put('/service_provider/transports/tech_passports', $file2);

                $disk->put('/service_provider/transports/pictures', $file3);
                $disk->put('/service_provider/transports/pictures', $file4);
                $disk->put('/service_provider/transports/pictures', $file5);
                $disk->put('/service_provider/transports/pictures', $file6);
    } catch(\Exception $e){

}

Now i'm worried. what if the first three files get uploaded and the fourth one has an error. so it will go to catch block and finally, only 3 files will be uploaded which is not what I need.

What I need is all get uploaded or no file gets uploaded just like atomicity. how do I achieve that?

Joan Grau Noël
  • 3,084
  • 12
  • 21
Nika Kurashvili
  • 6,006
  • 8
  • 57
  • 123
  • 1
    Google Cloud Storage supports "atomicity" only for a single object upload to a storage bucket. If you want atomicity for a group of files, you will need to upload one at a time or in parallel. If any part of the group upload fails you will need to delete the other sucessful objects. – John Hanley Jan 07 '19 at 20:20
  • what if it fails when trying to delete successful objects? – Nika Kurashvili Jan 07 '19 at 20:23
  • 1
    There are a number of issues, such as failing to delete, that your code or library will need to handle. In the case of a delete failing, for example your connect drops, you will need to keep track of state to recover at a later time. There is no "Google Cloud Storage Solution" to guarantee that a group of files will either upload as a "transaction" or completely fail. You will have to implement this yourself. – John Hanley Jan 07 '19 at 20:34
  • That's too bad. Thanks for your help so far. – Nika Kurashvili Jan 07 '19 at 20:37

1 Answers1

0

To be sure that all your files get uploaded, try using resumable uploads on each of your individual files, with the following steps:

Initiate upload > Process response > Upload The file > Check Upload Status > Process response > Resume the upload

Retry last three steps to resume upload as many times as necessary or implement another solution (for example: delete all files already uploaded), if it is what you need in your application.

You can also check Google Cloud Storage API PHP resumable behavior in the example from another question on StackOverflow.

Please note that also Cloud SDK utility Gsutil has resumable upload implemented in it. It can be used to Synchronize content of two buckets/directories.

Community
  • 1
  • 1
Pawel Czuczwara
  • 1,442
  • 9
  • 20