0

I don't have the necessary support package to ask Google directly. Therefor I'm posing this question anyone who has knowledge and a kind heart.

I’m planning on migrating some of my server needs from AWS to GCP. In the effort to make this happen, I’d like to get some guidance on the GCP product suite.

Some of the information in this segment might not be of relevance, but I’ll leave it in to brush out the situation and wants. I’m planning on sending files from an Amazon S3 bucket to a GCP gpu instance continuously, as the files are uploaded to my s3 bucket. When the files has been transferred to the GCP instance and is processed, it is to be send back the s3 bucket. This service is to run continuously and it’s the reason I’m planning on switching to GCP, as the GPU instances are much cheaper there in the long run.

Does anyone have suggestions to the architecture of this endeavor?

smac2020
  • 9,637
  • 4
  • 24
  • 38
Christian
  • 61
  • 1
  • 6

1 Answers1

0

One of the best option is to use the GCP buckets for the data storing/transfering. Create a GCP bucket and then connect it with the GCP instance, so that the data can be accessed via the instance.

You can go through this help center article on how to connect cloud storage buckets with instances(in this way the cloud bucket service will act as a common storage service between your instance and other resources (ex. Other cloud services)). Additionally, You can also mount the the GCP bucket as a local file system in your instance as well (check this document on mounting bucket).

After connecting the GCP buckets with the instance, you can make use of 'Storage Transfer Service' for transferring the files from external cloud service(ex:AWS S3 bucket) to the connected GCP storage bucket.

You might also need to check this discussion on exporting data from GCP storage to Amazon S3 as well.

Digil
  • 742
  • 4
  • 12
  • Thanks Digil! I've set up my GCP bucket and is able to sync the buckets using gsutil. I've used FUSE to add my bucket to my instance. So far so good. But I want to sync them continuously(automatically) or at least make transfers when new files are uploaded to S3(automatically). Do you by any chance know how to do this aswell? – Christian Feb 07 '18 at 23:06
  • In the '[Setting up a transfer job](https://cloud.google.com/storage/transfer/create-manage-transfer-console#configure)', you have an option to schedule your transfer job under the 'Configure transfer tab' – Digil Feb 08 '18 at 14:35
  • Thanks, but scheduling a transfer job isn't what I need. I need the buckets to be synchronized all the time. Not once a day like the options allow me. – Christian Feb 11 '18 at 16:47
  • This feature option does not exist as of now. You may want to raise a feature request on Google [public issue tracker](https://issuetracker.google.com) platform which meant for feature requests and bug tracking purpose. – D Saini Feb 14 '18 at 21:07