0

When developing with containers locally, docker-compose lets you create shared volumes that all of your containers can access. You can easily drop small credential files onto these volumes from one container, and have another container use them.

I'm trying to find something similar in Google Compute Engine but I haven't been able to find anything analagous.

  • Compute Engine disks cannot be shared between instances
  • Filestore instances start at a minimum of 1 Tb and are expensive overkill

Is there anything similar in Google Compute Engine to the concept of shared volumes in Docker, in terms of how it can be mounted to the instances, shared among instances, and small/cheap?

Does such a concept not exist in GCE, and is such a feature perhaps available, but only available, in Google Kubernetes Engine (GKE)?

cilphex
  • 6,006
  • 6
  • 34
  • 44
  • Do you need only read access or read/write access? – guillaume blaquiere Aug 25 '20 at 12:41
  • @guillaumeblaquiere read/write access. – cilphex Aug 25 '20 at 16:53
  • Will you have a large volume of data? Or an high throughput required (in term of I/O (operation) or bandwith (size of files))? Will you have concurrent access and risk to write in the same time the same file? or to read a file while it is written? – guillaume blaquiere Aug 25 '20 at 20:01
  • @guillaumeblaquiere I only need a few megabytes of space that is written to rarely. Concurrent access is not expected. The purpose is for one service to generate credential files for another service to read and use only on startup. – cilphex Aug 25 '20 at 22:34

2 Answers2

0

Actually Compute Engine disks can be shared between instances, but at this time this feature is in beta.
According to Google terminology, Persistent Disk in Multi-writer Mode is called Shared PD or PD multi-writer. Shared PD is a persistent disk created with multiWriter option set to True. Shared PD can be attached to up to 2 VMs in read-write mode.

Google Cloud > Cloud SDK: CLI > Doc > Reference > gcloud beta compute disks create:
gcloud beta compute disks create --multi-writer
create Compute Engine persistent disk in multi-writer mode so that it can be attached with read-write access to multiple VMs. Can only be used with Zonal SSD persistent disks. Disks in multi-writer mode do not support resize and snapshot operations.

As for GKE, it supports disk sharing as well. You can share persistent disk between multiple Pods in read-only mode.
See Google Cloud > GKE > Doc > Using persistent disks with multiple readers for more details.

mebius99
  • 2,495
  • 1
  • 5
  • 9
  • I'm looking for a solution that allows read/write from all instances that have access to the disk, and allows sharing by more than just 2 VMs. – cilphex Aug 26 '20 at 02:05
0

An alternative solution is to use Cloud Storage for this. If you have few Mb and acceptable I/O operations, you can use gcsfuse. The principle is simple: mount a CLoud Storage bucket in your file system and write to it as any other directory in your system.

GCSFuse convert the read/write operation in API Call and you are charged on API call (few $ for millions of calls, but if your app is I/O intensive, it can cost!). In addition, it's API calls, that means, it's not a local disk and latency (due to network, HTTPS handshake,...) is higher than with a local disk.

So, keep in mind that GCSFuse is simply a wrapper of Google Cloud Storage APIs.

Note: If you want to share credentials, why you don't use Google Secret Manager?

guillaume blaquiere
  • 66,369
  • 2
  • 47
  • 76
  • I'd like to find a solution that maps easily to the docker-compose experience, where I don't need to make API calls to a centralized service or use a 3rd party secret manager. – cilphex Aug 26 '20 at 18:35
  • You have equivalent, you have to cheat. I haven't a more low cost solution for read/write shared space. – guillaume blaquiere Aug 26 '20 at 20:01