0

I'm using Google Cloud Run to host some solutions. When the containers start, programs can write to disk, and the data persists until the container stops. However, from a system point of view, all partitions of the container always report zero free space. I confirmed this in a few ways:

  • Running df from start.sh shows zero free space when the container starts
  • Deleting a large file and then running df from start.sh still shows zero free space
  • It is possible to write to disk via start.sh, PHP scripts, etc, so the system DOES have free space to write to memory, yet df still reports zero free space

(All of the above are once the container is deployed to Cloud Run. Manually running the same container via docker from the Cloud Shell and executing df reports free space).

The problem is that certain applications perform disk space checks when they start, and they fail to load in Google Cloud Run. For example, MariaDB uses df in its init script, so commenting out these lines makes it possible to add a static yet functional MariaDB instance to a Cloud Run container.

MariaDB made it easy. Now, I'm trying to do the same thing with PostgreSQL and RabbitMQ, but I'm having trouble figuring out how to override their disk space checks. Here are the two options I am considering:

  • Keep digging through the source of PostgreSQL and RabbitMQ until I find the disk space check and override it. I don't speak Erlang, so this is a pain, and I would have to do it for every application with this issue
  • Programs are probably using coreutils to determine disk size. I could edit the source and rebuild it as part of my Dockerfile routine so the system always returns with free space available (could have unintentional side effects)

Is anyone either familiar with the source of Postgres or RabbitMQ or have a system-wide solution that I could implement that would "spoof" the free space available?

EDIT: Here are the error messages given by RabbitMQ and PostgreSQL

RabbitMQ:

{error,{cannot_log_to_file,"/var/log/rabbitmq/rabbit@localhost.log",{error,einval}}}

Postgres:

Error: /usr/lib/postgresql/10/bin/pg_ctl /usr/lib/postgresql/10/bin/pg_ctl start -D /var/lib/postgresql/10/main -l /var/log/postgresql/postgresql-10-main.log -s -o -c config_file="/etc/postgresql/10/main/postgresql.conf" exited with status 1:
Arthur Sommers
  • 83
  • 1
  • 9
  • I did a quick search through the PostgresSQL sources. I don't see any use of df, or statfs, or statvfs. Are you sure that it checks how much disk space is left? I think it might go ahead and write, and let the OS tell it about a lack of free space. Can you post the error messages you get from Postgres/RabbitMQ? It would make my searches easier. – Nick ODell Sep 11 '20 at 18:00
  • Yeah, MySQL uses df, but I'm not sure what Postgres or RabbitMQ use (likely not df, though). Error messages are added to the OP. – Arthur Sommers Sep 11 '20 at 18:26
  • 1
    It's highly [discouraged](https://stackoverflow.com/a/57864568/5495197) to run a database inside Cloud Run and although you can do it this isn't what is meant for and you will run on more issues besides the installation – Emmanuel Sep 11 '20 at 18:46
  • For the RabbitMQ, EINVAL doesn't suggest that the disk is out of space. If you were getting ENOSPC, that *would* suggest that's the problem. But I'm guessing that the filesystem doesn't support fsync: https://stackoverflow.com/questions/11055060/possible-reasons-of-linux-open-call-returning-einval – Nick ODell Sep 11 '20 at 20:06
  • For the Postgres, the error message posted has been cut off, so I can't tell what's wrong. (Postgres can exit with 1 for a lot of reasons.) – Nick ODell Sep 11 '20 at 20:07
  • @NickODell, any idea how to bypass the disk space checks in RabbitMQ? I've never looked at Erlang code, and I only got as far as [rabbit_prelaunch.erl](https://github.com/rabbitmq/rabbitmq-server/blob/e61c608380e9bd3dba2c2fe5656c19c69e130525/apps/rabbitmq_prelaunch/src/rabbit_prelaunch.erl). I suspect if I can override that, the application will load. As for Postgres . . . I've got nothing. That's the only error it gives. – Arthur Sommers Sep 11 '20 at 22:02
  • @Emmanuel - understood, but MySQL is being used in a static manner for my use case. The data loaded at runtime is what matters; changes will not be an issue nor reflected in the application. – Arthur Sommers Sep 11 '20 at 22:10

0 Answers0