standard_init_linux.go:219: exec user process caused: no such file or directory

Question

I am trying to move my rust server from Heroku to Google Cloud or AWS. Even though I like the simplicity of having a git push build and deploy to Heroku with just a buildpack specified, the service is not cost effective for me.

I identified Google Cloud Run and AWS Elastic Beanstalk as potential alternatives.

First, I need to build a docker image with a static binary.

Thus, I added this Dockerfile:

FROM rust AS build
WORKDIR /usr/src
RUN rustup target add x86_64-unknown-linux-musl
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential git clang llvm-dev libclang-dev libssl-dev pkg-config libpq-dev musl-tools brotli
RUN USER=root cargo new loxe-api
WORKDIR /usr/src/loxe-api
COPY Cargo.toml Cargo.lock ./
COPY data ./data
COPY migrations ./migrations
ENV RUSTFLAGS="-C target-feature=+crt-static" # this should be set by the target, but just to be sure
RUN cargo build --release
COPY src ./src
ENV PKG_CONFIG_ALLOW_CROSS=1
ENV OPENSSL_INCLUDE_DIR="/usr/include/openssl"
RUN cargo install --target x86_64-unknown-linux-musl --path .

FROM scratch
COPY --from=build /usr/local/cargo/bin/loxe-api .
COPY data ./data
COPY migrations ./migrations
USER 1000
CMD ["./loxe-api"]

The image gets build without errors, but if I run it via docker run, I'm getting this error:

standard_init_linux.go:219: exec user process caused: no such file or directory

By replacing the base image of the final step with rust, I checked whether the binary and other files are in fact in the image. They are, I can see them via ls, but I also cannot execute loxe-api when entering the shell.

dockerd logs this:

INFO[2020-07-05T13:04:42.368119033-07:00] shim containerd-shim started                  address=/containerd-shim/bf85e63468a9c1b3b9fe418b5a186673f0609bfff20c4832789ae87433e82473.sock debug=false pid=27032
INFO[2020-07-05T13:04:42.913438974-07:00] shim reaped                                   id=8cadeee800649ceca8a52d9a75cc9071b923d01a5d2a37497bf8b9a6e719267a
INFO[2020-07-05T13:04:42.925442900-07:00] ignoring event                                module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

Here's dependencies-section of the Cargo.toml:

[dependencies]
actix = "0.9"
actix-cors = "0.2"
actix-identity = "0.2"
actix-multipart = "0.2"
actix-rt = "1.0"
actix-web = "2.0"
argonautica = "0.2"
brotli = "3.3"
bytes = { version = "0.5", features = ["serde"]  }
chrono = { version = "0.4", features = ["serde"]  }
derive_more = "0.99"
diesel = { version = "1.4", features = ["postgres", "uuidv07", "r2d2", "chrono", "serde_json"]  }
diesel_migrations = "1.4"
dotenv = "0.15"
env_logger = "0.7"
futures = "0.3"
indexmap = { version = "1.3", features = ["serde-1"] }
lazy_static = "1.4"
log = "0.4"
openssl = { version = "0.10", features = ["vendored"] }
openssl-probe = "0.1.2"
percent-encoding = "2.1"
r2d2 = "0.8"
rand = "0.7"
redis = "0.15"
rusoto_core = { version = "0.44" }
rusoto_s3 = { version = "0.44" }
sanitize-filename = "0.2"
sendgrid = { version = "0.10", features = ["rustls"] }
serde = { version = "1.0", features = ["derive"]  }
serde_json = "1.0"
stripe-rust = "0.12"
uuid = { version = "0.8", features = ["serde", "v4"]  }
wana_kana = "2.0"

Further investigation from inside the container:

$ ldd /loxe-api
        linux-vdso.so.1 (0x00007ffcc219d000)
        libpq.so.5 => /usr/lib/x86_64-linux-gnu/libpq.so.5 (0x00007f2d3792d000)
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f2d3789b000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f2d375b2000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f2d37565000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f2d37511000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2d374f0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2d3732d000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2d37328000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f2d37248000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f2d37214000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f2d3720e000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f2d371ff000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f2d371f6000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f2d371dc000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f2d371cb000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f2d371ae000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f2d37002000)
        /lib/ld64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007f2d37983000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f2d36ed1000)
        libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f2d36eb2000)
        libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f2d36d2e000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f2d36b1b000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f2d36ae3000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f2d36aaa000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f2d36a25000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f2d36a1b000)

I also tried to build and execute from my host system. I ran cargo run --target x86_64-unknown-linux-musl --release which built the binary, but ultimately resulted in this error:

error: could not execute process target/x86_64-unknown-linux-musl/release/loxe-api (never executed)

I checked for the existence of the binary via du -h target/x86_64-unknown-linux-musl/release/loxe-api:

35M     target/x86_64-unknown-linux-musl/release/loxe-api

Further unsuccessful attempts with out-of-the-box and some modified cross, clux/muslrust and emk/rust-musl-builder repos have been made.

Building and running a new cargo default project with the x86_64-unknown-linux-musl target works. I assume, libclang, brotli, or some of the argonautica libraries may make this not work.

Here's a simplified Dockerfile that yields the same result.

FROM rust AS build
WORKDIR /usr/src

RUN rustup target add x86_64-unknown-linux-musl
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential git clang llvm-dev libclang-dev libssl-dev pkg-config libpq-dev musl-tools brotli

RUN USER=root cargo new loxe-api
WORKDIR /usr/src/loxe-api
COPY Cargo.toml Cargo.lock ./
COPY data ./data
COPY migrations ./migrations
COPY src ./src
ENV PKG_CONFIG_ALLOW_CROSS=1
ENV OPENSSL_INCLUDE_DIR="/usr/include/openssl"
ENV RUSTFLAGS="-C target-feature=+crt-static"
RUN cargo install --target x86_64-unknown-linux-musl --path .

FROM debian
COPY --from=build /usr/local/cargo/bin/loxe-api .
COPY .env ./.env
COPY data ./data
COPY migrations ./migrations
USER 1000
CMD ["./loxe-api"]

Does `loxe-api` have the execute permission (check with `ls -l`)? Does the filesystem it's on have the `noexec` flag (check with `mount`)? — Jmb, Jul 06 '20 at 06:51
@Jmb +x is set. Which filesystem are we talking about exactly? — manonthemat, Jul 06 '20 at 18:55
Are you sure you don't mean to `COPY src ./src` before running `cargo build --release`? It seems like you are trying to make a static build as well. You will need to have `musl-gcc` installed. IMHO I think you're better off using a slim Debian image instead of a scratch image and not worrying about having a static binary. Especially since you have some library dependencies, it will be very hard to get them to link via musl. — squiguy, Jul 06 '20 at 23:27
musl-gcc is part of musl-tools. I added a simpler Dockerfile that skips a build-step that was there for caching reasons. @squiguy, when I build locally, my target/release dir is about ~1gb. Add the actual docker image, and it'll grow into something that would push the limits of GCP Cloud Run really fast. Am I missing something here? — manonthemat, Jul 07 '20 at 05:45
The filesystem on which `loxe-api` is stored. Other question: does it fail also if you launch it manually or only when you try to have it start automatically? — Jmb, Jul 07 '20 at 06:21
No, noexec is not set. It also fails if I try to launch manually. With a brand new cargo helloworld project, all of this works though. — manonthemat, Jul 07 '20 at 18:06
I guess I am a little confused since `ldd` is showing that it isn't a statically linked binary. I'm not sure what you mean by "pushing the limits of GCP Cloud Run" either. — squiguy, Jul 07 '20 at 19:14
I just noticed: you compiled for the musl target (`x86_64-unknown-linux-musl`), but it looks like the shared libraries are compiled for glibc (at least they're in `/usr/lib/x86_64-linux-gnu`). — Jmb, Jul 08 '20 at 06:40
@Jmb thanks for pointing that out. I'm confused on why that is, but that explains the weirdness I'm seeing. — manonthemat, Jul 08 '20 at 16:53

score 2 · Accepted Answer · answered Jul 20 '20 at 01:38

While still a little bloated, at least now I have my service running on GCP Cloud Run. This is how I created a 241 mb docker image that I can ship to different services.

First, I replaced the aronautica crate with rust-argon2. Second, I modified the Dockerfile:

FROM rust AS build
WORKDIR /usr/src
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential git clang llvm-dev libclang-dev libssl-dev pkg-config libpq-dev brotli
RUN USER=root cargo new loxe-api
WORKDIR /usr/src/loxe-api
COPY Cargo.toml Cargo.lock ./
COPY data ./data
COPY migrations ./migrations
RUN cargo build --release
# Copy the source and build the application.
COPY src ./src
ENV PKG_CONFIG_ALLOW_CROSS=1
ENV OPENSSL_INCLUDE_DIR="/usr/include/openssl"
RUN cargo install --path .

FROM debian:buster-slim
COPY --from=build /usr/local/cargo/bin/loxe-api .
# standard env
COPY .env ./.env
COPY data ./data
COPY migrations ./migrations
RUN apt-get update && apt-get install -y libssl-dev pkg-config libpq-dev brotli
CMD ["/loxe-api"]

And that's essentially it. The produced Docker image now runs without problems on Google Cloud Run.

score 1 · Answer 2 · answered Jul 15 '20 at 00:36

I cannot build your simplified Dockerfile as-is because I do not have the source files you reference in COPY statements, so I get "COPY failed" errors. You say "Building and running a new cargo default project with the x86_64-unknown-linux-musl target works" and indeed this Dockerfile (your simplified Dockerfile with the COPY commands removed) works fine for me:

FROM rust AS build
WORKDIR /usr/src

RUN rustup target add x86_64-unknown-linux-musl
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential git clang llvm-dev libclang-dev libssl-dev pkg-config libpq-dev musl-tools brotli

RUN USER=root cargo new loxe-api
WORKDIR /usr/src/loxe-api

ENV PKG_CONFIG_ALLOW_CROSS=1
ENV OPENSSL_INCLUDE_DIR="/usr/include/openssl"
ENV RUSTFLAGS="-C target-feature=+crt-static"
RUN cargo install --target x86_64-unknown-linux-musl --path .

FROM debian
COPY --from=build /usr/local/cargo/bin/loxe-api .

USER 1000
CMD ["./loxe-api"]

However, when I build that, I get a statically linked executable, which is what I expect, but not what your ldd output shows. My ldd:

$ ldd loxe-api 
    not a dynamic executable

Similarly, I can build your larger Dockerfile after removing the COPY commands and the comment, and it builds fine for me.

Are you perhaps using Windows? You may be suffering from line-ending problems, and updating your git configuration and re-cloning your git repo could help.

# update git to automatically set line ending to LF
git config --global core.eol lf
git config --global core.autocrlf input

After that, you'll need to delete and re-clone your git repo. See this helpful doc from GitHub for more.

If that does not help, please post some code that reproduces the problem.

Thanks for the help. I'm on a linux system, so the proposal was not effective. Regarding the code, it's a few thousand lines of code at this point and not really open source material at this point. Maybe the posted Cargo.toml can give some clues? I really don't know how to debug this further. — manonthemat, Jul 15 '20 at 04:42
@manonthemat A CRLF from a Windows system committed to the git repo could still be the problem even if you're on Linux. Set `core.autocrlf false` instead of `input` and check out a new clone of the repo. If that doesn't help, run through the Docker build steps manually from inside the rust container. `docker run -it rust bash` and then run each step from the command line and verify you can run `loxe-api` at the end. — Old Pro, Jul 15 '20 at 09:14
No windows systems have been involved in creating any code. All steps succeed with the exception of running the binary. — manonthemat, Jul 17 '20 at 21:49
@manonthemat After `cargo install`, what does `ldd` say about the installed binary? It should complain that `loxe-api` is not a dynamic executable. If it *is* a dynamic executable, then review your setup and documentation to figure out how to make it statically linked. If, immediately after `cargo install`, the installed file is statically linked and produces a "file not found" error when run, that suggests it is the executable that is failing to find a required (configuration, data, migration, etc.) file. Review the source code for files it tries to open. — Old Pro, Jul 18 '20 at 00:31
Thanks for the heads up on line endings. I've got the same error message and I'm developing on Windows so that was my problem, in spite of not being OP's problem. — VBobCat, Jan 09 '21 at 12:57

standard_init_linux.go:219: exec user process caused: no such file or directory

2 Answers2

Linked