'exec user process caused: exec format error' in AWS Fargate Service

Question

I am totally new to AWS but I've been running my dockerized application locally for months now with no issues. Now that I am trying to deploy this app via AWS ECS/Fargate, my containers are stopped repeatedly with this linux error: standard_init_linux.go:219: exec user process caused: exec format error. This error seems to suggest that the architecture in Fargate does not recognize one of the Linux commands I'm running but I can't find a good answer anywhere for how to find the architecture that's running or how to track down the specific command that's causing the issue.

These are my Dockerfiles for the frontend and backend. The project is built in the MERN stack and is split into the frontend (React) and the backend (MongoDB/Express)

Frontend:

FROM alpine:3.10

ENV NODE_VERSION 15.9.0

WORKDIR /frontend

COPY package*.json ./

RUN apk add --no-cache nodejs npm

# some packages rely on gyp so we need this
# pulled from https://github.com/nodejs/docker-node/issues/282
RUN apk add --no-cache --virtual .gyp \
        python \
        make \
        g++ \
    && npm install

COPY . .

EXPOSE 3000

CMD ["npm", "start"]

Backend:

FROM alpine:3.10

ENV NODE_VERSION 15.9.0

WORKDIR /backend

COPY package*.json ./

RUN apk add --no-cache nodejs npm

# some packages rely on gyp so we need this
# pulled from https://github.com/nodejs/docker-node/issues/282
RUN apk add --no-cache --virtual .gyp \
        python \
        make \
        g++ \
    && npm install --silent\
    && apk del .gyp

COPY ./ ./

EXPOSE 8080

CMD ["npm", "start"]

Any help would be greatly appreciated!

By that, do you mean my computer?? It's a Mac on Apple Silicon — StephenKernan, May 02 '21 at 23:34
This is also useful: https://stackoverflow.com/questions/65365797 (install `qemu-user-static`) — McAngus, Sep 23 '21 at 19:16

score 142 · Answer 1 · answered Aug 15 '21 at 02:07

142

Short answer: docker buildx build --platform=linux/amd64 -t <image-name> .

answered Aug 15 '21 at 02:07

Ben Francom

1,605
2
8
7

4

thank you, this absolutely saved me. as has been discussed here and in another [thread](https://stackoverflow.com/questions/69739201/docker-error-standard-init-linux-go228-exec-user-process-caused-exec-format), this error is caused by building on Apple Silicon (ARM) architecture with an intention to deploy on AWS ECS (for me) that is x86 based. a binary built using different architectures is NOT compatible. – jacob Mar 01 '23 at 13:42

score 57 · Accepted Answer · answered May 02 '21 at 23:38

57

I think you've identified your problem.

You're building your images on Apple's M1 chip, which is an ARM architecture. Fargate is probably running on the more common Intel x86-64 architecture. Images you build locally on your Mac aren't going to be able to run there.

The easiest solution is probably to have your images build automatically in Docker Hub (or use a Github Action to build them in Github).

I don't have all the details about how you're building and deploying your images, so it's possible I'm missing some details.

answered May 02 '21 at 23:38

larsks

277,717
41
399
399

Ahhh that makes sense. Forgive me, I'm relatively uninformed when it comes to some of the architecture discussion. So to make sure I understand, it sounds like the fact that I am building the container on my machine locally means that it is being created in a way that is incompatible with Fargate and that as a result I should use one of your suggestions to build the containers in a different architecture? Am I understanding you correctly? – StephenKernan May 02 '21 at 23:42
1

Yes. Containers are not ubiquitous when it comes to different CPU architectures or operating systems. You can't run a windows containers on Linux (and viceversa) and you can't run a Linux x86 container on a Linux ARM system (and viceversa). There is a roadmap item to add ARM support for Fargate [here](https://github.com/aws/containers-roadmap/issues/793). Feel free to +1 that request. – mreferre May 03 '21 at 06:58
1

Just to be clear, this doesn't really have anything to do with containers. You're compiling code for your ARM platform and trying to run it on an X86-64 platform: you would see the same error with or without containers. – larsks May 03 '21 at 11:07
You can use ARM architectures in in ECS, so you don't necessarily have to build it on a server. – Geoffrey Burdett Aug 04 '22 at 23:26

score 7 · Answer 3 · answered Jan 18 '23 at 09:03

On Mac M1 use:

FROM --platform=linux/amd64  alpine:3.10

Annoying pitfall I'm in every day. I feel your pain. Your docker desktop should show the AMD64 badge. I feel this should be the other way around, it should show the "with nothing and no-one compatible M1" badge when compiling local. Or both.

score 6 · Answer 4 · answered Jan 23 '22 at 22:27

As other answers have mentioned, this can happen when you build the docker image using an ARM processor and try to run on an x86 processor in Fargate.

As of 2021-11-23 Fargate now supports ARM architecture, but it can be a little tricky to find. Make sure you enable the "New ECS Experience" and then you'll see the option to use ARM when creating a new task definition.

Source: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-arm64.html

score 4 · Answer 5 · answered May 07 '21 at 12:05

For anyone who comes back here, Docker now has ships with an experimental, multi-arch build command called buildx and you can specify the architectures and ship these containers together. This resolved my problem, so huge thank you to those above for pointing me in the right direction!

score 1 · Answer 6 · answered Oct 16 '21 at 23:38

Images build on a particular architecture do not work on other architecture.

I tried to build an image on Mac M1 chip and pushed it to dockerhub. I am trying to run a container on EKC cluster with Amazon Linux worker nodes which eventually failed. Below is the solution I found and worked perfectly for me.

Flask Application

#!/usr/bin/env python3
from flask import Flask
app = Flask("basicApp")
@app.route("/")
def sayHello():
    return "<h4 style='color:teal'>Hello World!</h4>"
if __name__ == "__main__":
    app.run()

Docker file

FROM python:3.7-alpine

COPY flaskApp/app.py /app.py
RUN chmod 755 /app.py
RUN pip3 install flask

ENTRYPOINT ["python3", "-m", "flask", "run", "--host=0.0.0.0", "-p", "3000"]

Kubernetes File

---
apiVersion: v1
kind: Pod
metadata:
  name: flask-app
  labels:
    name: flask-app
spec:
  containers:
  - name: flask-app
    image: vikrampruthvi5/flask-app:1.1
    ports:
      - containerPort: 3000

Commands used docker buildx build --platform=linux/amd64 -t vikrampruthvi5/flask-app:1.1 . docker push vikrampruthvi5/flask-app:1.1

score 0 · Answer 7 · answered Sep 01 '22 at 17:53

0

For some reason AWS Sage Maker does not like when we build from mac m1 but using target platform linux/amd64. Same code works if I use an intel mac instead.

answered Sep 01 '22 at 17:53

xesf

101
1
3

'exec user process caused: exec format error' in AWS Fargate Service

7 Answers7

Linked