Django collectstatic command fails in AWS Elastic Beanstalk Amazon Linux 2 Python 3 platform

Question

I've been struggling for several days now to deploy my Django application to Elastic Beanstalk using the Amazon Linux 2 Python 3.7 platform.

After managing to deploy the app I can't run the command python3 manage.py collectstatic --noinput in order for nginx to be able to serve the static files and thus have all the necessary CSS styles for the Django Admin page.

Right now I have a file named "static.config" in the .ebextensions directory that has the following YAML code:

container_commands:
  collectstatic:
    command: |
      source $PYTHONPATH/activate
      python3 manage.py collectstatic --noinput

I added the line source $PYTHONPATH/activate after the first failed attempt following this answer.

I also have a file named "01_python.config" with the following code:

option_settings: 
  "aws:elasticbeanstalk:application:environment": 
    DJANGO_SETTINGS_MODULE: "backend_project.settings" 
    "PYTHONPATH": "/var/app/current:$PYTHONPATH"
  "aws:elasticbeanstalk:container:python": 
    WSGIPath: backend_project.wsgi:application
    NumProcesses: 3
    NumThreads: 20
  "aws:elasticbeanstalk:environment:proxy:staticfiles":
    "/static/": "static/"

This is the stacktrace I'm seeing in the cfn-init.log:

2020-11-04 14:06:29,217 [ERROR] Command collectstatic (source $PYTHONPATH/activate
python3 manage.py collectstatic --noinput  ) failed
2020-11-04 14:06:29,218 [ERROR] Error encountered during build of postbuild_1_eduvaluer_api: Command collectstatic failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command collectstatic failed
2020-11-04 14:06:29,218 [ERROR] -----------------------BUILD FAILED!------------------------
2020-11-04 14:06:29,218 [ERROR] Unhandled exception during build: Command collectstatic failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command collectstatic failed

One question that comes to my mind when reading this trace is: why is python 2 being used to execute any scripts?

If anyone managed to solve this issue I would very much appreciate any help. All the documentation and answers I could find where quite outdated (for the Amazon Linux Python 3.6 platform).

The only static files my Django app needs are the styles for the Django Admin page as the application is the backend of an Angular application.

Thank you in advance!

Edit:

"PYTHONPATH":"/var/app/current:$PYTHONPATH". I assume $PYTHONPATH should be an environment variable but when I execute the command env to see the environment variables of the server, PYTHONPATH is not one of them.
As requested in the comments I tried to ssh into de EB instance and run the python3 manage.py collectstatic --noinput command. This is what I got:

(staging) [ec2-user@ip-172-31-94-128 current]$ python3 manage.py collectstatic --noinput /var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/environ/environ.py:639: UserWarning: Error reading /var/app/current/.env - if you're not configuring your environment separately, check this. "environment separately, check this." % env_file) Traceback (most recent call last): File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/environ/environ.py", line 273, in get_value value = self.ENVIRON[var] File "/usr/lib64/python3.7/os.py", line 681, in getitem raise KeyError(key) from None KeyError: 'SECRET_KEY'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 23, in <module>
    main()
  File "manage.py", line 19, in main
    execute_from_command_line(sys.argv)
  File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/core/management/__init__.py", line 231, in fetch_command
    settings.INSTALLED_APPS
  File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/conf/__init__.py", line 76, in __getattr__
    self._setup(name)
  File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/conf/__init__.py", line 63, in _setup
    self._wrapped = Settings(settings_module)
  File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/conf/__init__.py", line 142, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/app/current/backend_project/settings.py", line 45, in <module>
    SECRET_KEY =env("SECRET_KEY")
  File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/environ/environ.py", line 123, in __call__
    return self.get_value(var, cast=cast, default=default, parse_default=parse_default)
  File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/environ/environ.py", line 277, in get_value
    raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the SECRET_KEY environment variable

I activated the virtual environment before running the command and I know the environment variables (including the SECRET_KEY) are loaded because when I ran the command /opt/elasticbeanstalk/bin/get-config environment all of them were printed on the console. This keeps getting more and more confusing.

Another thing that could be useful is that all the static files seem to be in the /static directory.

In my settings.py file I have the following:

STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')

From what I understood this should indicate Django to look for the static files in the static directory. Perhaps I understood wrong and here lies the problem.

if not done yets, have a look at this tuto : https://realpython.com/deploying-a-django-app-and-postgresql-to-aws-elastic-beanstalk/ — cizario, Nov 04 '20 at 16:45
I have looked at almost every tutorial out there, the problem is that most of them, like the one you referenced, are outdated. AWS released AL2 Python 3.7 this year. The most recent tutorial I could find was this one: https://justabout.cloud/how-to-deploy-a-django3-application-on-elastic-beanstalk-with-python3-amazon-linux-2-platform/ It does not cover the subject of this question. — Zufra, Nov 04 '20 at 17:04
Why the command is failing? Do you know? Can you ssh into the EB instance and run it manually to verify? — Marcin, Nov 04 '20 at 21:58
@Marcin I updated the question with what happened when I tried to do it. — Zufra, Nov 05 '20 at 00:09
When you ssh to the instance, you have to load your env variables using `export $(cat /opt/elasticbeanstalk/deployment/env | xargs)`. PYTHONPATH should be then avaiable. — Marcin, Nov 05 '20 at 00:16
@Marcin I just updated the question, please refer to the edit 2 to see if any of that information is helpful. — Zufra, Nov 05 '20 at 00:19
@Marcin I ran that command and the PYTHONPATH environment variable wasn't displayed. In fact, none of the project variables are there. They only appear when I run the command /opt/elasticbeanstalk/bin/get-config environment — Zufra, Nov 05 '20 at 00:23
If you run the command as root user `export $(cat /opt/elasticbeanstalk/deployment/env | xargs)` does not display anything, but it loads all env variables. To check the variables use `env` — Marcin, Nov 05 '20 at 00:28
@Marcin by root user you mean the AWS account's root user? I deleted its access keys because AWS suggested it, but I can try creating new ones. Right now, when I execute that command I get the list of environment variables, each one preceded by "declare -x". — Zufra, Nov 05 '20 at 00:36
@Marcin I'm sorry, my Linux knowledge is very limited. It did what you said it would do but the value of the PYTHONPATH variable is still the one I specified in my question. — Zufra, Nov 05 '20 at 00:48
Instead of using `.ebextensions` for `collectstatic`, you could try creating a `.platform` hook, as described [here](https://stackoverflow.com/a/65199647). I believe using hooks is now considered best practice. About the `PYTHONPATH` env variable: this is created automatically (so you can remove it from your `option_settings`), but it is only available in platform hooks and in your app, unless you source it explicitly. See [AWS Knowledge Center](https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-linux2/) for details. — djvg, Dec 09 '20 at 13:44

Nabil Ahmed Khatri · Answer 1 · 2020-11-22T07:35:18.040

Hey I just deployed my django app successfully with static files. Here's what my config looks like.

django.config

container_commands:
  01_collectstatic:
    command: "source $PYTHONPATH/activate && python manage.py collectstatic --noinput"
option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: appname.wsgi:application
  aws:elasticbeanstalk:environment:proxy:staticfiles:
    /static: static

settings.py

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.1/howto/static-files/
STATIC_URL = '/static/'
STATIC_ROOT = 'static'

The environment variables are for the ebs environment not the EC2 instance itself I think. I also don't see PYTHONPATH when I ssh into the EC2 instance but I do see it in the environment variables for my ebs environment in Environment > Configuration > Software on AWS Console

/var/app/venv/staging-LQM1lest/bin

This is my file structure:

┣ .ebextensions/
┃ ┗ django.config
┣ .elasticbeanstalk/
┃ ┗ config.yml
┣ appname/

Ben Sturmfels · Answer 2 · 2021-07-23T02:57:48.993

The trick here is that the full output of container_commands in /var/log/cfn-init-cmd.log (Amazon Linux 2 Elastic Beanstalk released November 2020).

See related answer Running Django migrations when deploying to Elastic Beanstalk for full details.

PS. I highly recommend the whitenoise package rather than explicitly configuring aws:elasticbeanstalk:environment:proxy:staticfiles. This simplifies your configuration because all files are served though Django and closely aligns the behaviour of your development and production environments.

Django collectstatic command fails in AWS Elastic Beanstalk Amazon Linux 2 Python 3 platform

2 Answers2

Linked