15

I'm trying to deploy a project I've been working on with django. In development, i've been using SQLite, in production i'm trying to use MySQL.

Usually when I create the EB instance, everything runs fine, and the console says the status is OK. Upon trying to deploy (running eb deploy in ebcli), I get met with the following error

2020/06/18 15:59:50.357281 [INFO] Copying file /opt/elasticbeanstalk/config/private/rsyslog.conf to /etc/rsyslog.d/web.conf
2020/06/18 15:59:50.358945 [INFO] Running command /bin/sh -c systemctl restart rsyslog.service
2020/06/18 15:59:50.394223 [INFO] Executing instruction: PostBuildEbExtension
2020/06/18 15:59:50.394243 [INFO] No plugin in cfn metadata.
2020/06/18 15:59:50.394252 [INFO] Starting executing the config set Infra-EmbeddedPostBuild.
2020/06/18 15:59:50.394273 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-init -s arn:aws:cloudformation:eu-west-2:433403353655:stack/awseb-e-qamgvpp7ft-stack/3e6774d0-b17c-11ea-9476-0a5f6fd32d44 -r AWSEBAutoScalingGroup --region eu-west-2 --configsets Infra-EmbeddedPostBuild
2020/06/18 15:59:50.721919 [ERROR] Error occurred during build: Command 01_migrate failed

2020/06/18 15:59:50.721944 [ERROR] An error occurred during execution of command [app-deploy] - [PostBuildEbExtension]. Stop running the command. Error: Container commands build failed. Please refer to /var/log/cfn-init.log for more details. 

2020/06/18 15:59:50.721949 [INFO] Executing cleanup logic
2020/06/18 15:59:50.722079 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[]}]}

2020/06/18 15:59:50.722249 [INFO] Platform Engine finished execution on command: app-deploy

The culprit seems to be my db migration command, which is as follows, in '.ebextensions', named 'db-migrate.config'

container_commands:
  01_migrate:
    command: "django-admin.py migrate"
    leader_only: true

option_settings:
    aws:elasticbeanstalk:application:environment:
        DJANGO_SETTINGS_MODULE: djangomicroblog.settings

The original error message also points me to the log from '/var/log/cfn-init.log, error seen here is as follows

2020-06-18 14:18:23,279 [ERROR] -----------------------BUILD FAILED!------------------------
2020-06-18 14:18:23,279 [ERROR] Unhandled exception during build: Command 01_migrate failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command 01_migrate failed

I've tried searching for these error messages, but the results are very limited & don't seem to work.

The function in my settings.py for setting the database is also as follows

def get_db():
    try:
        return {
           'default': {
               'ENGINE': 'django.db.backends.mysql',
               'NAME': os.environ['RDS_DB_NAME'],
               'USER': os.environ['RDS_USERNAME'],
               'PASSWORD': os.environ['RDS_PASSWORD'],
               'HOST': os.environ['RDS_HOSTNAME'],
               'PORT': os.environ['RDS_PORT'],
           }
        }
    except:
        return {
            'default': {
                'ENGINE': 'django.db.backends.sqlite3',
                'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

DATABASES = get_db()

Any help would be greatly appreciated

BillieM
  • 195
  • 2
  • 11
  • You solved it? i have same issue. – Tim Jun 29 '20 at 20:53
  • 1
    I did yeah, it turned out the issue was that I setup the application for an Amazon linux 1 environment, but I was using an amazon linux 2 environment. Amazon linux 2 seems to be quite new, & i couldn't find a way to convert my application, so i remade the environment in amazon linux 1 (python 3.6 instead of 3.7). My other post has a bit more detail – BillieM Jun 30 '20 at 15:46
  • 1
    Solved the same way as me! Thanks for the answer, have a nice day. – Tim Jul 01 '20 at 08:44

1 Answers1

34

Edit 11/1/2022 - I posted this roughly same answer in two places that had the same question. Another user left an answer which appears to be more up-to-date, though I have not tested it. You can find that answer here: https://stackoverflow.com/a/65199647/5180047


Original Answer

I had this same issue, using Amazon Linux 2 with Python 3.7 just like OP. AWS has not updated their documentation to support the platform changes which made this very confusing. I found a fix, though I don't love it, but here is my solution:

TLDR - here's my solution. For how I got there, I've pasted in my notes as I debugged:

container_commands:
    01_migrate:
        command: "source /var/app/venv/*/bin/activate && python3 manage.py migrate"
        leader_only: true

Even though I don't love it, I have verified with AWS Support that this is in fact the recommended way to do this. You must source the python environment, as with AL2 they use virtual environments in an effort to stay more consistent.


From the AWS Announcement of Amazon Linux 2 in December I found that the installed python environment is stored in /var/app/venv/*/..

2020-07-22 19:20:41,376 [ERROR] -----------------------BUILD FAILED!------------------------
2020-07-22 19:20:41,376 [ERROR] Unhandled exception during build: Command 01_migrate failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command 01_migrate failed

I found a log file that actually tells me more useful information: /var/log/cfn-init-cmd.log

which gives me this error:

2020-07-22 21:08:32,757 P1620 [INFO] ============================================================
2020-07-22 21:08:32,757 P1620 [INFO] Command 01_migrate
2020-07-22 21:08:32,771 P1620 [INFO] -----------------------Command Output-----------------------
2020-07-22 21:08:32,772 P1620 [INFO]      File "manage.py", line 17
2020-07-22 21:08:32,772 P1620 [INFO]        ) from exc
2020-07-22 21:08:32,772 P1620 [INFO]             ^
2020-07-22 21:08:32,772 P1620 [INFO]    SyntaxError: invalid syntax
2020-07-22 21:08:32,772 P1620 [INFO] ------------------------------------------------------------
2020-07-22 21:08:32,772 P1620 [ERROR] Exited with error code 1

which is weird. This likely means I'm running the wrong python version. So I need to modify my command probably: so... the things I've tried are:

  1. python manage.py migrate
  2. python3 manage.py migrate

That gave me a better error message

2020-07-23 14:47:06,906 P14080 [INFO] ============================================================
2020-07-23 14:47:06,906 P14080 [INFO] Command 01_migrate
2020-07-23 14:47:06,936 P14080 [INFO] -----------------------Command Output-----------------------
2020-07-23 14:47:06,936 P14080 [INFO]   Traceback (most recent call last):
2020-07-23 14:47:06,936 P14080 [INFO]     File "manage.py", line 11, in main
2020-07-23 14:47:06,936 P14080 [INFO]       from django.core.management import execute_from_command_line
2020-07-23 14:47:06,936 P14080 [INFO]   ModuleNotFoundError: No module named 'django'
2020-07-23 14:47:06,937 P14080 [INFO]   
2020-07-23 14:47:06,937 P14080 [INFO]   The above exception was the direct cause of the following exception:
2020-07-23 14:47:06,937 P14080 [INFO]   
2020-07-23 14:47:06,937 P14080 [INFO]   Traceback (most recent call last):
2020-07-23 14:47:06,937 P14080 [INFO]     File "manage.py", line 28, in <module>
2020-07-23 14:47:06,937 P14080 [INFO]       main()
2020-07-23 14:47:06,937 P14080 [INFO]     File "manage.py", line 17, in main
2020-07-23 14:47:06,937 P14080 [INFO]       ) from exc
2020-07-23 14:47:06,937 P14080 [INFO]   ImportError: Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Did you forget to activate a virtual environment?
2020-07-23 14:47:06,937 P14080 [INFO] ------------------------------------------------------------
2020-07-23 14:47:06,937 P14080 [ERROR] Exited with error code 1

but it seems the python libraries aren't installed when I run with python3, so it's probably referencing the system python3 installation and not the virtual environment.

Little rant I left on this issue here: Python on Amazon Linux 2 platform · Issue #15 · aws/elastic-beanstalk-roadmap · GitHub which is due to.

  1. source /var/app/venv/*/bin/activate && python3 manage.py migrate I found the environment on the EC2 instance and manually sourced it to force it to use the correct python version. This appears to have solved the python environment issue. Now RDS seems to not have the database, but this seems much more fixable.
2020-07-23 15:26:32,016 P14702 [INFO] ============================================================
2020-07-23 15:26:32,016 P14702 [INFO] Command 01_migrate
2020-07-23 15:26:32,426 P14702 [INFO] -----------------------Command Output-----------------------
2020-07-23 15:26:32,427 P14702 [INFO]   Traceback (most recent call last):
2020-07-23 15:26:32,427 P14702 [INFO]     File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection
2020-07-23 15:26:32,427 P14702 [INFO]       self.connect()
2020-07-23 15:26:32,427 P14702 [INFO]     File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner
2020-07-23 15:26:32,427 P14702 [INFO]       return func(*args, **kwargs)
2020-07-23 15:26:32,427 P14702 [INFO]     File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/db/backends/base/base.py", line 197, in connect
2020-07-23 15:26:32,427 P14702 [INFO]       self.connection = self.get_new_connection(conn_params)
2020-07-23 15:26:32,427 P14702 [INFO]     File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner
2020-07-23 15:26:32,427 P14702 [INFO]       return func(*args, **kwargs)
2020-07-23 15:26:32,427 P14702 [INFO]     File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 185, in get_new_connection
2020-07-23 15:26:32,427 P14702 [INFO]       connection = Database.connect(**conn_params)
2020-07-23 15:26:32,428 P14702 [INFO]     File "/var/app/venv/staging-LQM1lest/lib64/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
2020-07-23 15:26:32,428 P14702 [INFO]       conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2020-07-23 15:26:32,428 P14702 [INFO]   psycopg2.OperationalError: FATAL:  database "ebdb" does not exist

for me, at this point I knew the migrate command had succeeded, so the solution is just:

container_commands:
    01_migrate:
        command: "source /var/app/venv/*/bin/activate && python3 manage.py migrate"
        leader_only: true

Of course, I had to connect to the RDS instance and create the database as well, but it worked from here.

Nick Brady
  • 6,084
  • 1
  • 46
  • 71
  • It's working for me. Just to highlight that "option_settings" part is not needed now. – David Weinberg Aug 11 '20 at 13:14
  • @DavidWeinberg Do you mean I can just remove everything under option_settings in my `.config` file? – Fawwaz Yusran Nov 15 '20 at 16:17
  • @FawwazYusran I think so. – David Weinberg Nov 16 '20 at 13:59
  • 2
    I am very grateful for your explanation. The migration was always giving me errors when deploying and the logs weren't helpful at all. I really can't understand why the official documentation gives us a command that won't work at all. Thank you very much. – Eduardo Matsuoka Nov 26 '20 at 15:19
  • I use `()` instead of doble quote "". `command: (source /var/app/venv/*/bin/activate && python3 manage.py migrate)` – aijogja Dec 19 '20 at 11:45
  • you save my days. Thanks! – mappy Dec 29 '20 at 13:44
  • I'm so grateful. You solved my problems and HEADACHES. wasting a lot of time looking for a solution and it worked as well. Thanks @Nick – Dedadino Jun 26 '21 at 13:38
  • Really glad to hear @Dedadino . Always makes me happy when my answers help! – Nick Brady Jun 28 '21 at 15:01
  • Stupid question... but what about makemigrations? – Josh Sep 11 '21 at 09:06
  • @josh the important thing is that you source the environment so that the correct python installation is referenced. so you could just swap `migrate` with `make migrations`. That said, you should make the migration files on your local dev environment, and put them in source control. You don't want you build system (beanstalk in this case) making them. The file should already be there for `migrate` to run essentially, so the question about `makemigrations` is moot. – Nick Brady Sep 11 '21 at 22:05
  • 2
    Thanks for the advice to check `var/log/cfn-init-cmd.log`. I had a different error, but the details were in there! – Josh Jan 16 '22 at 22:37
  • @NickBrady - if you have any insight into how to get around migrations completely (which I cannot do https://stackoverflow.com/questions/74649017/how-to-prevent-aws-eb-from-running-migration) I would greatly appreciate you sharing that. It seems like AWS EB is getting the command from somewhere I know not... – JET Dec 01 '22 at 22:57
  • Checking the `var/log/cfn-init-cmd.log` file helped me to solve my issue. Thanks a lot!! – Karthick Selvaraj Apr 12 '23 at 17:49