Django collectstatic not working on production with S3, but same settings work locally

Question

I've been moving around some settings to make more defined local and production environments, and I must have messed something up.

Below are the majority of relevant settings. If I move the production.py settings (which just contains AWS-related settings at the moment) to base.py, I can update S3 from my local machine just fine. Similarly, if I keep those AWS settings in base.py and push to production, S3 updates appropriately. In addition, if I print something from production.py, it does print. However, if I make production.py my "local" settings on manage.py, or when I push to Heroku with the settings as seen below, S3 is not updating.

What about my settings is incorrect? (Well, I'm sure a few things, but specifically causing S3 not to update?)

Here's some relevant code:

__init__.py (in the directory with base, local, and production)

from cobev.settings.base import *

base.py

INSTALLED_APPS = [
    ...
    'whitenoise.runserver_nostatic',
    'django.contrib.staticfiles',
    ...
    'storages',
]

...

STATIC_URL = '/static/'

STATICFILES_DIRS = [os.path.join(BASE_DIR, "global_static"),
                    os.path.join(BASE_DIR, "media", )
                    ]

MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
MEDIA_URL = '/media/'

local.py

# local_settings.py
from .base import *

...

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

production.py

from .base import *

# AWS Settings

AWS_ACCESS_KEY_ID = config('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = config('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = 'cobev'

AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'

AWS_DEFAULT_ACL = 'public-read'

STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'cobev.storage_backends.MediaStorage'
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)

ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'

# End AWS

wsgi.py

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "cobev.settings.production")

application = get_wsgi_application()

from whitenoise.django import DjangoWhiteNoise
application = DjangoWhiteNoise(application)

manage.py

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "cobev.settings.local") 
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)

Where are `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` coming from? Have you checked that these are correctly defined in the Heroku environment? Do the keys contain any special characters (`$`, `@`) that haven't been escaped (this will cause problems on Heroku)? — solarissmoke, Jan 03 '20 at 04:52
They are stored as environment variables on heroku. I’ll triple check, but those vars work if the AWS code is in base.py instead of production.py. — hookedonwinter, Jan 03 '20 at 14:59
What updates the S3? Is it a django management command or you are updating S3 from Django admin or etc on files uploads or etc? — Alexandr Shurigin, Jan 06 '20 at 17:32
@AlexandrShurigin I think `storages` takes care of that with the `collectstatic` command on push to heroku (or manually) — hookedonwinter, Jan 06 '20 at 17:53
Well, First maybe I'm wrong but, why try to use WhiteNoise when you what to use S3 ???? That makes no sense for me. Check the doc : http://whitenoise.evans.io/en/legacy-2.x/index.html 1st line — Sigdev, Jan 06 '20 at 18:21
@Sigdev likely legacy from setting up heroku a while back. I'll look into that! — hookedonwinter, Jan 06 '20 at 18:28

score 2 · Accepted Answer · answered Jan 06 '20 at 18:02

Ok, let me try, as discovered in comments from the question, you do S3 update using collectstatic, but this is a management command which is called using manage.py file where you set cobev.settings.local as settings which are not equal to cobev.settings.production which is used for wsgi.py file.

I think you should manage your settings file using, normal Django way, OS environment variable named DJANGO_SETTINGS_MODULE.

For sure you should be able to set it in any production environment you are running.

Sigdev · Answer 2 · 2020-01-06T19:06:30.070

Then if that can help here is my config for AWS/Django/S3 in production :

Common Static config :

# STATIC FILE CONFIGURATION
# ------------------------------------------------------------------------------
# See: https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT = str(ROOT_DIR('staticfiles'))

# See: https://docs.djangoproject.com/en/dev/ref/settings/#static-url
STATIC_URL = '/static/'

# See: https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#std:setting-STATICFILES_DIRS
STATICFILES_DIRS = [
    str(APPS_DIR.path('static'))
]

# See: https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#staticfiles-finders
STATICFILES_FINDERS = [
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder'
]

Common Media config :

# MEDIA CONFIGURATION
# ------------------------------------------------------------------------------

# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-root
MEDIA_ROOT = str(APPS_DIR('media'))

# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-url
MEDIA_URL = '/media/'

Production Static config :

# STATIC CONFIG PRODUCTION
# ------------------------------------------------------------------------------
# See: http://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html

AWS_STORAGE_BUCKET_NAME = 'mybucket-name-production'
AWS_ACCESS_KEY_ID = 'YOUR_KEY_ID'
AWS_SECRET_ACCESS_KEY = 'YOUR_SECRET_KEY'
AWS_S3_HOST = "s3.amazonaws.com"
AWS_S3_URL = 'https://{bucker_name}.s3.amazonaws.com/'.format(bucker_name=AWS_STORAGE_BUCKET_NAME)

AWS_LOCATION = 'static/'

AWS_S3_URL_PROTOCOL = 'https:'
AWS_S3_CUSTOM_DOMAIN = 'static.mydomain.com' # I use sub domaine to serve static 

STATIC_URL = '{AWS_S3_URL_PROTOCOL}//{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}'.format(
    AWS_S3_URL_PROTOCOL=AWS_S3_URL_PROTOCOL,
    AWS_S3_CUSTOM_DOMAIN=AWS_S3_CUSTOM_DOMAIN,
    AWS_LOCATION=AWS_LOCATION)

AWS_QUERYSTRING_AUTH = False

AWS_IS_GZIPPED = True
AWS_EXPIREY = 60 * 60 * 24 * 14

# For s3boto
AWS_HEADERS = {
    'Cache-Control': 'max-age=%d, s-maxage=%d, must-revalidate' % (AWS_EXPIREY, AWS_EXPIREY)
}

# For s3boto3
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=%d' % AWS_EXPIREY,
}

AWS_PRELOAD_METADATA = True
#AWS_S3_FILE_OVERWRITE = True

STATICFILES_STORAGE = 'config.storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'config.storages.DefaultStorage'



# MEDIA S3 CONFIG PRODUCTION
# --------------------------------------------------------------------------------

AWS_MEDIA_DIR = 'media'
MEDIA_URL = AWS_S3_URL + AWS_MEDIA_DIR + '/'
MEDIA_ROOT = MEDIA_URL

Here is my StaticStorage Class :

from storages.backends.s3boto3 import S3Boto3Storage


class StaticStorage(S3Boto3Storage):
    location = 'static'
    file_overwrite = False


class DefaultStorage(S3Boto3Storage):
    location = ''
    file_overwrite = False

After That I add command in the folder .ebextensions in .config file for collectstatic :

# ./ebextensions/02_container_commands.config file : 

container_commands:
  0.3.0.push.static.to.s3:
    command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --ignore=.scss --noinput"
    leader_only: true
    ignoreErrors: true

score 2 · Answer 3 · answered Jan 07 '20 at 08:51

It looks like you are using Whitenoise. Whitenoise allows django to serve its static files. It is different approach if you want to serve them from AWS.

So, you need to remove Whitenoise in order to utilize django-storages. Remove it from settings, middleware, wsgi.py, etc.

Also, you can remove everything from __init__.py in settings - settings file to be used is set by DJANGO_SETTINGS_MODULE env variable.

According to your STATICFILES_DIRS, media directory is included as static files. It is preferred for media to be served separately from static files (one of differences - static files are more likely to be cached and gziped) - i.e. also with AWS but from separate S3 bucket.

You may also add AWS CloudFront as CDN in front of your buckets later.

Django collectstatic not working on production with S3, but same settings work locally

3 Answers3