I’m trying to add a static directory of files, to contain javascript, css, and the like) to my Django app that I’m deploying on Heroku.
The storage is being done with AWS S3 and I know this part is working because before trying to add this static directory, command heroku run python manage.py collectstatic worked just fine, putting all the admin site’s assets in my AWS S3 bucket.
Basically I added a directory called ‘static’ at the same level as my project’s main application, which is called ‘core’. I was hoping when I ran collectstatic, the files in this directory would similarly be placed in my AWS S3 bucket, under ‘static’.
But this is what I get when I try to deploy the app on Heroku...
(primedminds-FA7a2hIW) bash-3.2$ git push heroku master
Counting objects: 2, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 236 bytes | 236.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Python app detected
remote: ! The latest version of Python 3.6 is python-3.6.6 (you are using python-3.6.5, which is unsupported).
remote: ! We recommend upgrading by specifying the latest version (python-3.6.6).
remote: Learn More: https://devcenter.heroku.com/articles/python-runtimes
remote: Skipping installation, as Pipfile.lock hasn't changed since last deploy.
remote: -----> $ python manage.py collectstatic --noinput
remote: /app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
remote: """)
remote: Traceback (most recent call last):
remote: File "manage.py", line 10, in <module>
remote: execute_from_command_line(sys.argv)
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
remote: utility.execute()
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
remote: self.fetch_command(subcommand).run_from_argv(self.argv)
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
remote: self.execute(*args, **cmd_options)
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
remote: output = self.handle(*args, **options)
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 189, in handle
remote: collected = self.collect()
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 105, in collect
remote: for path, storage in finder.list(self.ignore_patterns):
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/finders.py", line 125, in list
remote: for path in utils.get_files(storage, ignore_patterns):
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/utils.py", line 28, in get_files
remote: directories, files = storage.listdir(location)
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/core/files/storage.py", line 313, in listdir
remote: for entry in os.listdir(path):
remote: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/build_c0914e5498ba1b53e00fb3adb7ce14d8/static'
remote:
remote: ! Error while running '$ python manage.py collectstatic --noinput'.
remote: See traceback above for details.
remote:
remote: You may need to update application code to resolve this error.
remote: Or, you can disable collectstatic for this application:
remote:
remote: $ heroku config:set DISABLE_COLLECTSTATIC=1
remote:
remote: https://devcenter.heroku.com/articles/django-assets
remote: ! Push rejected, failed to compile Python app.
remote:
remote: ! Push failed
remote: Verifying deploy...
remote:
remote: ! Push rejected to primedminds.
remote:
To https://git.heroku.com/primedminds.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/primedminds.git'
And it complains of a different missing directory when I do the collectstatic command manually. Note that I have no app named ‘app’ and there is no mention of any ‘app' in my settings file:
(primedminds-FA7a2hIW) bash-3.2$ heroku run python manage.py collectstatic
Running python manage.py collectstatic on ⬢ primedminds... up, run.7871 (Free)
/app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
You have requested to collect static files at the destination
location as specified in your settings.
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 189, in handle
collected = self.collect()
File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 105, in collect
for path, storage in finder.list(self.ignore_patterns):
File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/finders.py", line 125, in list
for path in utils.get_files(storage, ignore_patterns):
File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/utils.py", line 28, in get_files
directories, files = storage.listdir(location)
File "/app/.heroku/python/lib/python3.6/site-packages/django/core/files/storage.py", line 313, in listdir
for entry in os.listdir(path):
FileNotFoundError: [Errno 2] No such file or directory: '/app/static’
This is my settings.py file:
"""
Django settings for primedminds project.
Generated by 'django-admin startproject' using Django 1.9.
For more information on this file, see
https://docs.djangoproject.com/en/1.9/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.9/ref/settings/
"""
import os
import socket
import django_heroku
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.9/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = ‘<redacted>'
# SECURITY WARNING: don't run with debug turned on in production!
if socket.gethostname().endswith('.local'): # True in your local computer
DEBUG = True
ALLOWED_HOSTS = ['localhost', '127.0.0.1',]
else:
DEBUG = False
ALLOWED_HOSTS = ['.herokuapp.com', 'primedminds.com',]
#ALLOWED_HOSTS = ['*']
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'console': {
'class': 'logging.StreamHandler',
},
},
'loggers': {
'django': {
'handlers': ['console'],
'level': os.getenv('DJANGO_LOG_LEVEL', 'DEBUG'),
},
},
}
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'import_export',
'storages',
'core',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'primedminds.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [os.path.join(BASE_DIR, 'templates')],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'primedminds.wsgi.application'
# Database
# https://docs.djangoproject.com/en/1.9/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': os.environ['DATABASE_NAME'],
'USER': os.environ['DATABASE_USER'],
'PASSWORD': os.environ['DATABASE_PASSWORD'],
'HOST': '',
'PORT': '',
}
}
# Password validation
# https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/1.9/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'America/Los_Angeles'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.9/howto/static-files/
STATIC_URL = '/static/'
STATIC_ROOT = os.path.normpath(os.path.join(BASE_DIR, 'static'))
STATICFILES_DIRS = [
os.path.normpath(os.path.join(BASE_DIR, 'static')),
]
# AWS S3
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
AWS_STORAGE_BUCKET_NAME = 'primedminds'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'primedminds.storage_backends.MediaStorage'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
#ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
# Import export
#IMPORT_EXPORT_USE_TRANSACTIONS = True
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
# Configure Django App for Heroku.
#django_heroku.settings(locals(), staticfiles=True)
django_heroku.settings(locals())
Update 1: I’d forgotten that I’d set up a .gitignore file with static and staticfiles listed. So it was behaving as if there weren’t any files to upload. Once I deleted those lines from the .gitignore it proceeded onwards.
Now the question becomes why the files aren’t being saved to the AWS S3 bucket, as I expected. Instead they seem to be going to a staticfiles directory under /tmp. Here’s what’s happening:
(primedminds-FA7a2hIW) bash-3.2$ git push heroku master
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 586 bytes | 586.00 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Python app detected
remote: ! The latest version of Python 3.6 is python-3.6.6 (you are using python-3.6.5, which is unsupported).
remote: ! We recommend upgrading by specifying the latest version (python-3.6.6).
remote: Learn More: https://devcenter.heroku.com/articles/python-runtimes
remote: Skipping installation, as Pipfile.lock hasn't changed since last deploy.
remote: -----> $ python manage.py collectstatic --noinput
remote: /app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
remote: """)
remote: 148 static files copied to '/tmp/build_53bed4ea57eaa119caf33247adadfed7/staticfiles', 178 post-processed.
remote:
remote: -----> Discovering process types
remote: Procfile declares types -> web
remote:
remote: -----> Compressing...
remote: Done: 193.6M
remote: -----> Launching...
remote: Released v85
remote: https://primedminds.herokuapp.com/ deployed to Heroku
remote:
remote: Verifying deploy... done.
To https://git.heroku.com/primedminds.git
fc1cde0..0033cfe master -> master
Update 2: It turns out that at least one of the reasons it was not saving to AWS was because it wanted to use WhiteNoise local file storage (whitenoise.storage.CompressedManifestStaticFilesStorage) not the AWS storage middleware I’d set up in my settings file with:
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage’
The culprit was this line at the end of the settings file:
django_heroku.settings(locals())
That line overrides my STATICFILES_STORAGE variable. So I changed it to:
django_heroku.settings(locals(), staticfiles=False)
I thought that would solve it, but collectstatic still isn’t putting the files I have in my STATICFILES_DIRS in my AWS S3 bucket. I ran this command on the Heroku server:
django-admin collectstatic --settings=primedminds.settings
And got the following output:
~ $ django-admin collectstatic --settings=primedminds.settings
/app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
You have requested to collect static files at the destination
location as specified in your settings.
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
0 static files copied, 148 unmodified.
At least now it’s not just copying files locally. But still no files copied to AWS. I tried it with the --clear flag too, but no dice.
[Incidentally I had to use the django-admin command, not python manage.py collectstatic, since the latter command didn’t source the project settings file for some reason!]
This has gotten so long that I’m probably only talking to myself at this point, but for the record, the problem I’m up against now is still, why is collectstatic not behaving as I’d expect, putting the files I have in my STATICFILES_DIRS into my AWS S3 bucket?
