How to host knowledge-repo (airbnb) on heroku?

Question

I tried to host a knowledge-repo (airbnb) on heroku.

I found several issues, e.g when trying to initialise the /app as knowledge repo repository:

I noticed that if I manually define the directory on heroku as knowledge repo using knowledge_repo --repo /app init, I can at least use the knowledge_repo --repo <repo_path> runserver command on the dyno. If I don't initialise the knowledge repo, I just get a wrong path error:

AssertionError: Provided path '/app' is not a valid repository.

It hence seems important to initialise the directory on heroku as knowledge repo app, i.e. knowledge_repo --repo ./example_repo init and I tried to accommodate for this requirement in the setup.py:

import subprocess
# define repo as knowledge repo in BASH
script = """echo yes "y" | knowledge_repo --repo /app init"""
subprocess.call(['sh', '-c', script])

But this does not work. The yes method is not implemented for knowledge repo. This now seems a hard problem to solve.

Full Error:

2017-02-26T20:59:58.780670+00:00 heroku[web.1]: State changed from crashed to starting
2017-02-26T21:00:09.750240+00:00 heroku[web.1]: Starting process  with command `gunicorn --log-file=- knowledge_app:app\(\) -w 3 -b  0.0.0.0:3572`
2017-02-26T21:00:14.022140+00:00 heroku[web.1]: State changed from starting to up
2017-02-26T21:00:16.240021+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path="/" host=carwow-knowledge-repo.herokuapp.com request_id=99f7867e-37ce-4417-8a04-53577e13e872 fwd="84.208.51.166" dyno=web.1 connect=5ms service=1158ms status=503 bytes=0
2017-02-26T21:00:27.907629+00:00 heroku[web.1]: Starting process with command `gunicorn --log-file=- knowledge_app:app\(\) -w 3 -b 0.0.0.0:58588`
2017-02-26T21:00:31.561254+00:00 heroku[web.1]: State changed from starting to up
2017-02-26T21:00:31.110520+00:00 app[web.1]: [2017-02-26 21:00:31 +0000] [4] [INFO] Starting gunicorn 19.6.0
2017-02-26T21:00:31.112610+00:00 app[web.1]: [2017-02-26 21:00:31 +0000] [4] [INFO] Listening at: http://0.0.0.0:58588 (4)
2017-02-26T21:00:31.112715+00:00 app[web.1]: [2017-02-26 21:00:31 +0000] [4] [INFO] Using worker: sync
2017-02-26T21:00:31.118097+00:00 app[web.1]: [2017-02-26 21:00:31 +0000] [9] [INFO] Booting worker with pid: 9
2017-02-26T21:00:31.162142+00:00 app[web.1]: [2017-02-26 21:00:31 +0000] [11] [INFO] Booting worker with pid: 11
2017-02-26T21:00:31.230349+00:00 app[web.1]: [2017-02-26 21:00:31 +0000] [13] [INFO] Booting worker with pid: 13
2017-02-26T21:00:35.074917+00:00 app[web.1]: Traceback (most recent call last):
2017-02-26T21:00:35.074937+00:00 app[web.1]:   File "/app/.heroku/python/bin/gunicorn", line 11, in <module>
2017-02-26T21:00:35.074940+00:00 app[web.1]:     sys.exit(run())
2017-02-26T21:00:35.074976+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
2017-02-26T21:00:35.075018+00:00 app[web.1]:     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
2017-02-26T21:00:35.075052+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 192, in run
2017-02-26T21:00:35.075095+00:00 app[web.1]:     super(Application, self).run()
2017-02-26T21:00:35.075118+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 72, in run
2017-02-26T21:00:35.075151+00:00 app[web.1]:     Arbiter(self).run()
2017-02-26T21:00:35.075174+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 218, in run
2017-02-26T21:00:35.075233+00:00 app[web.1]:     self.halt(reason=inst.reason, exit_status=inst.exit_status)
2017-02-26T21:00:35.075236+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 331, in halt
2017-02-26T21:00:35.075309+00:00 app[web.1]:     self.stop()
2017-02-26T21:00:35.075331+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 381, in stop
2017-02-26T21:00:35.075429+00:00 app[web.1]:     time.sleep(0.1)
2017-02-26T21:00:35.075509+00:00 app[web.1]:     self.reap_workers()
2017-02-26T21:00:35.075444+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 231, in handle_chld
2017-02-26T21:00:35.075531+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 506, in reap_workers
2017-02-26T21:00:35.075628+00:00 app[web.1]:     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
2017-02-26T21:00:35.075688+00:00 app[web.1]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
2017-02-26T21:00:35.147852+00:00 app[web.1]: Repository already exists. Do you want to convert it to a knowledge data repository? Note that this will override any existing `README.md` and `.knowledge_repo_config.py` files, and replace any submodule at `.resources`. (y/n) WARNING:knowledge_repo.repositories.gitrepository:Not updating existing repository. Aborting!

Answer 1

I think the problem was with the PORT, see heroku docs:

The web process must listen for HTTP traffic on $PORT, which is set by Heroku. EXPOSE in Dockerfile is not respected, but can be used for local testing. Only HTTP requests are supported.

I resolved it by dockerising the app and the content of the files is below:

content of setup.py

from setuptools import setup, find_packages
from pip.req import parse_requirements
install_reqs = parse_requirements("requirements.txt", session=False)

reqs = [str(ir.req) for ir in install_reqs]
setup(name='name_of_your_knowledge_repo',
         packages=find_packages(),
         install_reqs=reqs)

content of docker-python-setup.sh

#!/bin/sh

export PATH=/heroku/python/bin:$PATH
export PYTHONUNBUFFERED=true
export PYTHONHOME=/heroku/python
export LIBRARY_PATH=/heroku/vendor/lib:/heroku/python/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH=/heroku/vendor/lib:/heroku/python/lib:$LD_LIBRARY_PATH
export LANG=${LANG:-en_US.UTF-8}
export PYTHONHASHSEED=${PYTHONHASHSEED:-random}
export PYTHONPATH=${PYTHONPATH:-/app/};

exec $*

Content of Dockerfile

###
### This is mostly taken from github.com/heroku/docker-python, but the Docker Hub version of that
### now only supports Python 3, so we have an adapted copy here.
### The main change here is moving our code from /app/user to /app, and moving the python install
### from /app/.heroku to /heroku.
###

# Inherit from Heroku's stack
FROM heroku/cedar:14

# Which version of Python?
ENV PYTHON_VERSION python-2.7.11

# Add Python binaries to path.
ENV PATH /heroku/python/bin/:$PATH

# Create some needed directories
RUN mkdir -p /heroku/python /app/.profile.d
WORKDIR /app

# Install Python
RUN curl -s https://lang-python.s3.amazonaws.com/cedar-14/runtimes/$PYTHON_VERSION.tar.gz | tar zx -C /heroku/python

# Install Pip & Setuptools
RUN curl -s https://bootstrap.pypa.io/get-pip.py | /heroku/python/bin/python

# Export the Python environment variables in /heroku/python.sh
COPY docker-python-setup.sh /heroku/python.sh

ADD requirements.txt /app/
RUN /heroku/python/bin/pip install -r requirements.txt
ADD . /app

# Make locally-built containers run as not-root
# RUN useradd -m notroot
# USER notroot

ENTRYPOINT ["/heroku/python.sh"]
CMD /heroku/python/bin/knowledge_repo --repo /app deploy --port $PORT

Content of knowledge_app.py

from knowledge_repo import KnowledgeRepository

import subprocess
# define repo as knowledge repo in BASH
script = """echo yes "y" | knowledge_repo --repo /app init"""
subprocess.call(['sh', '-c', script])

app = KnowledgeRepository.for_uri('/app/.').get_app(config='server_config.py')

To deploy, I used the heroku guide for python app deployments and to update you need to do something like this:

Deploy on heroku via docker

1. Install Docker
2. cd into your local knowledge_repo dir: cd ~/repo/knowlegde_repo
3. docker build .
4. Install addin: heroku-container-registry add in.
5. heroku container:push web