0

When running my flask app, which uses Python's subprocess to use scrapy within a flask app as specified here (How to integrate Flask & Scrapy?), from a Docker Container and calling the appropriate endpoints specified in my flask app, I receive the error message: ERR_EMPTY_RESPONSE. Executing the flask app outside of my docker container (python app.py, where app.py has my flask code), everything works as intended and my spiders are called using subprocess within the flask app.

Instead of using flask & subprocess to call my spiders within a web app, I tried using twisted & twisted-klein python libraries, with the same result when called from a docker Container. I have also created a new, clean scrapy project, meaning no specific code of my own, just the standard scrapy code and project structure upon creation. This resulted in the same error. I am not quite certain whether my approach is anti-pattern, since flask and scrapy are in bundled into run image, resulting in one container for two purposes.

Here is my server.py code. When executing outside a container (using python interpreter) everything works as intended. When running it from a container, then I receive the error message (ERR_EMPTY_RESPONSE).

# server.py
import subprocess
from flask import Flask
from medien_crawler.spiders.firstclassspider import FirstClassSpider


app = Flask(__name__)

@app.route("/")
def return_hello():
    return "Hello!"

@app.route("/firstclass")
def return_firstclass_comments():
    spider_name = "firstclass"
    response = subprocess.call(['scrapy', 'crawl', spider_name, '-a', 'start_url=https://someurl.com'])
    return "OK!"

if __name__ == "__main__":
    app.run(debug=True)


FROM python:3
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 5000
CMD [ "python", "./server.py" ]

Finally I run docker run -p 5000:5000 . It does not work. Any ideas?

fcs
  • 1
  • Can you run `docker logs [your_container_name]` from the command line and post the output? That might help. – Matt L. Aug 14 '19 at 14:00
  • I suspect it might be the base image that you're using. `python` images are limited compared to a full linux image. Might be worth seeing if `subprocess` is actually calling `scrapy`; – fixatd Aug 14 '19 at 14:01
  • Here is the output from the docker logs (standard debugging output from flask): ```docker * Serving Flask app "server" (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: on * Running on http://localhost:5000/ (Press CTRL+C to quit) * Restarting with stat * Debugger is active! ``` – fcs Aug 14 '19 at 14:20
  • That log reflects a newly run container. Run the container, visit http://localhost:5000/firstclass and rerun docker logs command once it's errored? – Matt L. Aug 14 '19 at 16:50

1 Answers1

0

Please try it. .Dockerfile

FROM python:3.6
RUN apt-get update && apt-get install -y wget
WORKDIR /usr/src/app
ADD . /usr/src/app

RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 5000
CMD [ "python", "./server.py" ]
Reinis
  • 477
  • 1
  • 5
  • 13