What can I do to prevent an http 403 error in python requests in the docker container that does not occur outside the container?

Question

I am sending requests to web addresses and processing the responses. However, my program that uses requests and runs in a Docker container on my raspberry-pi only gets HTTP 403 errors no matter what address. If I use curl to call the address from within the Linux console of the Pi I get the normal response (http code 200).

How can I debug this and fix the error?

I used the following test.py:

import requests
response = requests.get("https://www.ebay.de", headers={"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"})
print(response.status_code) #-> gets 403 http error

and the following Dockerfile:

FROM arm32v7/python:3.9-buster
WORKDIR /opt/test
COPY . .
RUN openssl version  #result is OpenSSL 1.1.1n  15 Mar 2022
RUN pip install requests==2.25.0
CMD ["python", "test.py"]

working curl

curl https://www.ebay.de

The hint from POST request - works on local (200), but not on docker (403) - Python3 unfortunately did not help.

Jimpsoni · Answer 1 · 2022-10-21T09:49:58.850

The website detects that you are using automation and won't let you access the information through script. I guess the problem is that you don't have good enough headers and thats why the website is blocking your request. My best advice is to use headless browser ( Which is slower but works for any site ) or try to figure out what headers you are missing. I guess Ebay is quite common site to scrape so they have effective blockers for that.

What can I do to prevent an http 403 error in python requests in the docker container that does not occur outside the container?

1 Answers1