1

Writing a script to get notifications when certain items are in stock, not trying to scalp, just trying to get items that have been out of stock for a while. There is no RPA part to this, my script just sends me a text and email of something I'm looking for is in stock. Using Python 3 (requests and beautifulsoup4).

On my local computer (windows 10) everything works fine, but on my servers (linode) where I want to run the script the request just hangs when the GET is sent to their servers. I'm using modified headers (including the user agent) so the request is basically the same in both environments. I know the IP of these servers are not blocked, so what else could it be?

Example request and headers:


headers =  {
        "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0",
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "accept-language": "en-US,en;q=0.5",
        "accept-encoding": "gzip, deflate, br",
        "connection": "keep-alive",
        "upgrade-insecure-requests": "1",
        "sec-fetch-dest": "document",
        "sec-fetch-mode": "navigate",
        "sec-fetch-site": "none",
        "sec-fetch-user": "?1"
    }


resp = requests.get("https://bestbuy.com",headers=headers)

Logging output

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bestbuy.com:443
send: b'GET / HTTP/1.1\r\nHost: bestbuy.com\r\nuser-agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0\r\naccept-encoding: gzip, deflate, br\r\naccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\r\nconnection: keep-alive\r\naccept-language: en-US,en;q=0.5\r\nupgrade-insecure-requests: 1\r\nsec-fetch-dest: document\r\nsec-fetch-mode: navigate\r\nsec-fetch-site: none\r\nsec-fetch-user: ?1\r\n\r\n'
reply: 'HTTP/1.1 301 Moved Permanently\r\n'
header: Content-Length: 0
header: Location: https://www.bestbuy.com/
header: Date: Sun, 26 Dec 2021 15:26:09 GMT
header: Connection: keep-alive
header: X-Frame-Options: SAMEORIGIN
header: Strict-Transport-Security: max-age=31536000
DEBUG:urllib3.connectionpool:https://bestbuy.com:443 "GET / HTTP/1.1" 301 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): www.bestbuy.com:443
send: b'GET / HTTP/1.1\r\nHost: www.bestbuy.com\r\nuser-agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0\r\naccept-encoding: gzip, deflate, br\r\naccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\r\nconnection: keep-alive\r\naccept-language: en-US,en;q=0.5\r\nupgrade-insecure-requests: 1\r\nsec-fetch-dest: document\r\nsec-fetch-mode: navigate\r\nsec-fetch-site: none\r\nsec-fetch-user: ?1\r\n\r\n'
ogmilk
  • 11
  • 3
  • your log is incomplete and you cut the best part. – diggusbickus Dec 26 '21 at 19:53
  • it stalls there, once the request sends I get no response and the code execution halts. No timeout, just nothing. Maybe configured my logger as depicted here: https://stackoverflow.com/questions/10588644/how-can-i-see-the-entire-http-request-thats-being-sent-by-my-python-application – ogmilk Dec 27 '21 at 14:33

0 Answers0