6

I'm trying to scrape my own site from my local server. But when I use python requests on it, it gives me a response 503. Other ordinary sites on the web work. Any reason/solution for this?

import requests 
url = 'http://127.0.0.1:8080/full_report/a1uE0000002vu2jIAA/'

r = requests.get(url)

print r

prints out 

<Response [503]>

After further investigation, I've found a similar problem to mine. Python requests 503 erros when trying to access localhost:8000

However, I don't think he's solved it yet. I can access the local website via the web browser but can't access using the requests.get function. I'm also using Django to host the server.

python manage.py runserver 8080

When I use: curl -vvv http://127.0.0.1:8080

* Rebuilt URL to: http://127.0.0.1:8080/
*   Trying 10.37.135.39...
* Connected to proxy.kdc.[company-name].com (10.37.135.39) port 8099 (#0)
* Proxy auth using Basic with user '[company-id]'
> GET http://127.0.0.1:8080/ HTTP/1.1
> Host: 127.0.0.1:8080
> Proxy-Authorization: Basic Y2FhNTc2OnJ2YTkxQ29kZQ==
> User-Agent: curl/7.49.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: BlueCoat-Security-Appliance
< Location:http://10.118.216.201
< Connection: Close
<
<HTML>
<HEAD><TITLE>Redirection</TITLE></HEAD>
<BODY><H1>Redirect</H1></BODY>
* Closing connection 0
Community
  • 1
  • 1
Kenny Ho
  • 383
  • 1
  • 5
  • 12
  • He did solve it - if you read the comments. *Found what was going on, we have an internal software was interfering with the port. So when I tried on my home pc it worked perfectly.* – Craicerjack Nov 07 '16 at 16:31
  • You have no port number specified in your url. Django runs default server at `http://127.0.0.1:8000/` so you should try `url = 'http://127.0.0.1:8000/full_report/a1uE0000002vu2jIAA/' ` – Craicerjack Nov 07 '16 at 16:34
  • didnt see the last line so the above command url should be `http://127.0.0.1:8080/....` – Craicerjack Nov 07 '16 at 16:40
  • I forgot to edit that part. Port 8080 doesn't work as well. I edited the post to reflect that. – Kenny Ho Nov 07 '16 at 16:44
  • What happens if you `curl -vvv http://127.0.0.1:8080`? – cwallenpoole Nov 07 '16 at 16:45
  • I edited my original post with the response. – Kenny Ho Nov 07 '16 at 16:51
  • That looks like some network software is interfering with your request. – Craicerjack Nov 07 '16 at 17:06
  • It looks like we came to same conclusion as the the similar thread I posted. I don't think it's possible to change the network software. I'm going to arrive to the same conclusion that it won't be possible to request a local url through the company network. Thanks everyone! – Kenny Ho Nov 07 '16 at 17:16

4 Answers4

17

I cannot request a local url using python requests because the company's network software won't allow it. This is a dead end and other avenues must be pursued.

EDIT: Working Solution

>>> import requests
>>> session = requests.Session()
>>> session.trust_env = False
>>> r = session.get("http://127.0.0.1:8080")
>>> r
<Response [200]>
Kenny Ho
  • 383
  • 1
  • 5
  • 12
  • Are you sure that it is a setting on your network and not a setting on your local device? What happens if you use `curl --noproxy '*' 127.0.0.1:8080`? – cwallenpoole Nov 13 '16 at 10:11
  • 1
    Yes, a proxy was blocking me for pinging myself. The solution is now posted! – Kenny Ho Nov 14 '16 at 14:27
8

Maybe you should disable your proxies in your requests.

import requests

proxies = {
  "http": None,
  "https": None,
}

requests.get("http://127.0.0.1:8080/myfunction", proxies=proxies)

ref:

  1. https://stackoverflow.com/a/35470245/8011839
  2. https://2.python-requests.org//en/master/user/advanced/#proxies
Chengcheng Zhang
  • 474
  • 5
  • 9
0

HTTP Error 503 means:

The Web server (running the Web site) is currently unable to handle the HTTP request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. Some servers in this state may also simply refuse the socket connection, in which case a different error may be generated because the socket creation timed out.

You may do following things:

  1. Check you are able to open URL in the browser
  2. If URL is opening, then check the domain in your code, it might be incorrect.
  3. If in browser also it is not opening, your site may be overloaded or server resources are full to perform request
Moinuddin Quadri
  • 46,825
  • 13
  • 96
  • 126
0

The most common cause of a 503 error is that a proxy host of some form is unable to communicate with the back end. For example, if you have Varnish trying to handle a request but Apache is down.

In your case, you have Django running on port 8080. (That's what the 8080 means). When you try to get content from 127.0.0.1, though, you're going to the default HTTP port (80). This means that your default server (Apache maybe? NginX?) is trying to find a host to serve 127.0.0.1 and can't find one.

You have two choices. Either you can update your server's configuration, or you can include the port in the URL.

url = 'http://127.0.0.1:8080/full_report/a1uE0000002vu2jIAA/'
cwallenpoole
  • 79,954
  • 26
  • 128
  • 166