How to extract the HTTP error text from a requests response?

Question

I have the following code:

tries = 10
for n in range(tries):
    try:
        ....
        responsedata = requests.get(url, data=data, headers=self.hed, verify=False)
        responsedata.raise_for_status()
        ..
        if .... : 
            break   #exit loop condition

    except (ChunkedEncodingError, requests.exceptions.HTTPError) as e:
        print ("page #{0} run #{1} failed. Returned status code {2}. Msg: {3}. Retry.".format(page, n, responsedata.status_code, sys.exc_info()[0]))
        if n == tries - 1:
           raise e  # exit the process

The prints I see are:

page #53 run #0 failed. Returned status code 502. Msg: <class 'requests.exceptions.HTTPError'>. Retry.
page #1 run #1 failed. Returned status code 500. Msg: <class 'requests.exceptions.ChunkedEncodingError'>. Retry.

While this is Ok it doesn't give me actual information about the problem. The message just tell me the exception title.

If I print the: responsedata.text when exception happens I see:

 Returned status code 502. Message is: ...
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>502 - Web server received an invalid response while acting as a gateway or proxy server.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
...

This is a giant message most of it is garbage but it also says: 502 - Web server received an invalid response while acting as a gateway or proxy server. can I access this message and also print it to my log?

@PedroLobito I'm aware of that but the API can raise more types or errors, some can be unique. I'm asking how to access the message using request package — Programmer120, Sep 04 '18 at 13:32

score 47 · Accepted Answer · answered Sep 04 '18 at 13:33

47

You can access the response's status code using responsedata.status_code and its textual description via responsedata.reason (see more in http://docs.python-requests.org/en/master/api/)

answered Sep 04 '18 at 13:33

Vitor Baptista

2,016
1
23
28

Nice but it contains a diffrent message. For example it says: `Bad Gateway` instead of `Web server received an invalid response while acting as a gateway or proxy server.` – Programmer120 Sep 04 '18 at 13:47
1

That message is in the response's title, added by the webserver. In other words, it's specific to this URL you're accessing, not to any webserver (as my answer). If you want to extract it, you'll need to parse the HTML and extract its `title` tag. You can see how on https://stackoverflow.com/questions/26812470/how-to-get-page-title-in-requests. – Vitor Baptista Sep 04 '18 at 13:53
1

Adding to my comment, it seems this message is just the default message IIS adds for status code 500. It doesn't add more information, it's just more verbose. Considering that extracting it would involve parsing the HTML, I'd suggest just logging the HTTP Error code, as it has the same information. If you want to extract it anyway, the solution is linked in my previous comment. – Vitor Baptista Sep 04 '18 at 13:58
When raised with raise_for_status(), e.reason contains a default message that's put there by the requests library, [source code for requests](https://github.com/psf/requests/blob/3.0/requests/models.py#L1024). user1898153's answer gets to the actual response from the exception handler. – mijiturka Nov 23 '22 at 11:22

score 13 · Answer 2 · answered Dec 17 '21 at 21:40

13

If your endpoint returns a detailed, application-specific error message in the response body, you can make use of the fact that a requests HTTPError retains a reference to the Response that caused it to be raised:

from requests.exceptions import HTTPError

try:
    # Some code that makes requests
except HTTPError as e:
    print(e.response.text)

answered Dec 17 '21 at 21:40

user1898153

463
6
12

This is gold.Thank you – atomscale Nov 07 '22 at 21:26
For me, exceptions handle errors at code side and can't handle API response. Eg: with a bad route, no Exception is raised , just an error message in the response `try: req = requests.post(endpoint, headers=head, data=json.dumps(payload)) print(f'{req.text} - OK') except Exception as e: print(f'{req.text} - KO {e}')`. I this case, not exception. The `TRY`is ok and the`req.text`only shows `bad route` – Abpostman1 Mar 28 '23 at 14:06

How to extract the HTTP error text from a requests response?

2 Answers2

Linked