Python requests view before sending

Question

Can somebody please provide me with a way to see a request that I have generated before sending it to a server, here is the code:

import requests
import urllib2
import logging

from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

url = "https://10.1.1.254:4081/admin/api/jsonrpc/"
session = requests.Session()

data = {
    "jsonrpc": "2.0", "id": 1, "method": "Session.login",
    "params": {"userName": "test", "password":"test123"}
}
r = session.post(url, json=data, verify=False)

data = {"jsonrpc": "2.0", "id":3, "method":"Session.logout"}
r = session.post(url, json=data, verify=False)

So what I would like is to get that request sent with session.post, before Python sends it.

Answer 1

You have three options here:

• by using a prepared request,
• inspecting the request after it has been sent,
• you can enable logging in urllib3 as well as enable debugging in the http.client library.

They give you different levels of control over what is output.

Prepared requests

You can prepare a request, which gives you a new object with all the data set up in final form. You can look at the requests.PreparedRequest API docs for the full list of attributes.

Here is a sample function that simply formats that data as if it were sent over the network (except it doesn't actually use CRLF line separators, and replacing binary content with a placeholder string):

def format_prepped_request(prepped, encoding=None):
    # prepped has .method, .path_url, .headers and .body attribute to view the request
    encoding = encoding or requests.utils.get_encoding_from_headers(prepped.headers)
    body = prepped.body.decode(encoding) if encoding else '<binary data>' 
    headers = '\n'.join(['{}: {}'.format(*hv) for hv in prepped.headers.items()])
    return f"""\
{prepped.method} {prepped.path_url} HTTP/1.1
{headers}

{body}"""

Use a prepareded request instead of using session.[httpmethod](), and instead use session.send() to send out the prepared request after you've inspected it. Put the HTTP method as the first argument of Request() instead (uppercased). So session.post(...) becomes Request('POST', ...). All other arguments, except for verify are moved to the Request() call:

url = "https://10.1.1.254:4081/admin/api/jsonrpc/"
session = requests.Session()

data = {
    "jsonrpc": "2.0", "id":1, "method": "Session.login",
    "params": {"userName": "test", "password":"test123"}
}

request = requests.Request('POST', url, json=data)
prepped = session.prepare_request(request)

print("Sending request:")
print(format_prepped_request(prepped, 'utf8'))
print()
r = session.send(prepped, verify=False)

data = {"jsonrpc": "2.0", "id":3, "method":"Session.logout"}
request = requests.Request('POST', url, json=data)
prepped = session.prepare_request(request)

print("Sending request:")
print(format_prepped_request(prepped, 'utf8'))
print()
r = session.send(prepped, verify=False)

For your sample requests, this outputs:

Sending request:
POST /admin/api/jsonrpc/ HTTP/1.1
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 109
Content-Type: application/json

{"jsonrpc": "2.0", "id": 1, "method": "Session.login", "params": {"userName": "test", "password": "test123"}}

Sending request:
POST /admin/api/jsonrpc/ HTTP/1.1
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 55
Content-Type: application/json

{"jsonrpc": "2.0", "id": 3, "method": "Session.logout"}

Inspecting the request after sending

The response object references the prepared request object used to make the request, so you can print this after sending the request:

# ...
r = session.post(url, json=data, verify=False)
prepped = r.request
print("Request that was sent:")
print(format_prepped_request(prepped, 'utf8'))
print()

This isn't quite the same of course, the request has now already been sent. On the other hand, you don't need to use prepared requests directly, session.post() went through the steps to produce that object itself.

Do take into account that if your original request led to a redirect, that the final request may differ. Look at the response.history attribute to access any preceding responses (each with their own request attached).

Logging

Using the logging module, as well as rigging up the http.client.HTTPConnection class to log its debug output to the same framework, you can get debug-level information on requests too. I'm using my answer to Log all requests from the python-requests module here:

import logging

# function definition from https://stackoverflow.com/a/16337639/100297
# configures http.client to log rather than print
httpclient_logging_patch()
logging.basicConfig(level=logging.DEBUG)

at which point you'll see requests appear in logging output:

DEBUG:http.client:send: b'POST /admin/api/jsonrpc/ HTTP/1.1\r\nHost: 10.1.1.254:4081\r\nUser-Agent: python-requests/2.22.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 109\r\nContent-Type: application/json\r\n\r\n'
DEBUG:http.client:send: b'{"jsonrpc": "2.0", "id": 1, "method": "Session.login", "params": {"userName": "test", "password": "test123"}}'
DEBUG:http.client:reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:http.client:header: Date: Tue, 04 Feb 2020 13:41:52 GMT
DEBUG:http.client:header: Content-Type: application/json
DEBUG:http.client:header: Content-Length: 574
DEBUG:http.client:header: Connection: keep-alive
DEBUG:http.client:header: Server: gunicorn/19.9.0
DEBUG:http.client:header: Access-Control-Allow-Origin: *
DEBUG:http.client:header: Access-Control-Allow-Credentials: true
DEBUG:urllib3.connectionpool:https://10.1.1.254:4081 "POST /admin/api/jsonrpc/ HTTP/1.1" 200 574
DEBUG:http.client:send: b'POST /admin/api/jsonrpc/ HTTP/1.1\r\nHost: 10.1.1.254:4081\r\nUser-Agent: python-requests/2.22.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 55\r\nContent-Type: application/json\r\n\r\n'
DEBUG:http.client:send: b'{"jsonrpc": "2.0", "id": 3, "method": "Session.logout"}'
DEBUG:http.client:reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:http.client:header: Date: Tue, 04 Feb 2020 13:43:56 GMT
DEBUG:http.client:header: Content-Type: application/json
DEBUG:http.client:header: Content-Length: 574
DEBUG:http.client:header: Connection: keep-alive
DEBUG:http.client:header: Server: gunicorn/19.9.0
DEBUG:http.client:header: Access-Control-Allow-Origin: *
DEBUG:http.client:header: Access-Control-Allow-Credentials: true
DEBUG:urllib3.connectionpool:https://10.1.1.254:4081 "POST /admin/api/jsonrpc/ HTTP/1.1" 200 574

This isn't as flexible as using prepared requests but would work with existing code-bases. It also outputs data for the response, but you can do the same with the response object that requests gives you and print that out like we did with prepared requests above.

Answer 2

What you want is to see how the original method requests.post() handles your input data and transforms it to the packets sent through the HTTP request.

If you look at the post() source code you can see that two objects are defined, Request() and then a PreparedRequest() (see here). These objects will provide you what you're looking for, and you can access them using method prepare().

See this SO answer that gives you the necessary code:

>>> import requests
>>> req = requests.Request('POST','http://stackoverflow.com',headers={'X-Custom':'Test'},data='a=1&b=2')
>>> prepared = req.prepare()
>>> print(prepared.body, prepared.headers)
a=1&b=2 {'X-Custom': 'Test', 'Content-Length': '7'}

Answer 3

You can't do it with the high-level helper functions. Instead you have to create a request.PreparedRequest() object. Usually you'd instantiate a request.Request object (which is a bit higher level), prepare() that to get a PreparedRequest, then you can session.send(req).

Before sending, you can dump the various data from the PrepareRequest, I don't think there's a good builtin way to do that but for the most part the HTTP adapter will just write the request's method, url, headers and body to the connection it opened. So printing those before sending should be a good enough approximation, possibly with some formatting.

An alternative is to just configure httplib/urllib3/requests in debug (don't remember what it's called exactly) so that it automatically prints the request/response.

Answer 4

For those who want to view the RAW request before sending:

If you just want a quick and dirty (emphasis on the dirty) way of seeing and manipulating the exact bytes being sent to and from the server, you can override the following methods:

from http.client import HTTPConnection, HTTPResponse, _MAXLINE

# Spy on request
oldsend = HTTPConnection.send
def newsend(self, data):
    print(data)
    oldsend(self, data)

HTTPConnection.send = newsend

# Spy on response
def newinit(self, sock, *args, **kwargs):
    # Increase timeout value if we are reading the socket too soon
    sock.settimeout(1.0)
    output = b""
    while True:
        try:
            output += sock.recv(4096)
        except:
            break        
    print(output)

HTTPResponse.__init__ = newinit

# Doing the request
import requests

requests.get("https://httpbin.org/get")

This generates the following output (tested on Python 3.10):

>>> requests.get("https://httpbin.org/get")
b'GET /get HTTP/1.1\r\nHost: httpbin.org\r\nUser-Agent: python-requests/2.28.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
b'HTTP/1.1 200 OK\r\nDate: Sun, 21 Aug 2022 17:34:56 GMT\r\nContent-Type: application/json\r\nContent-Length: 306\r\nConnection: keep-alive\r\nServer: gunicorn/19.9.0\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Credentials: true\r\n\r\n{\n  "args": {}, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate", \n    "Host": "httpbin.org", \n    "User-Agent": "python-requests/2.28.1", \n    "X-Amzn-Trace-Id": "Root=1-63026cc0-36c4e3020ecbfde82b5587e9"\n  }, \n  "origin": "84.105.13.27", \n  "url": "https://httpbin.org/get"\n}\n'
Traceback (most recent call last):
  ...
AttributeError: 'HTTPResponse' object has no attribute 'fp'

This override only depends on the python standard library and will work with any library that uses http.client or urllib3 under the hood.

Answer 5

It depends on what you mean by "see a request". Given that the return value of session.post is a requests.Response, check

print(dir(requests.Response))

['__attrs__', '__bool__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'apparent_encoding', 'close', 'content', 'is_permanent_redirect', 'is_redirect', 'iter_content', 'iter_lines', 'json', 'links', 'next', 'ok', 'raise_for_status', 'text']

Perhaps what you are looking for is content, json, or text. I am not certain. So far I couldn't test it because I get

TimeoutError: [WinError 10060] ...

in your line

r = session.post(url, data_json, verify=False)

(I am not sure if it is because of my poor connection here).