How to replay HTTP requests in Python

Question

I would like to replay an arbitrary raw HTTP request in python. As an example, lets use an arbitrary GET request from my chromium browser to Google:

GET / HTTP/1.1
Host: google.de
Cookie: CONSENT=PENDING+071; AEC=ARSKqsKvfEvPS9Vp1bDM6YMPdHpVOoH357W-7q3cqtdDwYeHf3MqPEO1xA; SOCS=CAISHAgBEhJnd3NfMjAyMzAyMjMtMF9SQzEaAmRlIAEaBgiAn-WfBg; NID=511=N6YvXcWd_hnVVnV8w6JK4jscqE2pEt8MuTrw3yZJp-84ZxV6RJLee_yj2DEo2UJuOse0sqLjdnAD7qgPw9al7aEJqsQOCAQPIs21rLy5HQ5IAoObj7icI7ayKJttejI9Va2jDFkk0ZLvUC7P_VPJuxRJyhvLspqU1YVUcYCThrYizbo; 1P_JAR=2023-2-25-20
Sec-Ch-Ua: "Not A(Brand";v="24", "Chromium";v="110"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Linux"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.5481.78 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
X-Client-Data: CO3/ygE=
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: close

Since I want my request to look exactly like the above, an obvious way would be to use the socket library:

def send(host, port, message):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect((host, port))
        s.sendall(message)
        print("%s" % message.decode('us-ascii'))
        data = s.recv(1024)

    print("%s" % data)

However, that way, I obviously can't use TLS or proxies (without writing my own libraries).

So, I thought the request library could be worth a try since it already supports TLS and proxies out of the box. But I can't figure out how to load my HTTP request that I want to replay. I suppose the library was not designed for that.

I could try something like this (from the docs):

from requests import Request, Session

s = Session()

req = Request('POST', url, data=data, headers=headers)
prepped = req.prepare()

# do something with prepped.body
prepped.body = 'No, I want exactly this as the body.'

# do something with prepped.headers
del prepped.headers['Content-Type']

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)

but I still would have to first parse the headers of my request manually. If there a library that takes care of that?

Thanks for your input!

You can parse headers in this format using the `email` module from the standard library, see here: https://stackoverflow.com/questions/4685217/parse-raw-http-headers — C4ffeine Add1ct, Mar 01 '23 at 18:56
Thanks for the pointer! The `email` library works fine to extract the headers. But I'll still go with the `aioreq` solution, because like that I can simply send raw HTTP requests over TLS without fiddling them into the `request` library. — CryptoPerson, Mar 03 '23 at 15:31

score 0 · Accepted Answer · answered Mar 02 '23 at 09:10

TLS can be used with sockets; you just need to wrap it in ssl library sockets.

However, there are many libraries that can do it, so you don't have to use TLS yourself.

You can use the aioreq library, which is a simple asynchronous HTTP client that can be used to send raw HTTP request messages.

Aioreq does not support proxies for raw requests, but it does allow you to do many interesting things and see how the HTTP protocol works.

This is your request (without the Cookie header, it was quite large:D).

import asyncio
import aioreq
import socket

async def main():
    hostname = "google.de"
    transport = aioreq.Transport()
    await transport.make_connection(
        ip=socket.gethostbyname(hostname),
        port=443,
        ssl=True,  # True if you want to use SSL/TLS
        server_hostname=hostname
    )

    raw_request_bytes = ("GET / HTTP/1.1\r\n"
"Host: google.de\r\n"
'Sec-Ch-Ua: "Not A(Brand";v="24", "Chromium";v="110"\r\n'
"Sec-Ch-Ua-Mobile: ?0\r\n"
'Sec-Ch-Ua-Platform: "Linux"\r\n'
"Upgrade-Insecure-Requests: 1\r\n"
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                         "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.5481.78 Safari/537.36\r\n"
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,'
                         'image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\r\n'
"X-Client-Data: CO3/ygE=\r\n"
"Sec-Fetch-Site: none\r\n"
"Sec-Fetch-Mode: navigate\r\n"
"Sec-Fetch-User: ?1\r\n"
"Sec-Fetch-Dest: document\r\n"
"Accept-Encoding: gzip, deflate\r\n"
"Accept-Language: en-US,en;q=0.9\r\n"
"Connection: close\r\n\r\n"
).encode("ascii")

    status_line, headers_line, content = await transport.send_http_request(
        raw_request_bytes
    )
    resp = aioreq.parsers.ResponseParser.parse(status_line, headers_line, content)
    print(resp)
asyncio.run(main())

As a result, we have the variable resp, which is our HTTP Response with all of its required attributes, such as (.headers, .content, .status, ...)

Thanks! Works perfectly! (Except for HTTP proxies of course, but I can do without proxy support for now). — CryptoPerson, Mar 03 '23 at 15:27

How to replay HTTP requests in Python

1 Answers1