Process raw HTTP request to python requests

Question

I'd like to pass a raw HTTP request like:

GET /foo/bar HTTP/1.1
Host: example.org
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Accept: */*
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Referer: http://example.org/test
Cookie: foo=bar; lorem=ipsum;

And generate the python request such as:

import requests

burp0_url = "http://example.org:80/foo/bar"
burp0_cookies = {"foo": "bar", "lorem": "ipsum"}
burp0_headers = {"User-Agent": "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8", "Accept": "*/*", "Accept-Language": "fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3", "Accept-Encoding": "gzip,deflate", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Keep-Alive": "115", "Connection": "keep-alive", "Content-Type": "application/x-www-form-urlencoded", "X-Requested-With": "XMLHttpRequest", "Referer": "http://example.org/test"}
requests.get(burp0_url, headers=burp0_headers, cookies=burp0_cookies)

Is there a library for that?

Does this answer your question? [Creating a raw HTTP request with sockets](https://stackoverflow.com/questions/5755507/creating-a-raw-http-request-with-sockets) — SuperStormer, Apr 07 '20 at 23:00

score 2 · Accepted Answer · answered Apr 07 '20 at 22:58

I could not find an existing library that does this conversion, but there is a Python library to convert curl commands to python requests code. https://github.com/spulec/uncurl

e.g.

import uncurl
print(uncurl.parse('curl --header "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" --compressed --header "Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3" --header "Connection: keep-alive" --header "Content-Type: application/x-www-form-urlencoded" --cookie "foo=bar; lorem=ipsum;" --header "Keep-Alive: 115" --header "Referer: http://example.org/test" --user-agent "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8" --header "X-Requested-With: XMLHttpRequest" https://example.org/foo/bar '))

I haven't found a Python library to transform raw HTTP into such a curl command. However, this Perl program does it.

Like this:

$ cat basic
GET /index.html HTTP/2
Host: example.com
Authorization: Basic aGVsbG86eW91Zm9vbA==
Accept: */*

$ ./h2c < basic
curl --http2 --header User-Agent: --user "hello:youfool" https://example.com/index.html

You could either call it from your python script, use a Python-Perl bridge or try to port it.

Postman also allows you to convert raw HTTP requests directly to python requests code, using its Code snippet generator. Although, it seems this can only be done via the GUI. It's also not Open-Source, so you can't access the code that does this transformation.

score 1 · Answer 2 · answered Jun 18 '20 at 22:28

I needed something that can generate a request and couldn't find it so ended up writing it in gist:

class RequestParser(object):
    def __parse_request_line(self, request_line):
        request_parts = request_line.split(' ')
        self.method = request_parts[0]
        self.url = request_parts[1]
        self.protocol = request_parts[2] if len(request_parts) > 2 else DEFAULT_HTTP_VERSION

    def __init__(self, req_text):
        req_lines = req_text.split(CRLF)
        self.__parse_request_line(req_lines[0])
        ind = 1
        self.headers = dict()
        while ind < len(req_lines) and len(req_lines[ind]) > 0:
            colon_ind = req_lines[ind].find(':')
            header_key = req_lines[ind][:colon_ind]
            header_value = req_lines[ind][colon_ind + 1:]
            self.headers[header_key] = header_value
            ind += 1
        ind += 1
        self.data = req_lines[ind:] if ind < len(req_lines) else None
        self.body = CRLF.join(self.data)

    def __str__(self):
        headers = CRLF.join(f'{key}: {self.headers[key]}' for key in self.headers)
        return f'{self.method} {self.url} {self.protocol}{CRLF}' \
               f'{headers}{CRLF}{CRLF}{self.body}'

    def to_request(self):
        req = requests.Request(method=self.method,
                               url=self.url,
                               headers=self.headers,
                               data=self.data, )
        return req

Process raw HTTP request to python requests

2 Answers2