How can I get the raw HTTP message body using the request library in Node.js?

Question

The npm-request library allows me to construct HTTP requests using a nice JSON-style syntax, like this.

request.post(
    {
        url: 'https://my.own.service/api/route',
        formData: {
            firstName: 'John',
            lastName: 'Smith'
        }
    },
    (err, response, body) => {
        console.log(body)
    }
);

But for troubleshooting, I really need to see the HTTP message body of the request as it would appear on the wire. Ideally I'm looking for a raw bytes representation with a Node.js Buffer object. It seems easy to get this for the response, but not the request. I'm particularly interested in multipart/form-data.

I've looked through the documentation and GitHub issues and can't figure it out.

Use a network sniffer and see what's actually being sent over the socket, independent of your node.js code. — jfriend00, Aug 02 '19 at 23:51
I don't particularly want to set up a network sniffer and deal with the hassle of intercepting TLS connections. I want to do this in code. — Nic, Aug 03 '19 at 00:05
If it's your own target server, you can temporarily allow http just for debugging or just from your endpoint or just in your local test network to avoid TLS for the sniffer. Sometimes you need to create conditions favorable for debugging. — jfriend00, Aug 03 '19 at 00:19
Well `http://my.own.service` in your question made it look like it was. Anyway, you have plenty of other options. — jfriend00, Aug 03 '19 at 00:21
I don't know if this is an acceptable answer so I leave it as a comment for now but you can modify the file `node_modules/request.js` and add `console.log(arguments)` after this line: https://github.com/request/request/blob/master/request.js#L1493 — slebetman, Aug 03 '19 at 02:48

score 5 · Answer 1 · answered May 26 '20 at 13:18

5

Simplest way to do this is to start a netcat server on any port:

$ nc -l -p 8080

and change the URL to localhost in your code:

https://localhost:8080/v1beta1/text:synthesize?key=API_KEY

Now, any requests made will print the entire, raw HTTP message sent to the localhost server.

Obviously, you won't be able to see the response, but the entire raw request data will be available for you to inspect in the terminal you have netcat running

answered May 26 '20 at 13:18

Swayam

580
6
11

1

In macOS, you should use `nc -l 8080` and worked great! – fagiani Dec 27 '20 at 14:24

score 4 · Accepted Answer · answered Aug 03 '19 at 00:26

4

I figured out how to dump the HTTP message body with Request. In both cases, I'm just copying the same approach that request uses internally.

Multipart Form Uploads

req._form.pipe(process.stdout);

URL-encoded Forms

console.log(req.body);

answered Aug 03 '19 at 00:26

Nic

4,319
5
29
36

score 0 · Answer 3 · answered Aug 03 '19 at 00:04

0

You could try @jfriend00 suggestion an use a network sniffer like wireshark but as you're fetching an https URL this might not be the easiest route as it requires some setup to intercept TLS connections.

So maybe it would be enough turning on debug mode for the request module itself, you can do that by simply setting require('request').debug = true. As a third option you could go with the dedicated debug module for request here which allows you to view request and response headers and bodies.

answered Aug 03 '19 at 00:04

exside

3,736
1
12
19

The debug mode for `request` still doesn't print the HTTP message body, it just echoes the JSON object back to me in the same format I provided. – Nic Aug 03 '19 at 00:08
The `request-debug` module in its default configuration only shows the headers, not the message body. – Nic Aug 03 '19 at 00:11

jfriend00 · Answer 4 · 2019-08-03T00:17:46.387

I can think of a number of ways to see the bytes of the request:

Turn on debugging in the request module. There are multiple ways to do that documented here including setting NODE_DEBUG=request or require('request').debug = true or using the request-debug module.
Use a network sniffer to see what's actually being sent over the socket, independent of your node.js code.
Create your own dummy http server that does nothing but log the exact incoming request and send your same request to that dummy server so it can log it for you.
Create or use a proxy (like nginx) that can dump the exact incoming request before forwarding it on to its final destination and send the request to the proxy.
Step through the sending of the request in the debugger to see exactly what it is writing to the socket (this may be time consuming, particularly with async callbacks, but will eventually work).

#4) You can use an existing proxy. It seems reasonable to insert nginx in front of the server to dump info like this: https://stackoverflow.com/questions/4939382/logging-post-data-from-request-body — Wyck, Aug 03 '19 at 00:16

score 0 · Answer 5 · answered Aug 27 '22 at 10:40

0

you could use a nodejs server capable of logging the raw request/response string , then direct your request to that server

i gave an example using both http and https server - no dependencies

nodejs getting raw https request and response

answered Aug 27 '22 at 10:40

matt richards

142
7

How can I get the raw HTTP message body using the request library in Node.js?

5 Answers5

Multipart Form Uploads

URL-encoded Forms

Linked