Get raw request body using perl CGI.pm

Question

I am building a web server using Apache and Perl CGI which processes the POST requests sent to it. The processing part requires me to get the completely unprocessed data from the request and verify its signature.

The client sends two different kinds of POST requests: one with the content-type set as application/json, and the second one with content type as application/x-www-form-urlencoded.

I was able to fetch the application/json data using cgi->param('POSTDATA'). But if I do the same for application/x-www-form-urlencoded data, i.e. cgi->param('payload'), I get the data but it's already decoded. I want the data in its original URL-encoded format. i.e I want the unprocessed data as it is sent out by the client.

I am doing this for verifying requests sent out by Slack.

CGi.pm doesn't save the raw form of what it received when it's form data. — ikegami, Jul 30 '18 at 19:02
@ikegami Ouch, then I am stuck. What if I can get what encoding it is using, and then encode it? If yes, how do I get what type of encoding client is using? — Siddharth Pathak, Jul 30 '18 at 19:22
Can't, because `abc=def` and `%61bc=def` are two equivalent ways of encoding `abc=def`. You could fool CGI.pm into thinking it's not form data, or you could avoid using CGI.pm and handle the inputs yourself. — ikegami, Jul 30 '18 at 19:29
@Borodin - That is not very helpful, as there is a lot of useful functionality. — Ωmega, Oct 07 '19 at 12:53

Ωmega · Answer 1 · 2019-10-08T14:43:05.623

To handle all cases, including those when Content-Type is multipart/form-data, read (and put back) the raw data, before CGI does.

use strict;
use warnings;

use IO::Handle;
use IO::Scalar;

STDIN->blocking(1); # ensure to read everything
my $cgi_raw = '';

{ 
  local $/; 
  $cgi_raw = <STDIN>;
  my $s;
  tie  *STDIN, 'IO::Scalar', \$s;
  print STDIN $cgi_raw;
  tied(*STDIN)->setpos(0);
}

use CGI qw /:standard/;
...

score -2 · Answer 2 · answered Aug 01 '18 at 18:44

Though I'm not sure which Perl Module can handle it all for you, but here is a basic rundown.

Your HTML form should submit to a .cgi file (or any other handler which is properly defined).

The raw request is something similar to this:

POST HTTP/1.1
UserAgent: Mozilla/5.0
Content-Length: 69
Host: 127.0.0.1
(More headers depending on situation and then a single blank line)

(Message Body Containing data)
username=John&password=123J (example)

https://en.wikipedia.org/wiki/List_of_HTTP_header_fields)

What will happen is that, this data is available via the CGI (not CGI perl module aka CGI.pm) using the environment variables and stdin (header feilds are passed using EV and message body using stdin).

In Perl, I think you need this to read those EVs: http://perldoc.perl.org/Env.html

And this to read stdin: https://perlmaven.com/read-from-stdin

From there on, you can process as needed.

BE CAREFULL, when reading any of these. You can be sent malformed information like 100GB valid data in one of the HTTP headers, or in message body, which can break havoc on you or dangerous system calls, etc. Sterilizing is necessary, before passing the data to other places.

Get raw request body using perl CGI.pm

2 Answers2