How to get file from POST data in Bash CGI script?

Question

I am trying to post a file using cURL and receive it on the other side via a CGI Bash script and store it with the same name. After upload is completed, diff between the original file and reconstructed one should return zero.

The way cURL sends data:

curl --request POST --data-binary "@dummy.dat" 127.0.0.1/cgi-bin/upload-rpm

Receiver script:

#!/bin/bash

echo "Content-type: text/html"
echo ""

echo '<html>'
echo '<head>'
echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">'
echo '<title>Foo</title>'
echo '</head>'
echo '<body>'

echo "<p>Start</p>"

if [ "$REQUEST_METHOD" = "POST" ]; then
    echo "<p>Post Method</p>"
    if [ "$CONTENT_LENGTH" -gt 0 ]; then
        echo "<p>Size=$CONTENT_LENGTH</p>"
        while read -n 1 byte -t 3; do echo -n -e "$byte" >> ./foo.dat ; done
    fi
fi
echo '</body>'
echo '</html>'

exit 0

But it's not working. File is not created on the server side. And how can I get the file name?

You could get the file name by specifying a custom header on the sending side and then including it in the request. i.e. `curl -H "x-filename: foo.dat"`. Regarding the debugging of the write script, perhaps add `-x` to the shebang and see if you can catch where the script is exiting. One more thing - reading 1 byte at a time via `read` would be very slow. What is the cgi program that is setting the variables that your receive script is using to process the data ? — hmedia1, Sep 09 '18 at 10:06
What is your server? (apache, nginx, netcat, other)?? The file is recieved by your server first, then passed to CGI script as *internal* data, in a way depending on server and version of server. — F. Hauri - Give Up GitHub, Sep 17 '18 at 10:51
Care! Using [tag:bash] for *CGI scripting* seem very unsecure!!! — F. Hauri - Give Up GitHub, Sep 17 '18 at 10:55
At line 2 of your script, add `echo "$0 -- $@" >/tmp/trace-cgi-$$.txt ; set >>/tmp/trace-cgi-$$.txt` this will help you to see how environment variables are populated by apache, befor running CGI. — F. Hauri - Give Up GitHub, Sep 18 '18 at 07:38

hanshenrik · Accepted Answer · 2022-06-24T09:18:09.867

as long as you're always uploading exactly 1 file using the multipart/form-data format and the client always put the Content-Type as the last header of the file upload (which curl always seem to do), this seem to work:

#!/bin/bash

echo "Content-type: text/html"
echo ""

echo '<html>'
echo '<head>'
echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">'
echo '<title>Foo</title>'
echo '</head>'
echo '<body>'

echo "<p>Start</p>"

if [ "$REQUEST_METHOD" = "POST" ]; then
    echo "<p>Post Method</p>"
    if [ "$CONTENT_LENGTH" -gt 0 ]; then
    in_raw="$(cat)"
    boundary=$(echo -n "$in_raw" | head -1 | tr -d '\r\n');
    filename=$(echo -n "$in_raw" | grep --text --max-count=1 -oP "(?<=filename=\")[^\"]*");
    file_content=$(echo -n "$in_raw" | sed '1,/Content-Type:/d' | tail -c +3 | head --lines=-1 | head --bytes=-4  );
    echo "boundary: $boundary"
    echo "filename: $filename"
    #echo "file_content: $file_content"
    fi
fi
echo '</body>'
echo '</html>'

exit 0

example upload invocation (which I used to debug the above):

curl http://localhost:81/cgi-bin/wtf.sh -F "MyFile=@myfile.txt"

with all that said, THIS IS INSANITY! bash is absolutely not suitable for handling binary data, such as http file uploads, use something else. (PHP? Python? Perl?)

It's very educational regardless of the insanity. – Sridhar Sarnobat Jun 16 '21 at 06:47 — Sridhar Sarnobat, Jun 16 '21 at 06:47

score 11 · Answer 2 · answered Sep 17 '18 at 07:53

11

The ways curl sends data

With fields:

curl --data "param1=value1&param2=value2" https://example.com/resource.cgi

With fields specified individually:

curl --data "param1=value1" --data "param2=value2" https://example.com/resource.cgi

Multipart:

curl --form "fileupload=@my-file.txt" https://example.com/resource.cgi

Multipart with fields and a filename:

curl --form "fileupload=@my-file.txt;filename=desired-filename.txt" --form param1=value1 --form param2=value2 https://example.com/resource.cgi

For a RESTful HTTP POST containing XML:

curl -X POST -d @filename.txt http://example.com/path/to/resource --header "Content-Type:text/xml"

or for JSON, use this:

curl -X POST -d @filename.txt http://example.com/path/to/resource --header "Content-Type:application/json"

This will read the contents of the file named filename.txt and send it as the post request.

For further information see

Reading HTTP POST data using BASH

How to get the content length:

if [ "$REQUEST_METHOD" = "POST" ]; then
    if [ "$CONTENT_LENGTH" -gt 0 ]; then
        read -n $CONTENT_LENGTH POST_DATA <&0
        echo "$CONTENT_LENGTH"
    fi
fi

To save a binary or text file:

boundary=$(export | \
    sed '/CONTENT_TYPE/!d;s/^.*dary=//;s/.$//')
filename=$(echo "$QUERY_STRING" | \
    sed -n '2!d;s/\(.*filename=\"\)\(.*\)\".*$/\2/;p')
file=$(echo "$QUERY_STRING" | \
    sed -n "1,/$boundary/p" | sed '1,4d;$d')

The uploaded file is now contained in the $file variable, and file name in the $filename variable.

answered Sep 17 '18 at 07:53

Bharata

13,509
6
36
50

I don't see where do you read actual POST data? both `$filename` and `$file` variables are empty at the end – sorush-r Sep 18 '18 at 05:59
@sorush-r, how do you send it? – Bharata Sep 18 '18 at 10:19
`curl --form "fileupload=@deploy.zip;filename=deploy.zip" --form branch=dev 10.0.0.167/cgi-bin/upload-rpm.bash` – sorush-r Sep 18 '18 at 12:04
@sorush-r, you have to write without `branch=dev`. Try to open your web site via `https://10.0.0.167/cgi-bin/upload-rpm.bash` (**or may be it is http** only and not https). **At first you have to be sure that your website is accessible via this address**. And then write it without `branch=dev`and without **LAST** `--form` (it is only to add if you send form field values) like follows `curl --form "fileupload=@deploy.zip;filename=deploy.zip" https://10.0.0.167/cgi-bin/upload-rpm.bash` OR `curl --form "fileupload=@deploy.zip;filename=deploy.zip" https://example.com/cgi-bin/upload-rpm.bash` – Bharata Sep 18 '18 at 14:33
@sorush-r, and do not forget about `http` only `https` difference. – Bharata Sep 18 '18 at 14:33
Sorry I have no access to my machine at work till Saturday morning. I will test and return back – sorush-r Sep 20 '18 at 19:41
No it does not. Binary data gets truncated. @hanshenrik s answer works – sorush-r Sep 23 '18 at 09:51

How to get file from POST data in Bash CGI script?

2 Answers2

The ways curl sends data

Reading HTTP POST data using BASH

Linked