How do I get POST "data" variable in python3.x without using CGI?

Question

When I attempt to call cgi.FieldStorage() in a python3.x cgi script, I get the following error:

[Traceback: error in module x on line y]:
    cgi.FieldStorage()
File "/usr/lib64/python3.3/cgi.py", line 553, in __init__
    self.read_single()
File "/usr/lib64/python3.3/cgi.py", line 709, in read_single
    self.read_binary()
File "/usr/lib64/python3.3/cgi.py", line 731, in read_binary
    self.file.write(data)
TypeError: must be str, not bytes

How do I get my POST data variable from an ajax call?

Example ajax call:

function (param) {
    $.ajax({
       type: "POST",
       url: "/cgi-bin/mycgi.py/TestMethod",
       data: JSON.stringify({"foo": "bar"}),
       contentType: "application/json; charset=utf-8",
       dataType: "json",
       success: function (result) {
           alert("Success " + result);
       },
       error: function () {
           alert("Failed");
       }
    });
}

I'm not familiar with jQuery, so I'll post this as a comment. Try `contentType: "x-www-form-urlencoded"`. — Yosh, Dec 02 '14 at 04:40

NuclearPeon · Accepted Answer · 2019-02-12T19:00:00.207

According to http://lucumr.pocoo.org/2013/7/2/the-updated-guide-to-unicode/:

"There are also some special cases in the stdlib where strings are 
very confusing. The cgi.FieldStorage module which WSGI applications are 
sometimes still using for form data parsing is now treating QUERY_STRING 
as surrogate escaping, but instead of using utf-8 as charset for the URLs 
(as browsers) it treats it as the encoding returned by 
locale.getpreferredencoding(). I have no idea why it would do that, but 
it's incorrect. As workaround I recommend not using cgi.FieldStorage for 
query string parsing."

The solution to this problem is to use sys.stdin.read to read in POST data parameters. However please note that your cgi application can hang if it expects to read in something and nothing is sent. This is solved by reading in the number of bytes that is found in the HTTP Header:

#!/usr/bin/env python3
import os, sys, json
data = sys.stdin.read(int(os.environ.get('HTTP_CONTENT_LENGTH', 0)))
# To get data in a native python dictionary, use json.loads
if data:
    print(list(json.loads(data).keys())) # Prints out keys of json

# (You need to wrap the .keys() in list() because it would otherwise return 
#  "dict_keys([a, b, c])" instead of [a, b, c])

You can read more about the internals of CGI here: http://oreilly.com/openbook/cgi/ch04_02.html

In my case I needed `CONTENT_LENGTH` instead of `HTTP_CONTENT_LENGTH` in case that trips someone up. Python 3.6. — Mike Howsden, Mar 16 '21 at 20:42

score 1 · Answer 2 · answered Jul 18 '15 at 13:45

The accepted answer didn't work for me and returned no data (using Windows Server 2012 & Python 3.4).

Of course I appreciate it may have worked for others, but wanted to post this in case it helps anyone who finds the same situation I have experienced.

After trawling many similar questions & related blog posts, and testing different approaches myself, the winning combination for me was:

totalBytes=int(os.environ.get('HTTP_CONTENT_LENGTH'))
reqbin=io.open(sys.stdin.fileno(),"rb").read(totalBytes)

Those 2 lines were all that I needed to receive raw binary data (images, audio files etc) to then dump out to a file.

If you want to turn the received data into a string you can then use:

reqstr=reqbin.decode("utf-8")

Finally, to meet the question's requirements, you can parse that as JSON using:

thejson=json.loads(reqstr)

I really hope this helps others who were not able to find another way!

score 0 · Answer 3 · answered Feb 15 '19 at 01:30

0

in my case, I tcpdump -x -Xed the data being sent from JavaScript and noticed a Content-type header of text/plain was being sent. so I set it to application/x-www-form-urlencoded as recommended here and that fixed the problem.

answered Feb 15 '19 at 01:30

jcomeau_ictx

37,688
6
92
107

How do I get POST "data" variable in python3.x without using CGI?

3 Answers3