How to POST a local file using urllib2 in Python?

Question

I am a complete Python noob and am trying to run cURL equivalents using urllib2. What I want is a Python script that, when run, will do the exact same thing as the following cURL command in Terminal:

curl -k -F docfile=@myLocalFile.csv http://myWebsite.com/extension/extension/extension

I found the following template on a tutorial page:

import urllib
import urllib2


url = "https://uploadWebsiteHere.com"
data = "{From: 'sender@email.com', To: 'recipient@email.com', Subject:   'Postmark test', HtmlBody: 'Hello dear Postmark user.'}"
headers = { "Accept" : "application/json",
        "Conthent-Type": "application/json",
        "X-Postmark-Server-Token": "abcdef-1234-46cc-b2ab-38e3a208ab2b"}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()

but I am completely lost on the 'data' and 'headers' vars. The urllib2 documentation (https://docs.python.org/2/library/urllib2.html) defines the 'data' input as "a string specifying additional data to send to the server" and the 'headers' input as "a dictionary". I am totally out of my depth in trying to follow this documentation and do not see why a dictionary is necessary when I could accomplish this same task in terminal by only specifying the file and URL. Thoughts, please?

You should consider using python-requests instead: http://docs.python-requests.org/en/latest/. As it says at the top of its homepage "Python’s standard urllib2 module provides most of the HTTP capabilities you need, but the API is thoroughly broken. It was built for a different time — and a different web. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks". — dano, May 15 '14 at 20:44
I looked into that but couldn't even download "pip" to download requests. When I ran the "python get-pip.py" command in Terminal, I got an error saying that the file doesn't even exist (even though I downloaded it). — IntegrateAllDay, May 15 '14 at 20:50
Without having `pip` installed, you will not go very far in Python. Resolve this very basic prerequisity first. — Jan Vlcinsky, May 15 '14 at 20:56
On the assumption that you're on Mac OS, [this](http://stackoverflow.com/questions/17271319/installing-pip-on-mac-os-x) might help ... @JanVlcinsky is right; you need pip. — Zero Piraeus, May 15 '14 at 20:57
Thanks, @ZeroPiraeus! But now when I run "pip install requests" in Terminal, I get the error "Storing debug log for failure in /Users/ME/Library/Logs/pip.log" — IntegrateAllDay, May 15 '14 at 21:01
... well, once you've read that log, you'll either know how to resolve the problem or have the basis for a new SO question :-) ... (possibly you need `sudo pip install requests`, but I'm not a Mac person). — Zero Piraeus, May 15 '14 at 21:04
@ZeroPiraeus Never mind, I fixed it (used "$ sudo easy_install requests"). — IntegrateAllDay, May 15 '14 at 21:10
Most HTTP headers are not required. You could probably get away with using an empty dictionary. — b4hand, May 15 '14 at 22:28
related: [How to send a “multipart/form-data” with requests in python?](http://stackoverflow.com/q/12385179/4279) — jfs, May 16 '14 at 23:37

score 1 · Accepted Answer · answered May 15 '14 at 22:43

The data you are posting doesn't appear to be valid JSON. Assuming the server is expecting valid JSON, you should change that.

Your curl invocation does not pass any optional headers, so you shouldn't need to provide much in the request. If you want to verify the exact headers you could add -vi to the curl invocation and directly match them in the Python code. Alternatively, this works for me:

import urllib2

url = "http://localhost:8888/"
data = '{"From": "sender@email.com", "To": "recipient@email.com", "Subject": "Postmark test", "HtmlBody": "Hello dear Postmark user."}'
headers = {
    "Content-Type": "application/json"
}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()

It probably is in your best interest to switch over to using requests, but for something this simple the standard library urllib2 can be made to work.

this code doesn't emulate the `curl` command from the question — jfs, May 16 '14 at 23:38

score 1 · Answer 2 · answered May 16 '14 at 23:36

What I want is a Python script that, when run, will do the exact same thing as the following cURL command in Terminal:

$ curl -k -F docfile=@myLocalFile.csv https://myWebsite.com/extension...

curl -F sends the file using multipart/form-data content type. You could reproduce it easily using requests library:

import requests # $ pip install requests

with open('myLocalFile.csv','rb') as input_file:
    r = requests.post('https://myWebsite.com/extension/...', 
                      files={'docfile': input_file}, verify=False)

verify=False is to emulate curl -k.

How to POST a local file using urllib2 in Python?

2 Answers2