Python: Upload a file in chunk to a server including additional fields

Question

I excuse myself right here... I am a HTTP noob... but I need to create a python script that takes a file and uploads it to a server in chunks including additional data.

EDIT: It's only one file that I want to upload... but in chunks

So I started researching and came across pycurl which seems to be pretty complicated. So I went on to requests which looks really nice and intuitive.

And basically I got it working in parts... but... not all of it combined :)

I've seen that I can use generators to provide chunks of my data. That's awesome! But I also need to send (sorry... my vocabulary when it comes to that kind of stuff is very limited) multipart boundaries??? Other fields that contain JSON information...

So my requests should look something like that:

POST / HTTP/1.1
Host: some server
Content-Type: multipart/form-data;

Connection: Keep Alive
Transfer-Encoding: chunked

--boundary
Content-Disposition: form-data; name="Name of my field"
Content-Type: application/JSON; charset=utf-8
Content-Transfer-Encoding: 8bit

{"Some" : "content", "in" : "here"}

--boundary
Content-Disposition: form-data; name="My File"
Content-Type: audio/mpeg
Content-Transfer-Encoding: binary

... chunk 1 ....

--boundary
Content-Disposition: form-data; name="My File"
Content-Type: audio/mpeg
Content-Transfer-Encoding: binary

... chunk 2 ....

and so on...

I figured that I can create the chunked upload using a generator. And I also figured that I can create the non-file-boundaries by using the file= option.

But the problem is that I can't use both :( And also when using my genrator I can't define f.e. the Content-Type of my chunks, nor a name...

Again... sorry for my bad vocabulary :)

Any help is very much appreciated

Answer 1

import requests
import json

#from here http://stackoverflow.com/a/23816211/642096
def pretty_print_POST(req):
    """
    At this point it is completely built and ready
    to be fired; it is "prepared".

    However pay attention at the formatting used in 
    this function because it is programmed to be pretty 
    printed and may differ from the actual request.
    """
    print('{}\n{}\n{}\n\n{}'.format(
        '-----------START-----------',
        req.method + ' ' + req.url,
        '\n'.join('{}: {}'.format(k, v) for k, v in req.headers.items()),
        req.body,
    ))

url = 'http://httpbin.org/post'
data = {'input_name': json.dumps({
    'json': 'here'
})}
files = {
    'file1': ('doc.txt', open('/tmp/doc.txt', 'rb'), 'text/plain'),
    'file2': ('doc2.html', open('/tmp/doc2.html', 'rb'), 'text/html'),    
}
r = requests.post(url, data=data, files=files)
pretty_print_POST(r.request)
print r.text