1

In Python's requests documentation, it suggests sending multipart form data like so:

Requests makes it simple to upload Multipart-encoded files:

>>> url = 'http://httpbin.org/post'
>>> files = {'file': open('report.xls', 'rb')}

>>> r = requests.post(url, files=files)

This question shows the same behavior. However, this question by the same author uses a with statement:

with open('data','rb') as payload:
    headers = {'content-type': 'application/x-www-form-urlencoded'}
    r = requests.post('https://IP_ADDRESS/rest/rest/2', auth=('userid', 'password'),
                      data=payload, verify=False, headers=headers)

My intuition is that I should always use a with statement when using files; do I need it in this case or not? To be extra clear, I am not looking for advice as to whether or not to use with. I am specifically referring to the interaction with the requests library, and if I can omit the with statement as the docs imply.

TylerH
  • 20,799
  • 66
  • 75
  • 101
Athena
  • 3,200
  • 3
  • 27
  • 35
  • I think using `with` is a way to automatically close the connection you opened in your `with`-statement. You can also do it without `with`, but then you need to close the connection explicitly, like `payload.close()`. I think. – KenHBS Aug 22 '18 at 18:34
  • @downvoter care to comment? – Athena Aug 22 '18 at 18:40
  • @Ares I'm not the downvoter but that person likely did so because this is an opinion-based question. – TylerH Aug 22 '18 at 19:55
  • 1
    @TylerH I should clarify; I'm asking about the interaction between requests and `with`, not looking for an opinion. – Athena Aug 22 '18 at 20:06

3 Answers3

3

tl;dr: Your intuition is right.

Without a with statement (or an explicit payload.close() call), you're not closing the file.

Of course it's always possible that some function you call, passing the file as an argument, might close that file. But in general, they won't; and, in particular, requests.post doesn't. (Since that's pretty rare and unusual behavior, it would probably be documented if a function did that. But if you need to be absolutely sure, you can always test it—call the function and then check f.closed—or read the source.)

So, what happens if you don't close it? It just sits there open until the garbage collector deletes the file object. When does that happen? Well, you have a global variable named files that holds a dict that holds the file object. So the file object is accessible all the way until you quit the program. It never becomes garbage, so it never gets deleted, so the file never gets closed, until you exit.

Since you aren't writing anything to the file, this doesn't result in anything horrible like unflushed data sitting around in a buffer when you expected it to be on disk.

But it does mean you're wasting resources. The OS kernel has to keep some kind of structure for every open file, and the the process needs some other thing to reference that kernel structure. If you open thousands of files without closing them, eventually, one of those tables fills up, and the next time you try to open a file, you get an error about having too many files open.

And the kinds of programs that do a lot of work with requests tend to be exactly the kinds of programs that need to work with thousands of files, so this can be a real problem.

So, do you need a with statement here? Maybe not. But you definitely want one.

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • 1
    The footnote is the key information for me; I know how `with` works, but it was unclear to me given the number of reputable examples that don't use it if `requests` does or does not close the resource for me. – Athena Aug 22 '18 at 18:41
  • 1
    @Ares OK, I edited the answer to make the footnote part of the main answer, and to expand on it. – abarnert Aug 22 '18 at 18:50
1

Yes. Using a with statement handles closing resources for you. You should use this to properly handle external resources.

J. Blackadar
  • 1,821
  • 1
  • 11
  • 18
1

The documentation is just trying to demonstrate how the files parameter works, without worrying too much about whether the file gets closed. A better example would be

with open('report.xls', 'rb') as f:
    r = requests.post(url, files={'file': f})
chepner
  • 497,756
  • 71
  • 530
  • 681