0

I am surprised to see very different behavior of tempfile.TemporaryFile() on Windows vs. Linux.

As is best pratice, I am using Python's tempfile module, as also suggested here.

Intent: I want to create a temporary file, and pass its name/path to some function (not the file object). The following code works fine on Windows (Win 10, Python 3.8.6), but fails in a docker container on Amazon ECS (python:3.8.6-slim-buster).

import tempfile
import boto3

with tempfile.TemporaryFile() as tmpfile:
    s3 = boto3.client('s3')
    s3.download_file("my_bucket", "my_object", tmpfile.name)

This code fails not for S3 access rights/roles (that's all ok), but indeed due to the file name being invalid! A valid file name is produced on Windows (see output below), but is clearly invalid in the docker container on ECS:

  • Windows: tmpfile.name: C:\Users\micha\AppData\Local\Temp\tmppky63n59
  • Docker/ECS: tmpfile.name: 6

Why does tempfile.TemporaryFile().name produce valid output in one case, but no the other? How can I fix it?

KingOtto
  • 840
  • 5
  • 18
  • 2
    There is no guarantee that `TemporaryFile` creates an actual file. If you need a file with a name, use `NamedTemporaryFile`. – tripleee May 04 '23 at 08:40

1 Answers1

2

This is because for tempfile.TemporaryFile():

The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose file attribute is the underlying true file object.

instead, you should use tempfile.NamedTemporaryFile():

This function operates exactly as TemporaryFile() does, except that the file is guaranteed to have a visible name in the file system (on Unix, the directory entry is not unlinked). That name can be retrieved from the name attribute of the returned file-like object.

so:

import tempfile
import boto3

with tempfile.NamedTemporaryFile() as tmpfile:
    s3 = boto3.client('s3')
    s3.download_file("my_bucket", "my_object", tmpfile.name)
Paolo
  • 21,270
  • 6
  • 38
  • 69
  • What's the point of `TemporaryFile` then, anyway? Seems like it's only coming with the downside of not guaranteeing to provide a valid name... Performance? – KingOtto May 04 '23 at 08:48
  • Indeed, if Python can get away with using a simple memory buffer without hitting the disk, it will be like 1000x more performant. – tripleee May 04 '23 at 09:30