0

I have this python function to upload file through sftp. It works fine.

def sftp_upload(destination, username, password,
                     remote_loc, source_file):
    import pysftp
    with pysftp.Connection(destination, username,
                           password, log="pysftp.log") as sftp:
        sftp.cwd(remote_loc)
        sftp.put(source_file)

    sftp.close()
    return None

The code works as expected. However, I always receive this error at the end ImportError: sys.meta_path is None, Python is likely shutting down.

How to remove this error? I'm also puzzled why code runs smoothly to the end despite the error.

In the log file, I saw the following;

INF [20220304-18:49:14.727] thr=2   paramiko.transport.sftp: [chan 0] sftp session closed.
DEB [20220304-18:49:14.727] thr=2   paramiko.transport: [chan 0] EOF sent (0)
DEB [20220304-18:49:14.728] thr=1   paramiko.transport: EOF in transport thread

Here's the stack trace;

Exception ignored in: <function Connection.__del__ at 0x000001A8B08765E0>
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\pysftp\__init__.py", line 1013, in __del__
  File "C:\ProgramData\Anaconda3\lib\site-packages\pysftp\__init__.py", line 795, in close
ImportError: sys.meta_path is None, Python is likely shutting down

I am using python v3.9

user3848207
  • 3,737
  • 17
  • 59
  • 104
  • You don't need `pysftp.close()`, that's done automatically by `with`. – Barmar Mar 04 '22 at 11:37
  • 1
    Also, note that the signature of [pysftp.Connection](https://bitbucket.org/dundeemt/pysftp/src/1c0791759688a733a558b1a25d9ae04f52cf6a64/pysftp/__init__.py#lines-76) is `host, username, private_key, password, ...` - meaning that your function calls it incorrectly as it supplies the password as private key. – FObersteiner Mar 07 '22 at 10:39

2 Answers2

2

Note: import pysftp outside the function somehow resolves the issue for me.

It's a bug in pysftp==0.2.9.

You can fix it by overriding close() to only run once:

class SFTPConnection(pysftp.Connection):
    def close(self):
        if getattr(self, '_closed', False):
            return
        self._closed = True
        super().close()

Usage:

# with pysftp.Connection(destination, username, password=password, log="pysftp.log") as sftp:  # -
with SFTPConnection(destination, username, password=password, log="pysftp.log") as sftp:       # +

References:

aaron
  • 39,695
  • 6
  • 46
  • 102
  • your solution to place `import pysftp` is the simplest and most elegant solution. Bounty given to you :). Upvoted and marked as correct answer. Thanks! – user3848207 Mar 11 '22 at 00:55
  • Any plausible explanation why placing `import pysftp` outside the function solves the problem? – user3848207 Mar 11 '22 at 02:21
  • 1
    No, somehow it just causes the [frame object](https://docs.python.org/3/reference/datamodel.html#frame-objects), which holds a reference to the `sftp` connection, to not be tracked after the function call. – aaron Mar 11 '22 at 06:03
  • I see. The error is still there but just not displayed. – user3848207 Mar 11 '22 at 06:18
  • No, the error is not there because the object is released as soon as there are no references to it, instead of when the program ends. You can observe this with `print('close')` in the overridden method and `print('sftp_upload')` after calling `sftp_upload()`. In the error case, it's `close sftp_upload close` instead of `close close sftp_upload`. – aaron Mar 11 '22 at 08:41
1

It looks like your program ends before the sftp object is garbage-collected. Then, the sftp.__del__ method is called at the middle of the program's teardown, which is causing the error.

From pysftp.py code:

def __del__(self):
    """Attempt to clean up if not explicitly closed."""
    self.close()

I personally think that it should be considered as a bug in the pysftp project.

I can think of two workarounds:

  1. Override the __del__ method:

    pysftp.Connection.__del__ = lambda x: None

  2. (Less recomended - less efficient) Explicitly delete the sftp object and trigger garbage collection:

    del sftp; import gc; gc.collect() right after the with block

guy szweigman
  • 514
  • 5
  • 8