How can I check if a file contains a string on an SFTP server in Python?

Question

I am able to connect to the server and see all the files in a given directory, however, I seem to always run into problems when I try to open a file.

The scenario is: I want to get the latest file based on date from an SFTP server, then I want to check if that file has a particular string in it. If it does, return true, if not return false. Here is my solution below:

def does_file_exists_on_sftp_server_and_contains_given_value(value):
    latest_date = 0
    latest_file = None
    retry_attempt = 0
    value_is_present= False

    while retry_attempt < 50:
        # the line below does the sftp server login and that works successfully 
        conn = vm_sftp_login()
        files = conn.listdir_attr("directory")
        for file in files:
            if file.filename.endswith(".xml") and file.st_mtime > latest_date:
                latest_date = file.st_mtime
                latest_file = file.filename
                retry_attempt = 50
        retry_attempt += 1
    latest_file_obj = conn.get(latest_file)
    file_obj = open(latest_file_obj)
    for line in file_obj:
        if value in line:
            value_is_present = True
            break
    return value_is_present

The code seems to break at this point: latest_file_obj = conn.get(latest_file).

Traceback

self = <paramiko.sftp_client.SFTPClient object at 0x000001EF8E156748>
msg = paramiko.Message(b'\x00\x00\x00\x05\x00\x00\x00\x02\x00\x00\x00\x0cNo such file\x00\x00\x00\x00')

    def _convert_status(self, msg):
        """
            Raises EOFError or IOError on error status; otherwise does nothing.
            """
        code = msg.get_int()
        text = msg.get_text()
        if code == SFTP_OK:
            return
        elif code == SFTP_EOF:
            raise EOFError(text)
        elif code == SFTP_NO_SUCH_FILE:
            # clever idea from john a. meinel: map the error codes to errno
>           raise IOError(errno.ENOENT, text)
E           FileNotFoundError: [Errno 2] No such file

Please add the full traceback of the exception you're getting. — glibdud, Jul 05 '18 at 17:37
Catch the exception and print the value of `latest_file`. That's what the server is looking for, and it's not finding it. — glibdud, Jul 05 '18 at 17:45
You sure latest_file is set? seems like there is a hole in your logic there to where latest_file cannot be set, and you blindly get what could be None — sehafoc, Jul 05 '18 at 18:03
@sehafoc latest_file is set, the value when i check the variable is something like latestfile = str'filename.xml' — floormind, Jul 05 '18 at 18:12
You're providing the directory when you do the listdir, but it looks like it's not in the variable. Do you need to add it? — glibdud, Jul 05 '18 at 18:14
glib beat me to it haha, does the get need to be "directory/filename.xml" — sehafoc, Jul 05 '18 at 18:24
you guys are right!... thanks a lot but still unable to open the file and read through it. — floormind, Jul 05 '18 at 18:33
@glibdud you're right but then i now do something like this latest_file_obj = conn.get(f"vm-hints/{latest_file}") ... the result of lates_file_obj is None.. how come — floormind, Jul 05 '18 at 18:39
See the [Paramiko SFTP documentation](http://docs.paramiko.org/en/2.4/api/sftp.html#paramiko.sftp_client.SFTPClient.get). `get()` expects you to pass a local filename to copy the file to. Although it doesn't explicitly say, it probably always returns `None`. — glibdud, Jul 05 '18 at 18:42

score 0 · Answer 1 · answered Jul 05 '18 at 18:25

0

I think your problem might be that that particular line of code is trying to access the file latest.xml instead of directory/latest.xml. I'm not personally familiar with this library, but if conn.listdir_attr() works similarly to python's os.listdir(), then it omits the name of the folder from each of the filenames it returns. Thus, if you want to open a file from that list, you have to add the directory name back in.

So, try

latest_file_obj = conn.get('directory/' + latest_file)

instead and see if it works.

answered Jul 05 '18 at 18:25

Green Cloak Guy

23,793
4
33
53

you're right but then i now do something like this latest_file_obj = conn.get(f"vm-hints/{latest_file}") ... the result of lates_file_obj is None.. how come – floormind Jul 05 '18 at 18:39
If you can, I recommend doing step-by-step debugging with a tool like pdb, or at least having the console print the value of every variable in the scope - it could be that the functions aren't behaving quite as you expect them to, and unfortunately I don't have enough experience with this library to make any more direct suggestions. – Green Cloak Guy Jul 05 '18 at 18:48

score 0 · Answer 2 · answered Jul 09 '18 at 09:01

You need to specify a path to the file in Connection.get, as already answered by @a625993.
Connection.get does not return anything. It downloads the remote file to a local path specified by the localpath argument. If you do not specify the argument, it downloads the file to the current working directory.

If you really want to read the file to a variable (what I understand that you actually do not want), you need to use .getfo, like:
```
flo = BytesIO()
sftp.getfo(remotepath, flo)
```
Alternatively, use Paramiko library directly (without the pysftp wrapper).
See Read a file from server with ssh using python.

How can I check if a file contains a string on an SFTP server in Python?

2 Answers2