Stuck at translation of FTP-uploadscript from Python2.x towards Python3.x

Question

Python script for ftp-upload of various types of files from local Raspberry to remote Webserver: original is running on several Raspberries under Python2.x & Raspian_Buster (and earlier Raspian_versions) without any problems.

The txt-file for this upload is generated by a lua-script-setup like the one below

   file = io.open("/home/pi/PVOutput_Info.txt", "w+")
   -- Opens a file named PVOutput_Info.txt (stored under the designated sub-folder of Domoticz)
    file:write(" === PV-generatie & Consumptie === \n")
    file:write("  Datum = " .. WSDatum .. "\n")
    file:write("  Tijd  = " .. WSTijd ..  "\n")
    file:close() -- closes the open file

    os.execute("chmod a+rw /home/pi/PVTemp_Info.txt")

Trying to upgrade this simplest version towards use with Python3.x & Raspian_Bullseye, but stuck with solving the reported error. It looks as if the codec now has a problem with a byte 0xb0 in the txt-file. Any remedy or hint to circumvent this problem?

#!/usr/bin/python3
# (c)2017 script compiled by Toulon7559 from various material from forums, version 0.1 for upload of *.txt to /
# Original script running under Python2.x and Raspian_Buster
# Version 0165P3 of 20230201 is an experimental adaptation towards Python3.x and Raspian_Bullseye
# --------------------------------------------------
# Line006 = Function for FTP_UPLOAD to Server
# --------------------------------------------------
# Imports for script-operation
import ftplib
import os
# Definition of Upload_function
def upload(ftp, file):
    ext = os.path.splitext(file)[1]
    if ext in (".txt", ".htm", ".html"):
        ftp.storlines("STOR " + file, open(file))
    else:
        ftp.storbinary("STOR " + file, open(file, "rb"), 1024)

# --------------------------------------------------
# Line020 = Actual FTP-Login & -Upload
# --------------------------------------------------
ftp = ftplib.FTP("<FTP_server>")
ftp.login("<Login_UN>", "<login_PW>")
# set path to destination directory
ftp.cwd('/')
# set path to source directory
os.chdir("/home/pi/")
# upload of TXT-files
upload(ftp, "PVTemp_Info.txt")
upload(ftp, "PVOutput_Info.txt")
# reset path to root
ftp.cwd('/')
print ('End of script Misc_Upload_0165P3')
print

Putty_CLI_Command

sudo python3 /home/pi/domoticz/scripts/python/Misc_upload_0165P3a.py

Resulting report at Putty's CLI

Start of script Misc_Upload_0165P3
Traceback (most recent call last):
  File "/home/pi/domoticz/scripts/python/Misc_upload_0165P3a.py", line 39, in <module>
    upload(ftp, "PVTemp_Info.txt")
  File "/home/pi/domoticz/scripts/python/Misc_upload_0165P3a.py", line 25, in upload
    ftp.storlines("STOR " + file, open(file))
  File "/usr/lib/python3.9/ftplib.py", line 519, in storlines
    buf = fp.readline(self.maxline + 1)
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 175: invalid start byte

Martin Prikryl · Accepted Answer · 2023-02-15T10:15:54.010

I'm afraid that there's no easy mapping to the Python 3. Two simple, but not 1:1 solutions for Python 3 would be:

Consider uploading all files using a binary mode. I.e. get rid of the

if ext in (".txt", ".htm", ".html"):
     ftp.storlines("STOR " + file, open(file))
else:

Or open the text file using the actual encoding that the files use (you have to find out):
```
open(file, encoding='cp1252')
```
See Error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

If you really need the exact functionality that you had in Python 2 (that is: Upload any text file, in whatever encoding, using FTP text transfer mode), it would be more complicated. The Python 2 basically just translates any of CR/LF EOL sequences in the file to CRLF (what is the requirement of the FTP specification), keeping the rest of the file intact.

You can copy FTP.storbinary code and implement the above translation of buf byte-wise (without decoding/recording which Python 3 FTP.storlines/readline does).

If the files are not huge, a simple implementation is to load whole file to memory, convert in memory and upload. This is not difficult, if you know that all your files use the same EOL sequence. If not, the translation might be more difficult.
Or you may even give up on the translation, as most FTP servers do not care (they can handle any common EOL sequence). Just use the FTP.storbinary code as it is, only change TYPE I to TYPE A (what you need to do even if you implement the translation as per the previous point).

Btw, you also need to close the file in any case, so the correct code would be like:

with open(file) as f:
    ftp.storlines("STOR " + file, f)

Likewise for storbinary.

Thanks! Your first suggested change already effective. But for proper cleanup also looking at the bottomlines of your message. — Toulon7559, Feb 17 '23 at 08:41

Stuck at translation of FTP-uploadscript from Python2.x towards Python3.x

1 Answers1