Error Handling in python

Question

I need to create a program that will download files from FTP Server and zip it and upload it to the upload folder in FTP Server. My old code is good but my instructor requested that I should organize my code into logical functions and after organizing my code, it did not run. This is the old code that runs perfectly:

import os
import upload
import download
import zipfile
import ConfigParser
import ftputil
import shutil
import time


def main():

    #create a folder Temp on drive D:\ for later use
    path = r'D:\Temp'
    os.makedirs(path)

    #parse all the  values at configuration file named config.ini
    config = ConfigParser.ConfigParser()
    config.readfp(open('config.ini'))
    server = config.get('main', 'Server')
    username = config.get('main', 'Username')
    password = config.get('main', 'Password')
    uploads = config.get('main', 'Upload folder')
    downloads = config.get('main', 'Download folder')

    #connect to ftp
    ftp = ftputil.FTPHost(server, username, password)

    dirlist = ftp.listdir(downloads)

    #download all files from download folder to the created temp folder
    for list in dirlist:
        ftp.chdir(downloads)
        target = os.path.join(path, list)
        ftp.download(list, target)

    #zipping files
    absolute_path = r'D:\Temp'
    dirlist = os.listdir(absolute_path)
    filepath = r'D:\Temp\part2b.zip'
    zip_name = zipfile.ZipFile(filepath, 'w')
    for list in dirlist:
        get_file = os.path.join(absolute_path, list)
        zip_name.write(get_file, 'part2b\\' + list)

    #upload the zipfile named project2.zip to upload folder at ftp
    ftp.chdir(uploads)
    ftp.upload(filepath, uploads + '/part2b.zip')
    ftp.close()

#close the zipfile to remove the temp folder
    zip_name.close()

    #message
    print "Successfully uploaded all files in the upload folder"
    print ""
    print "this will close in 5 seconds....."


#delete temp folder
    shutil.rmtree(path, 'true')
    time.sleep(5)


if __name__ == '__main__':
    main()

And this is the new organize code but didn't run properly, but when I try all the function in the python command line, it is good. I don't know what's the problem here:

import os
import zipfile
import ConfigParser
import ftputil
import shutil
import time
import tempfile



def dl_function(source_folder, target_folder):
    dirlist = ftp.listdir(source_folder)
    for list in dirlist:
        source = os.path.join(source_folder, list)
        target = os.path.join(target_folder, list)
        ftp.download(source, target)

def zip_function(temp_folder):
    dirlist = os.listdir(temp_folder)
    filepath = os.path.join(temp_folder, 'project2b.zip')
    zip_name = zipfile.ZipFile(filepath, 'w')
    for list in dirlist:
        get_file = os.path.join(temp_folder, list)
        zip_name.write(get_file, 'part2b\\' + list)

def upload_function(target_folder):
    ftp.upload(filepath, target_folder + '/part2b.zip')


def main():

    temp_folder = tempfile.mkdtemp()

    #parse all the  values at configuration file named config.ini
    config = ConfigParser.ConfigParser()
    config.read(open('config.ini'))
    server = config.get('main', 'Server')
    username = config.get('main', 'Username')
    password = config.get('main', 'Password')
    uploads = config.get('main', 'Upload folder')
    downloads = config.get('main', 'Download folder')

    ftp = ftputil.FTPHost(server, username, password)

    try:

        dl_function(downloads, temp_folder)
        zip_function(temp_folder)
        upload_function(uploads)

    finally:
        ftp.close() 
        zip_name.close()        

    print "This will close in 5 seconds"

    #delete temp folder
    shutil.rmtree(temp_folder, 'true')
    time.sleep(5)

if __name__ == '__main__':
    main()

Answer 1

This is a fantastic example of how you should be using logging in your application. Normally when things blow up you can debug it in two ways

• Attach a debugger
• Look at some verbose logging

The latter is much more suitable for deployed code since you don't always have a debugger handy.

Have a look at the Logging info on the pydocs page. For your code add :-

import os
import zipfile
import ConfigParser
import ftputil
import shutil
import time
import tempfile
import logging



def dl_function(source_folder, target_folder):
    logging.trace("Running dl_function")
    dirlist = ftp.listdir(source_folder)
    logging.trace("Got a dirlist")
    for list in dirlist:
        logging.trace("Running running for item " + list + " in dirlist")
        source = os.path.join(source_folder, list)
        logging.trace("Setting source to " + source)
        target = os.path.join(target_folder, list)
        logging.trace("Setting target to " + source)
        logging.trace("Attempting FTP download from " + source + " to " + target)
        ftp.download(source, target)
        logging.info("File downloaded from FTP")

...
...
...

if __name__ == '__main__':
    logging.info("Starting downloading application")
    main()

The most important part in logging is splitting up the logging properly, so you use the correct levels (Debug, Info, Warning, Error, Critical). There is a tutorial over at the pydocs page.

By looking at the output of the logs you can see how far your program is getting in its execution process and where its dying and narrow it down to a single line or two.

Answer 2

Your upload_function and dl_function both try to use undefined variables (ftp and filepath in upload_function and ftp in dl_function).

You also have a zip_name.close() call in the finally block in main that should really belong to zip_function (which will require it's own try/finally block).

wrt/ the error message, you should run the script from a command-line window to see the whole traceback and error message.

Totally unrelated, but your naming could be improved (ie "download_files" instead of "dl_function")...