0

I am writing a python script to download files from aws to a local directory using the aws commands. I also created a log file for the script so that whenever there is something wrong, the error messages could be recorded. However, it seems that log file just cannot capture any of the error messages or any regular output message from the aws command, which I can normally see from my own computer screen.

My script looks like the following:

def getAWSFiles(files,localdir,logger):
    total_files = len(files)
    count=0
    logger.info("Need to copy " + str(total_files) + "files")
    logger.info("Copying files")

    for file in files:
        if not os.path.exists(localdir):
            os.mkdir(localdir)
        awscmd=["aws","s3","cp","--profile","xyz","s3://dir0/dir1/" + str(file),localdir]
        try:
            exitcode = subprocess.call(awscmd,stdout=sys.stdout)
        except Exception:
            logger.exception('Error writing ' + file + ' to ' + localdir)

        if logger is not None:
            logger.info("Finished pulling "+ str(file))
        count += 1 ...
xyin
  • 417
  • 2
  • 7
  • 19
  • Instead of asking some AWS specific question, if you just look at what you are doing you will see that you need to log the output of a command that you are running via `subprocess.call()`. If you search for that you will find lots of answers. – Mark B Dec 15 '15 at 19:52
  • @mbaird Thanks a lot! That post helped solve the problem. – xyin Dec 15 '15 at 21:59

0 Answers0