0

I inherited some code written in python 2.7 that I updated to 3.7 However, a piece of the logging broke in the process. The code uses a wrapper script (A) to invoke the main script (B) with subprocess.Popen(). Script B uses multiprocess.Pool(). Activity in the Pool() used to write to the logfile. After switching to Python3, it does not. No portion of this code was modified in the switch to 3.7. I tried adding sys.stdout = open(logfile, 'w') to script A per this thread with no luck either.

Script A:

def runTool(path, args):
    env = pathToPythonExe
    logfile = getLogFile()  # other function to create text file
    proc = subprocess.Popen(scriptB, env=env, shell=False, stdout=logfile, stderr=logfile)
    out = proc.communicate(None)
    if out is not None and len(out) > 0:
        print(out)

Script B:

def processMyData(arg1, arg2, data_dictionary):
    index = str(data_dictionary[1][2])
    print('Processing item: ' + index)  # No longer logged at python3
    doWork

def mainFunction():
    print('This message gets logged successfully')
    pool = multiprocessing.Pool(processes=4, maxtasksperchild=1)
    pool.map(partial(processMyData, arg1, arg2), data_dictionary.items(), chunksize=1)  #Prints in here do not get logged anymore
    pool.close()
    pool.join()
PythonScrub
  • 3
  • 3
  • As a follow-up test, I wrote a tiny pair of scripts to print some numbers using Pool().map(). If I run it with python3, the prints within the pool are not logged. Using 2.7 they are. – PythonScrub Jan 13 '21 at 18:27

1 Answers1

0

I found an incredibly inelegant solution by modifying script B.

import glob

fileList = glob.glob('C:\Temp\*.log')
newest_file = max(fileList, key=os.path.getctime)
sys.stdout = open(newest_file, 'a+')

...

def processMyData(arg1, arg2, data_dictionary):
    index = str(data_dictionary[1][2])
    print('Processing item: ' + index)
    sys.stdout.flush()  # Added line
    doWork
PythonScrub
  • 3
  • 3