I want to run a python script and capture the output on a text file as well as want to show on console.
I want to specify it as a property of the python script itself. NOT to use the command echo "hello world" | tee test.txt on command prompt every time.
Within script I tried:
sys.stdout = open('log.txt','w')
But this does not show the stdout output on screen.
I have heard about logging module but I could not get luck using that module to do the job.
You can use shell redirection while executing the Python file:
python foo_bar.py > file
This will write all results being printed on stdout from the Python source to file to the logfile.
Or if you want logging from within the script:
import sys
class Logger(object):
def __init__(self):
self.terminal = sys.stdout
self.log = open("logfile.log", "a")
def write(self, message):
self.terminal.write(message)
self.log.write(message)
def flush(self):
# this flush method is needed for python 3 compatibility.
# this handles the flush command by doing nothing.
# you might want to specify some extra behavior here.
pass
sys.stdout = Logger()
Now you can use:
print "Hello"
This will write "Hello" to both stdout and the logfile.
I got the way to redirect the out put to console as well as to a text file as well simultaneously:
te = open('log.txt','w') # File where you need to keep the logs
class Unbuffered:
def __init__(self, stream):
self.stream = stream
def write(self, data):
self.stream.write(data)
self.stream.flush()
te.write(data) # Write the data of stdout here to a text file as well
sys.stdout=Unbuffered(sys.stdout)
Use logging module to debug and follow your app
Here is how I managed to log to file and to console / stdout
import logging
logging.basicConfig(level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
filename='logs_file',
filemode='w')
# Until here logs only to file: 'logs_file'
# define a new Handler to log to console as well
console = logging.StreamHandler()
# optional, set the logging level
console.setLevel(logging.INFO)
# set a format which is the same for console use
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
# Now, we can log to both ti file and console
logging.info('Jackdaws love my big sphinx of quartz.')
logging.info('Hello world')
read it from source: https://docs.python.org/2/howto/logging-cookbook.html
Based on Amith Koujalgi's answer, here's a simple module you can use for logging -
transcript.py:
"""
Transcript - direct print output to a file, in addition to terminal.
Usage:
import transcript
transcript.start('logfile.log')
print("inside file")
transcript.stop()
print("outside file")
"""
import sys
class Transcript(object):
def __init__(self, filename):
self.terminal = sys.stdout
self.logfile = open(filename, "a")
def write(self, message):
self.terminal.write(message)
self.logfile.write(message)
def flush(self):
# this flush method is needed for python 3 compatibility.
# this handles the flush command by doing nothing.
# you might want to specify some extra behavior here.
pass
def start(filename):
"""Start transcript, appending print output to given filename"""
sys.stdout = Transcript(filename)
def stop():
"""Stop transcript and return print functionality to normal"""
sys.stdout.logfile.close()
sys.stdout = sys.stdout.terminal
from IPython.utils.io import Tee
from contextlib import closing
print('This is not in the output file.')
with closing(Tee("outputfile.log", "w", channel="stdout")) as outputstream:
print('This is written to the output file and the console.')
# raise Exception('The file "outputfile.log" is closed anyway.')
print('This is not written to the output file.')
# Output on console:
# This is not in the output file.
# This is written to the output file and the console.
# This is not written to the output file.
# Content of file outputfile.txt:
# This is written to the output file and the console.
The Tee class in IPython.utils.io does what you want, but it lacks the __enter__ and __exit__ methods needed to call it in the with-statement. Those are added by contextlib.closing.
I devised an easier solution. Just define a function that will print to file or to screen or to both of them. In the example below I allow the user to input the outputfile name as an argument but that is not mandatory:
OutputFile= args.Output_File
OF = open(OutputFile, 'w')
def printing(text):
print text
if args.Output_File:
OF.write(text + "\n")
After this, all that is needed to print a line both to file and/or screen is: printing(Line_to_be_printed)
Here is a simple context manager that prints to the console and writes the same output to an file. It also writes any exceptions to the file.
import traceback
import sys
# Context manager that copies stdout and any exceptions to a log file
class Tee(object):
def __init__(self, filename):
self.file = open(filename, 'w')
self.stdout = sys.stdout
def __enter__(self):
sys.stdout = self
def __exit__(self, exc_type, exc_value, tb):
sys.stdout = self.stdout
if exc_type is not None:
self.file.write(traceback.format_exc())
self.file.close()
def write(self, data):
self.file.write(data)
self.stdout.write(data)
def flush(self):
self.file.flush()
self.stdout.flush()
To use the context manager:
print("Print")
with Tee('test.txt'):
print("Print+Write")
raise Exception("Test")
print("Print")
I've tried a few solutions here and didn't find the one that writes into file and into console at the same time. So here is what I did (based on this answer)
class Logger(object):
def __init__(self):
self.terminal = sys.stdout
def write(self, message):
with open ("logfile.log", "a", encoding = 'utf-8') as self.log:
self.log.write(message)
self.terminal.write(message)
def flush(self):
#this flush method is needed for python 3 compatibility.
#this handles the flush command by doing nothing.
#you might want to specify some extra behavior here.
pass
sys.stdout = Logger()
This solution uses more computing power, but reliably saves all of the data from stdout into logger file and uses less memeory. For my needs I've added time stamp into self.log.write(message) aswell. Works great.
This way worked very well in my situation. I just added some modifications based on other code presented in this thread.
import sys, os
orig_stdout = sys.stdout # capture original state of stdout
te = open('log.txt','w') # File where you need to keep the logs
class Unbuffered:
def __init__(self, stream):
self.stream = stream
def write(self, data):
self.stream.write(data)
self.stream.flush()
te.write(data) # Write the data of stdout here to a text file as well
sys.stdout=Unbuffered(sys.stdout)
#######################################
## Feel free to use print function ##
#######################################
print("Here is an Example =)")
#######################################
## Feel free to use print function ##
#######################################
# Stop capturing printouts of the application from Windows CMD
sys.stdout = orig_stdout # put back the original state of stdout
te.flush() # forces python to write to file
te.close() # closes the log file
# read all lines at once and capture it to the variable named sys_prints
with open('log.txt', 'r+') as file:
sys_prints = file.readlines()
# erase the file contents of log file
open('log.txt', 'w').close()
The following works on Linux where tee is natively available. To also use it on Windows, either use unxutils or find another tee-like option. Run your script using,
python -u <YOUR_SCRIPT>.py | tee log.txt
This will print in the console as well as log to the file.
Also, don't forget to use the -u flag otherwise, you won't see any output in the console.
Based on Brian Burns edited answer, I created a single class that is easier to call:
class Logger(object):
"""
Class to log output of the command line to a log file
Usage:
log = Logger('logfile.log')
print("inside file")
log.stop()
print("outside file")
log.start()
print("inside again")
log.stop()
"""
def __init__(self, filename):
self.filename = filename
class Transcript:
def __init__(self, filename):
self.terminal = sys.stdout
self.log = open(filename, "a")
def __getattr__(self, attr):
return getattr(self.terminal, attr)
def write(self, message):
self.terminal.write(message)
self.log.write(message)
def flush(self):
pass
def start(self):
sys.stdout = self.Transcript(self.filename)
def stop(self):
sys.stdout.log.close()
sys.stdout = sys.stdout.terminal
To redirect output to a file and a terminal without modifying how your Python script is used outside, you could use pty.spawn(itself):
#!/usr/bin/env python
"""Redirect stdout to a file and a terminal inside a script."""
import os
import pty
import sys
def main():
print('put your code here')
if __name__=="__main__":
sentinel_option = '--dont-spawn'
if sentinel_option not in sys.argv:
# run itself copying output to the log file
with open('script.log', 'wb') as log_file:
def read(fd):
data = os.read(fd, 1024)
log_file.write(data)
return data
argv = [sys.executable] + sys.argv + [sentinel_option]
rc = pty.spawn(argv, read)
else:
sys.argv.remove(sentinel_option)
rc = main()
sys.exit(rc)
If pty module is not available (on Windows) then you could replace it with teed_call() function that is more portable but it provides ordinary pipes instead of a pseudo-terminal -- it may change behaviour of some programs.
The advantage of pty.spawn and subprocess.Popen -based solutions over replacing sys.stdout with a file-like object is that they can capture the output at a file descriptor level e.g., if the script starts other processes that can also produce output on stdout/stderr. See my answer to the related question: Redirect stdout to a file in Python?
Based on @Arnold Suiza's answer, here is a function you can run once in the beginning and afterwards all will be immediately printed to stdout & file:
def print_to_file(filename):
orig_stdout = sys.stdout # capture original state of stdout
class Unbuffered:
def __init__(self, filename):
self.stream = orig_stdout
self.te = open(filename,'w') # File where you need to keep the logs
def write(self, data):
self.stream.write(data)
self.stream.flush()
self.te.write(data) # Write the data of stdout here to a text file as well
self.te.flush()
sys.stdout=Unbuffered(filename)
Now just run print_to_file('log.txt') at program start and you're good to go!
The answer provided by user2033758 is concise; however, it does not work if the code involves libraries using sys.stdout methods such as sys.stdout.flush(). I slightly modified this answer by adding the flush() method which simply calls the sys.stdout's flush() method, and I also moved the file object into the Unbuffered class:
import sys
class Unbuffered(object):
def __init__(self, stream, filepath):
self.stream = stream
self.file = open(filepath, 'w') # File where you need to keep the logs
def write(self, data):
self.stream.write(data)
self.stream.flush()
self.file.write(data) # Write the data of stdout here to a text file as well
def flush(self):
self.stream.flush()
sys.stdout=Unbuffered(sys.stdout, filepath='console_text.txt')
I tried this:
"""
Transcript - direct print output to a file, in addition to terminal.
Usage:
import transcript
transcript.start('logfile.log')
print("inside file")
transcript.stop()
print("outside file")
"""
import sys
class Transcript(object):
def __init__(self, filename):
self.terminal = sys.stdout, sys.stderr
self.logfile = open(filename, "a")
def write(self, message):
self.terminal.write(message)
self.logfile.write(message)
def flush(self):
# this flush method is needed for python 3 compatibility.
# this handles the flush command by doing nothing.
# you might want to specify some extra behavior here.
pass
def start(filename):
"""Start transcript, appending print output to given filename"""
sys.stdout = Transcript(filename)
def stop():
"""Stop transcript and return print functionality to normal"""
sys.stdout.logfile.close()
sys.stdout = sys.stdout.terminal
sys.stderr = sys.stderr.terminal
you can redirect the output to a file by using >> python with print rint's "chevron" syntax as indicated in the docs
let see,
fp=open('test.log','a') # take file object reference
print >> fp , "hello world" #use file object with in print statement.
print >> fp , "every thing will redirect to file "
fp.close() #close the file
checkout the file test.log you will have the data and to print on console just use plain print statement .
