How to save errors from Jupyter Notebook cells to a file?

Question

I am trying to save all output (stdout and all errors) of a cell to a file. To save stdout, I am using the following:

import sys
old_stdout = sys.stdout
sys.stdout = open('test.txt', 'w')
print("Hello World! ")

In this case, the output is not displayed and gets saved in the file as expected. To save errors, I used:

#Doesn't work
sys.stderr = open('error.txt','w')
print(a) #Should raise NameError

When I run this cell, I get the error in the notebook, and not in the file as expected:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-5-de3efd936845> in <module>()
      1 #Doesn't work
----> 2 sys.stderr = open('error.txt','w')
      3 print("Test")
      4 print(a)

NameError: name 'sys' is not defined

I would like this saved in a file and not shown in the notebook. What is the correct code for this?

Have a look at this stackoverflow answer: https://stackoverflow.com/questions/34145950/is-there-a-way-to-redirect-stderr-to-file-in-jupyter — Oppy, Dec 13 '17 at 09:05

jcb91 · Accepted Answer · 2017-12-16T15:53:12.527

I think that the problem here is that IPython kernels spawned for the notebook use a ZMQInteractiveShell instance, which catches the errors before they make it to stderr, in order to send the error information to the various potential frontends (consoles, jupyter notebooks, etc). See ipykernel/ipkernel.py#L397-L413 for the code which catches exceptions, then InteactiveShell._showtraceback for the base implementation (print to sys.stderr), and ZMQInteractiveShell._showtraceback for that used by notebook kernels (send stderr-channel messages over zmq to frontends).

If you're not bothered about getting exact stderr output, you could take advantage of IPython's existing error logging, which logs errors to a StreamHandler with the prefix "Exception in execute request:". To use this, set the ipython log level, and alter the provided handler's stream:

import logging
import sys

my_stderr = sys.stderr = open('errors.txt', 'w')  # redirect stderr to file
get_ipython().log.handlers[0].stream = my_stderr  # log errors to new stderr
get_ipython().log.setLevel(logging.INFO)  # errors are logged at info level

Alternatively, to get your shell errors to print directly to a file without alteration, you can monkey-patch the _showtraceback method to print traceback to file as well as zmq message queues:

import sys
import types

# ensure we don't do this patch twice
if not hasattr(get_ipython(), '_showtraceback_orig'):
    my_stderr = sys.stderr = open('errors.txt', 'w')  # redirect stderr to file
    # monkeypatch!
    get_ipython()._showtraceback_orig = get_ipython()._showtraceback

    def _showtraceback(self, etype, evalue, stb):
        my_stderr.write(self.InteractiveTB.stb2text(stb) + '\n')
        my_stderr.flush()  # make sure we write *now*
        self._showtraceback_orig(etype, evalue, stb)

    get_ipython()._showtraceback = types.MethodType(_showtraceback, get_ipython())

Thanks for your explanation. This code however doesn't do the expected: import logging import sys sys.stderr = open('errors.txt', 'w') # redirect stderr to file get_ipython().log.setLevel(logging.INFO) A NameError is not logged in the error log file. For the second piece of code, this line throws an error: print(self.InteractiveTB.stb2text(stb), file=sys.stderr) I could not quite understand this code, hence could not debug the error. Could you run once and check whether it works on your part? I have python 2.7, if that matters. — RRC, Dec 15 '17 at 00:19
Not sure why the first variant isn't working for you, might be worth flushing the file? I'll have to check on PC tomorrow — jcb91, Dec 15 '17 at 00:44
The second variant will need a little alteration for python 2 because of the different print syntax, you could try `sys.stderr.write(self.InteractiveTB.stb2text(stb) + '\n')` instead? — jcb91, Dec 15 '17 at 00:47
Ok, I've updated both the python snippets to hopefully fix both the problems ;) — jcb91, Dec 16 '17 at 18:45

How to save errors from Jupyter Notebook cells to a file?

1 Answers1