How to debug python CLI that takes stdin?

Question

I'm trying to debug a Python CLI I wrote that can take its arguments from stdin. A simple test case would have the output of

echo "test" | python mytool.py

be equivalent to the output of

python mytool.py test

I'd like to debug some issues with this tool, so I tried to run this:

echo "test" | pdb mytool.py

But I get this output, then pdb exits:

> /path/to/mytool.py(5)<module>()
-> '''
(Pdb) *** NameError: name 'test' is not defined
(Pdb)

The same thing occurs when I add -m python to the shebang, and if I run pdb.set_trace() inside the script.

What's going on here?

Can you change your script to accept input from a file other than stdin? — nmichaels, Jun 19 '13 at 18:57
possible duplicate of [Python CLI program unit testing](http://stackoverflow.com/questions/13493288/python-cli-program-unit-testing) — Paul Sweatte, Feb 04 '14 at 01:30
@user1901786 How do you access the stdin in the CLI script? Using `sys.stdin`? Also, do you just want to write a test for this or do you need to start the debugger within the script? If you need the debugger, at which point do you want it to engage the script? Sorry, the question is a bit unclear to me. — famousgarkin, Mar 18 '14 at 20:31
The general idea is to be able to start the debugger in a script that takes stdin (in this case, via `sys.stdin`). I'd prefer to not have to take apart the script and make it take file input, since I'm testing the logic that determines whether the script reads stdin or not. Really, I'm just trying to do what I described in the question, and the error PDB throws doesn't make much sense to me. — whereswalden, Mar 20 '14 at 00:26
What I would do is get PyCharm community edition and try debugging with that. Using standard local debugging should work if you are ok with manually entering or pasting your input. If you require debugging inputs from a pipe or redirect, you may have to use remote debugging which is slightly more complicated to set up. — Preston Landers, Oct 22 '14 at 12:37
I agree with @PrestonLanders. Nobody actually uses pdb directly. — simonzack, Nov 14 '14 at 16:10

score 20 · Answer 1 · edited Aug 29 '21 at 16:11

Another option is to create you own Pdb object, and set there the stdin and stdout. My proof of concept involves 2 terminals, but for sure some work can be merged some kind of very unsecure network server.

Create two fifos:
```
mkfifo fifo_stdin
mkfifo fifo_stdout
```
In one terminal, open stdout on background, and write to stdin:
```
cat fifo_stdout &
cat > fifo_stdin
```

In your python code/console create the pdb object, and use it:

import pdb
mypdb=pdb.Pdb(stdin=open('fifo_stdin','r'), stdout=open('fifo_stdout','w'))
...
mypdb.set_trace()
...

Profit!

You should be able to use pdb on the first console.

The only drawback is having to use your custom pdb, but some monkey patching at init (PYTHONSTARTUP or similar) can help:

import pdb
mypdb=pdb.Pdb(stdin=open('fifo_stdin','r'), stdout=open('fifo_stdout','w'))
pdb.set_trace=mypdb.set_trace

Handy, also for example if you want to debug Flask when using Pytest. Since Python 3.7 there is breakpoint(): `breakpoint(stdin=open('fifo_stdin', 'r'), stdout=open('fifo_stdout', 'w'))` — DZet, Sep 14 '21 at 13:36

score 6 · Answer 2 · answered Nov 14 '14 at 16:17

You can use another file descriptor. With bash you can create a new file descriptor with:

exec 3<> test.txt

And then on your python file have something like:

#!/usr/bin/python

# Use fd 3 as another stdin file.
import os
stdin=os.fdopen(3)

while True:
    s=stdin.readline()
    import pdb; pdb.set_trace()
    print len(s)

Just runing your script will use that test.txt as input, and you can use stdin on stdin. It can be used as well with pipes if you need.

I'd prefer a solution that doesn't require modifying the tool's code. — whereswalden, Nov 14 '14 at 18:09

score 6 · Answer 3 · answered Jan 24 '18 at 19:35

Your controlling TTY is still a terminal, right? Use this instead of pdb.set_trace.

def tty_pdb():
    from contextlib import (_RedirectStream,
                            redirect_stdout, redirect_stderr)
    class redirect_stdin(_RedirectStream):
        _stream = 'stdin'
    with open('/dev/tty', 'r') as new_stdin, \
         open('/dev/tty', 'w') as new_stdout, \
         open('/dev/tty', 'w') as new_stderr, \
         redirect_stdin(new_stdin), \
         redirect_stdout(new_stdout), redirect_stderr(new_stderr):
        __import__('pdb').set_trace()

Haven't gotten readline to autocomplete under these circumstances. Up arrow won't work either, or any other of the readline niceties.

score 2 · Answer 4 · edited Nov 14 '14 at 16:21

2

When you use pdb(or any other python debugger) it acquires stdin for debug commands, that's why you get NameError: name 'test' is not defined.

For example this command will quit debugger at the begging of a runtime and you wont get this error(nor interactive debugging) for one run:

(echo cont;echo "test") | python -m pdb mytool.py

edited Nov 14 '14 at 16:21

lennon310

12,503
11
43
61

answered Nov 14 '14 at 16:04

irqed

649
4
15

2

While this explains what's going on, it doesn't provide an answer. – whereswalden Nov 14 '14 at 18:08
2

@whereswalden in fact, the 'answer' is that pdb is broken and stupid for taking input from stdin. :/ – Otheus Aug 24 '18 at 10:10

How to debug python CLI that takes stdin?

4 Answers4

Linked