Argparse optional stdin argument

Question

I am trying to specify an optional argument that takes stdin. This will be mainly used for piping data in my program, so someprog that outputs | python my_prog.

I followed the argparse documentation and I read a lot of questions/answers on this on Stackoverflow but none of them seem to work for me.

Here's what I originally have:

parser = argparse.ArgumentParser(description='Upgrade Instance.')
parser.add_argument('--app', '-a', dest='app', action='store', required=True)
parser.add_argument('--version', '-v', dest='version', action='store', default='', required=False)
parser.add_argument('--config', '-c', dest='config', action='store', default = '', required=False)
args = parser.parse_args()

Now what I want to do is allow the user to pass in version using a pipe, instead of passing it in.

I added parser.add_argument('infile', nargs='?', type=argparse.FileType('r'), default=sys.stdin) to the top but that makes it a positional argument. How is that possible? I thought nargs=? makes it optional.

I need it to be an optional argument. So I changed it to:

parser.add_argument('--infile', nargs='?', type=argparse.FileType('r'), default=sys.stdin)

This makes it an optional argument, but the program hangs waiting for stdin as thats default, if no pipe is passed. Removing the default=sys.stdin and piping something into my program I get:

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

when running it. When I print args, I get: Namespace(app='app', config='', g=False, hosts='03.app', infile=None, version='').

It seems what I am doing is very simple, common and many people asked about it. But it doesn't seem to be working with me.

Any suggestions on how I can get it working?

Answer 1

This does it... without specifying arguments. If you pass pipe input to the program it goes, it you don't, it still goes. raw_input() will work as well.

import sys

if not sys.stdin.isatty():
    stdin = sys.stdin.readlines()
    print stdin
    sys.stdin = open('/dev/tty')
else:
    print "No stdin"

test_raw = raw_input()
print test_raw

Demo -

rypeck$ echo "stdin input" | python test_argparse.py -a test 
['stdin input\n']
raw_input working!
raw_input working!
rypeck$ python test_argparse.py -a test
No stdin
raw_input working!
raw_input working!

Answer 2

I was poking at this issue myself and found a small improvement on the options here--hopefully it'll help future travelers. It's not 100% clear if you were hoping to only read from stdin when you provide a flag, but it seems that may have been your goal; my answer is predicated on this assumption. I'm working in 3.4, in case that becomes an issue...

I can declare a single optional argument like:

parser.add_argument("-v", "--version", nargs='?', const=sys.stdin, action=StreamType)

I'm not specifying a default because it is used only if the option is completely absent, while the const is used only if the flag is present but has no arguments. StreamType is a tiny argparse.Action subclass I wrote which just tries to read a line from the stdin stream and just saves the raw value if that doesn't work:

class StreamType(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        try:
            setattr(namespace, self.dest, values.readline().strip())
        except AttributeError:
            setattr(namespace, self.dest, values)

This should produce something like:

$ blah --version v1.1.1
Namespace(version='v1.1.1')

$ echo "v1.0.3" | blah --version
Namespace(version='v1.0.3')

Answer 3

What do you do with args.infile? since you get a Namespace, argparse is not the part that is hanging or giving the error.

p = argparse.ArgumentParser()
p.add_argument('--infile', type=argparse.FileType('r'),default='-')
# p.add_argument('infile', nargs='?', type=argparse.FileType('r'),default='-') # positional version
args = p.parse_args()
print(args)
print args.infile.read()
-----------
$ cat myprog.py | python myprog.py --infile -
$ cat myprog.py | python myprog.py
$ python myprog.py myprog.py  # with the positional version
$ python myprog.py - < myprog.py  # with the positional version

echos the code nicely. The second call works with the 'optional positional' as well.

There is an unfortunate overlap in terminology, optional/positional and optional/required.

If a positional argument (yes, another use of 'positional') has a prefix character like - or -- it is called optional. By default its required parameter is False, but you may set it to True. But if the argument is 'infile' (no prefix), it is positional, even though with ? is is optional (not required).

By the way, default action is 'store', so you don't need to specify that. Also you don't need to specify required, unless it is True.

With a FileType, a handy way of specifying stdin is -.

Don't use '?' with --infile unless you really want a None

Answer 4

I'm not sure if I got your question correctly, but even if not, as I came here trying to solve my problem, this answer may help.

import argparse
import sys

def get_args():
    parser = argparse.ArgumentParser(prog="my program")
    parser.add_argument("--version", type=str, default = None)
    args,unknown = parser.parse_known_args()
    print(f"normal unknown: {unknown}")

    if not sys.stdin.isatty():
        stdin = sys.stdin.readlines()
        stdin = ' '.join( (x.strip() for x in stdin ))
        args, unknown2 = parser.parse_known_args(stdin.split(), namespace = args)
        unknown.extend(unknown2)
        print(f"stdin unknown: {unknown2}")

    print(f"args: {args}")


get_args()

Now, I get

echo A --version=1.2.3 B | parse.py --C=some_value
normal unknown: ['--C=some_value']
stdin unknown: ['A', 'B']
args: Namespace(version='1.2.3')