Unexpected output with glob.glob

Question

Im trying to provide wildcard support to one of CLI scripts and I am using pythons glob module for it. For a test i tried this:

>>> import glob
>>> for f in glob.glob('/Users/odin/Desktop/test_folder/*.log'):
...     print f
...
/Users/odin/Desktop/test_folder/test1.log
/Users/odin/Desktop/test_folder/test2.log
/Users/odin/Desktop/test_folder/test3.log

This works perfectly and gives the right output as i do have the 3 files given above. However when I have the same code under my argument in the CLI it fails. I do this

#code...
parser.add_argument( "-f", "--file", type=str, help="Full path to the file to upload." )
#code...
if args.file:
    for f in glob.glob(args.file):
        _upload_part(f)

I run this as

python cli.py -f /Users/odin/Desktop/test_folder/*.log

This gives me the error:

cli.py: error: unrecognized arguments: /Users/odin/Desktop/test_folder/test2.log /Users/odin/Desktop/test_folder/test3.log

I dont understand why all files are being added to the argument at once when I am going through the list one by one.

EDIT-

nargs was a step in the right direction but now this error shows up:

 `Traceback (most recent call last):
      File "cli.py", line 492, in <module>
        for f in glob.glob(args.file):
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/glob.py", line 27, in glob
        return list(iglob(pathname))
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/glob.py", line 38, in iglob
        dirname, basename = os.path.split(pathname)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 92, in split
        i = p.rfind('/') + 1
    AttributeError: 'list' object has no attribute 'rfind'`

is this on a Linux/Mac? beware of shell wildcard expansions. basically, the OS will figure out what matches your wildcard and then hand the expanded results over to your program's arguments. not what you want. vaultah's answer is correct. see also http://stackoverflow.com/questions/11456403/stop-shell-wildcard-character-expansion — JL Peyret, May 01 '15 at 18:12
if you are getting an nargs from wildcard expansion then you don't need to/shouldn't use glob.glob. one or the other. — JL Peyret, May 01 '15 at 18:30

vaultah · Accepted Answer · 2015-05-01T18:34:18.033

3

Here's what your script receives:

python cli.py -f file1 file2 file3 fileN

Where N is a total number of files matched by that pattern (Shell expands wildcards automatically). On the other hand, your file argument is configured to receive just one file/pattern, thus the simplest (and the best, in my opinion) solution is to add the nargs='+' argument:

parser.add_argument('-f', '--file', type=str, help='...', nargs='+')
# code
for f in args.file:
    print(f)

This will allow you to remove all glob calls and if args.file checks.

Another option is to quote your command line argument:

python cli.py -f '/Users/odin/Desktop/test_folder/*.log'

edited May 01 '15 at 18:34

answered May 01 '15 at 18:07

vaultah

44,105
12
114
143

I tried this, I get - `No File named *.log found at /Users/odin/Desktop/test_folder/*.log` – letsc May 01 '15 at 18:09
what does a print(":%s:" % (args.file)) show when you do this? glob.glob calls generally work very well once setup, but almost always requires me a bit of fiddling to adjust to any given use of it. – JL Peyret May 01 '15 at 18:15
It wont get to the print statement since the program receives all the files at once and throws the error I posted in my question. – letsc May 01 '15 at 18:19
I added an edit to the question on the error I get now, it was too long to post as a comment – letsc May 01 '15 at 18:28
but it shouldn't if you implement the suggestion correctly about avoiding expansion. yeah, i am ambivalent about wildcard expansions most of the time but they are quite useful when you need them. – JL Peyret May 01 '15 at 18:29
The looked so simple when I tried it out in a test script. – letsc May 01 '15 at 18:32
1

@letsc see the update. In short, you don't need to use `glob`. – vaultah May 01 '15 at 18:34
Yeah. Got rid of glob entirely and works like a charm. – letsc May 01 '15 at 18:37

Unexpected output with glob.glob

1 Answers1