Importing a python script/module that uses argparse into another python script

Question

This question may be regarded to be redundant but, to my defense, I've considered similar questions and given solutions but they don't work for me (or I just didn't understand them) as I will show.

What I want to do: I wrote a couple of python scripts that are each command-line tools (accepting several arguments using argparse) when built with pyinstaller. They all work as expected both from within pycharm's terminal and from the Ubuntu terminal. Now, I would like each of these scripts to be a module that could be called from another python script passing required arguments as I would do in the terminal.

This is one of the original scripts (I've shortened the script a bit so that it serves as a minimal example):

import sys
import argparse
import getpass


if len(sys.argv) < 2:
    print "You haven't specified any arguments. Use -h to get more details on how to use this command."
    sys.exit(1)


parser = argparse.ArgumentParser()
parser.add_argument('--username', '-u', type=str, default=None, help='Username for the login to the WebCTRL server')
parser.add_argument('--password', '-p', type=str, default=None, help='Password for the login to the WebCTRL server')
parser.add_argument('--node', '-n', type=str, default=None,
    help='Path to the point or node whose children you want to retrieve. Start querying at the lowest level with "-n /trees/geographic"')
parser.add_argument('-url', type=str, default='https://my.server.de',
    help="URL of the WebCTRL server as e.g. http://google.de")
args = parser.parse_args()


if args.username is None:
    print 'No user name specified. Login to WebCTRL needs a user name and password. Check all options for this command via -h'
    sys.exit(1)
else:
    username = args.username
if args.password is None:
    password = getpass.getpass('No password specified via -p. Please enter your WebCTRL login password: ')
else:
    password = args.password
if args.node is None:
    print 'No path to a node specified. Check all options for this command via -h'
    sys.exit(1)
if args.url is None:
    print 'No URL given. Specify the URL to the WebCTRL server analogous to http://google.de'
    sys.exit(1)
else:
    wsdlFile = args.url + '/_common/webservices/Eval?wsdl'


# This doesn't belong to my original code. It's rather for demonstration:
# Print the arguments and leave the script
print 'Username: ' + args.username
print 'Node: ' + args.node
print 'URL: ' + args.url

sys.exit(0)

As I said: From within my IDE (Pycharm) ...

$ python wc_query_test.py -u wsdl -n /trees/geographic
No password specified via -p. Please enter your WebCTRL login password: 
Username: wsdl
Node: /trees/geographic
URL: https://my.server.de

... and from the Ubuntu terminal it works just fine:

$ pyinstaller --distpath dist/. wc_query_test.py
$ ./dist/wc_query_test/wc_query_test -u wsdl -n /trees/geographic
No password specified via -p. Please enter your WebCTRL login password: 
Username: wsdl
Node: /trees/geographic
URL: https://my.server.de

Coming to the actual problem: I want the script wc_query_test.py to be a module that can be imported into another python script and executed there passing along the arguments as I do in the command line. To achieve this I followed @Waylan's instructions in this stackoverflow question.

This is the code I came up with (wc_query_test.py):

import sys
import argparse
import getpass

def main(**kwargs):
    if kwargs.username is None:
        print 'No user name specified. Login to WebCTRL needs a user name and password. Check all options for this command via -h'
        sys.exit(1)
    else:
        username = kwargs.username
    if kwargs.password is None:
        password = getpass.getpass('No password specified via -p. Please enter your WebCTRL login password: ')
    else:
        password = kwargs.password
    if kwargs.node is None:
        print 'No path to a node specified. Check all options for this command via -h'
        sys.exit(1)
    if kwargs.url is None:
        print 'No URL given. Specify the URL to the WebCTRL server analogous to http://google.de'
        sys.exit(1)
    else:
        wsdlFile = kwargs.url + '/_common/webservices/Eval?wsdl'

    # This doesn't belong to my original code. It's rather for demonstration:
    # Print the arguments and leave the script
    print 'Username: ' + username
    print 'Node: ' + kwargs.node
    print 'URL: ' + kwargs.url

    sys.exit(0)

def run():
    parser = argparse.ArgumentParser()
    parser.add_argument('--username', '-u', type=str, default=None, help='Username for the login to the WebCTRL server')
    parser.add_argument('--password', '-p', type=str, default=None, help='Password for the login to the WebCTRL server')
    parser.add_argument('--node', '-n', type=str, default=None,
        help='Path to the point or node whose children you want to retrieve. Start querying at the lowest level with "-n /trees/geographic"')
    parser.add_argument('-url', type=str, default='https://my.server.de',
        help="URL of the WebCTRL server as e.g. http://google.de")
    args = parser.parse_args()

    main(**args)

... and the script that imports the module and calls it (test.py):

import wc_query_test
wc_query_test.main(username='wsdl', password='aaaaaa', node='/trees/geographic')

When I run it in the python terminal I get:

~/PycharmProjects/webctrl$ python wc_query_test.py
~/PycharmProjects/webctrl$ python test.py 
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    wc_query_test.main(username='wsdl', password='aaaaaa', node='/trees/geographic/#geb_g', url='https://webctrl.rz-berlin.mpg.de')
  File "/home/stefan/PycharmProjects/webctrl/wc_query_test.py", line 23, in main
    if kwargs.username is None:
AttributeError: 'dict' object has no attribute 'username'

Running wc_query_test.py yields no output and I see why this is. That was just a test. But running test.py yields an error as well. I get an idea why this can't work either but I cannot put it into words. What about this run() method? Does it make sense to have it? How would I need to modify my code in order to obtain this dual functionality as described in "What I want to do:"? Thank you in advance for your help!

UPDATE: I got rid of the error message. I changed e.g. kwargs.username to kwargs.get('username') since kwargs is a dictionary. The code now looks like this:

import sys
import argparse
import getpass

def main(**kwargs):
    if kwargs.get('username') is None:
        print 'No user name specified. Login to WebCTRL needs a user name and password. Check all options for this command via -h'
        sys.exit(1)
    else:
        username = kwargs.get('username')
    if kwargs.get('password') is None:
        password = getpass.getpass('No password specified via -p. Please enter your WebCTRL login password: ')
    else:
        password = kwargs.get('password')
    if kwargs.get('node') is None:
        print 'No path to a node specified. Check all options for this command via -h'
        sys.exit(1)
    if kwargs.get('url') is None:
        print 'No URL given. Specify the URL to the WebCTRL server analogous to http://google.de'
        sys.exit(1)
    else:
        wsdlFile = kwargs.get('url') + '/_common/webservices/Eval?wsdl'

    # This doesn't belong to my original code. It's rather for demonstration:
    # Print the arguments and leave the script
    print 'Username: ' + username
    print 'Node: ' + kwargs.get('node')
    print 'URL: ' + kwargs.get('url')

    sys.exit(0)

def run():
    parser = argparse.ArgumentParser()
    parser.add_argument('--username', '-u', type=str, default=None, help='Username for the login to the WebCTRL server')
    parser.add_argument('--password', '-p', type=str, default=None, help='Password for the login to the WebCTRL server')
    parser.add_argument('--node', '-n', type=str, default=None,
        help='Path to the point or node whose children you want to retrieve. Start querying at the lowest level with "-n /trees/geographic"')
    parser.add_argument('-url', type=str, default='https://webctrl.rz-berlin.mpg.de',
        help="URL of the WebCTRL server as e.g. http://google.de")
    args = parser.parse_args()

    main(**args)

Running it in the python terminal yields as expected:

$ python test.py 
Username: wsdl
Node: /trees/geographic
URL: https://my.server.de

But building it via pyinstaller and running it as command line tool there is no output:

~/PycharmProjects/webctrl$ ./dist/wc_query_test/wc_query_test -h
~/PycharmProjects/webctrl$

How can I modify wc_query_test.py so that it takes arguments and serves as command line tool?

`kwargs` is a dict, which has no attribute 'username', just like the error states. You want item access: `kwargs['username']`. Note that `main(**args)` will also fail, since the `Namespace` object from `parse_args()` is not a mapping. A quick and dirty solution is to `main(**vars(args))`. — Ilja Everilä, May 31 '17 at 11:42
Also your `main()` function should not print an erreor message and call `sys.exit()` but raise an exception (with the error message). — bruno desthuilliers, May 31 '17 at 11:56
Without studying your code in detail, I noticed one glaring problem. It is best to call the parser in a `if __name__` block. That way it is used only when the script is called as a script, not when imported by another. Running parsers in several imported scripts is asking for trouble. — hpaulj, May 31 '17 at 19:00

score 6 · Accepted Answer · answered Jun 19 '17 at 12:00

Thanks to everyone who responded. With the help of a colleague I got the answer to my question. This is the functioning code:

wc_query_test.py:

import sys
import argparse
import getpass

def main(args):
    if args['username'] is None:
        print 'No user name specified. Login to WebCTRL needs a user name and password. Check all options for this command via -h'
        sys.exit(1)
    else:
        username = args['username']
    if args['password'] is None:
        password = getpass.getpass('No password specified via -p. Please enter your WebCTRL login password: ')
    else:
        password = args['password']
    if args['node'] is None:
        print 'No path to a node specified. Check all options for this command via -h'
        sys.exit(1)
    if args['url'] is None:
        print 'No URL given. Specify the URL to the WebCTRL server analogous to http://google.de'
        sys.exit(1)
    else:
        wsdlFile = args['url'] + '/_common/webservices/Eval?wsdl'

    # This doesn't belong to my original code. It's rather for demonstration:
    # Print the arguments and leave the script
    print 'Username: ' + args['username']
    print 'Node: ' + args['node']
    print 'URL: ' + args['url']


# The parser is only called if this script is called as a script/executable (via command line) but not when imported by another script
if __name__=='__main__':
    if len(sys.argv) < 2:
        print "You haven't specified any arguments. Use -h to get more details on how to use this command."
        sys.exit(1)
    parser = argparse.ArgumentParser()
    parser.add_argument('--username', '-u', type=str, default=None, help='Username for the login to the WebCTRL server')
    parser.add_argument('--password', '-p', type=str, default=None, help='Password for the login to the WebCTRL server')
    parser.add_argument('--node', '-n', type=str, default=None,
        help='Path to the point or node whose children you want to retrieve. Start querying at the lowest level with "-n /trees/geographic"')
    parser.add_argument('-url', type=str, default='https://webctrl.rz-berlin.mpg.de',
        help="URL of the WebCTRL server as e.g. http://google.de")
    args = parser.parse_args()

    # Convert the argparse.Namespace to a dictionary: vars(args)
    main(vars(args))
    sys.exit(0)

Now, there are three ways of executing wc_query_test which is what I wanted to achive:

1) Calling wc_query_test.py from command line:

~/PycharmProjects/webctrl$ python wc_query_test.py -u aawrg -p wgAWER -n YWERGAEWR

2) Compiling and calling wc_query_test from command line:

~/PycharmProjects/webctrl$ pyinstaller --distpath dist/. wc_query_test.py
~/PycharmProjects/webctrl$ ./dist/wc_query_test/wc_query_test -u aawrg -p wgAWER -n YWERGAEWR

3) Calling wc_query_test from another python script which goes into the direction of a module-type usage:

import wc_query_test
myDictonary = {'username':'wsdl', 'password':'aaaaaa', 'node':'/trees/geographic', 'url':'https://my.server.de'}
wc_query_test.main(myDictonary)

All three versions yield the same output as expected, for example:

~/PycharmProjects/webctrl$ ./dist/wc_query_test/wc_query_test -u aawrg -p wgAWER -n YWERGAEWR
Username: aawrg
Node: YWERGAEWR
URL: https://webctrl.rz-berlin.mpg.de

I had the same issue. Apparently I had to put the parser code inside the main function rather in the global scope of the imported script. — Wildhammer, Dec 17 '19 at 16:52

score 0 · Answer 2 · answered May 31 '17 at 21:13

The **kwargs is for passing a dictionary, or keyword=value pairs to a function. args = parser.parse_args() returns an argparse.Namespace object, not a dictionary. But vars(args) produces a dictionary from it.

In [2]: def foo1(args):
   ...:     print(args)
   ...:     print(args.foo)
   ...:     
In [3]: def foo2(**kwargs):
   ...:     print(kwargs)
   ...:     print(kwargs['foo'])

In [12]: p = argparse.ArgumentParser()
In [13]: p.add_argument('--foo');
In [14]: args = p.parse_args('--foo one'.split())
In [15]: args
Out[15]: Namespace(foo='one')

pass args to foo1:

In [16]: foo1(args)
Namespace(foo='one')
one

try it with foo2

In [17]: foo2(args)
...
TypeError: foo2() takes 0 positional arguments but 1 was given

In [20]: foo2(**args)
TypeError: foo2() argument after ** must be a mapping, not Namespace

but pass the expanded dictionary version:

In [18]: vars(args)
Out[18]: {'foo': 'one'}
In [19]: foo2(**vars(args))
{'foo': 'one'}
one

Importing a python script/module that uses argparse into another python script

2 Answers2