2

I am trying to pass a dictionary consisting of arguments to a python script and later use that dictionary in the script. I am saving the dictionary in the environment variable of airflow.

For example, the script should run on the command line as, python test.py dict_name

The dict_name looks like this,

{ "path": "firstlevel/secondlevel", "name": "xyz", "key": "12345" }

And, I need to use the keys of above dictionary in place of the actual values in my python script i.e. test.py

I tried using data = sys.argv[1] in the init function of a class. But, it gives me a list instead of dictionarly. And that way, using features of dictionary is not possible.

def __init__(self, data = sys.argv[1]):
    print data

I am instantiating the object like this,

f = class_name(sys.argv[1:])

The output I get is,

['path', ':', 'firstlevel/secondlevel', 'name', ':', 'xyz', 'key', ':', '12345']

My objective is to get the data as a dictionary and not a list in python script because I want to replace the values on my python script by keys only.

Here is the usecase to explain my point in a clearer way.

fun(name = 'xyz', key = 12345)

To 

fun(sys.argv['name'], sys.argv['key'])

Traceback

Traceback (most recent call last): File "test.py", line 13, in objec_name= class_name(sys.argv[1:]) File "test.py", line 6, in init data = json.loads(sys.argv[1]) File "C:\Users\Sarvesh Pandey\AppData\Local\Programs\Python\Python38\lib\json_init_.py", line 357, in loads return _default_decoder.decode(s) File "C:\Users\Sarvesh Pandey\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end())

Harshal Parekh
  • 5,918
  • 4
  • 21
  • 43
Sarvesh Pandey
  • 322
  • 7
  • 17
  • 2
    You cannot pass in a python dictionary. However, you can pass in a JSON string and convert it to a dictionary, checkout this answer https://stackoverflow.com/questions/18006161/how-to-pass-dictionary-as-command-line-argument-to-python-script – mkhayata Mar 21 '20 at 21:30
  • I tried following this thread. Following is the error I got, data = json.load(data) File "C:\Python27\lib\json\__init__.py", line 287, in load return loads(fp.read(), AttributeError: 'list' object has no attribute 'read' – Sarvesh Pandey Mar 21 '20 at 21:47
  • What are you trying to accomplish exactly? What's the use case of calling the script on the command line? I feel like this is an [XY problem](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem), and maybe you want to use something like `argparse` or `pickle` instead, depending on what you're trying to do. – wjandrea Mar 21 '20 at 22:06

1 Answers1

1

JSON - or JavaScript Object Representation is one of the way of taking Python objects and converting them into a string-like representation, suitable for passing around to multiple languages. 

python saver.py '{"names": ["J.J.", "April"], "years": [25, 29]}'

In your python script, do this:

import json
data=json.loads(argv[1])

This will give you back a dictionary representing the data you wanted to pass in.

You can then apply to the function as you intended. 

fun(data['name'], data['key'])

Edit: You also need to consider the parsing issue

E.g.

if you run print sys.argv[1] you probably get '{favorited: which the json module cannot decode into a json object.

try escaping your inner quotes so it is passed as 1 argument like so:

"{"\""favorited"\"": false, "\""contributors"\"": null}"
AzyCrw4282
  • 7,222
  • 5
  • 19
  • 35
  • I used the similar command in the command line. My command is, – Sarvesh Pandey Mar 21 '20 at 22:28
  • command: python .\customer_trading_profile.py '{"path": "firstlevel/secondlevel","name": "xyze","key": "12345"}'. When I print data without using json, I get ['{"path": "firstlevel/secondlevel","name": "xyze","key": "12345"}']. the dict args in list. With json, I get the error. data = json.loads(sys.argv[1]) in loads return _default_decoder.decode(s) in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) in raw_decode obj, end = self.scan_once(s, idx)ValueError: Expecting property name: line 1 column 2 (char 1) – Sarvesh Pandey Mar 21 '20 at 22:41
  • 1
    For json, seems like you are not passing in a correct type of string that can be parsed. See [here](https://stackoverflow.com/a/36599122/6505847) and change the strring you are passing in. – AzyCrw4282 Mar 21 '20 at 22:47
  • You are right. Following this stackoverflow thread now. https://stackoverflow.com/questions/25707558/json-valueerror-expecting-property-name-line-1-column-2-char-1. Probably that would help. Thanks. – Sarvesh Pandey Mar 21 '20 at 22:52
  • Json Data I am passing is this. { "cust_raw_path": "abc/xyx", "cust_feat_path": "pqr/mno", "account_name": "abc", "account_key": "BA==F+C/S" } Is it not a correct type of string? – Sarvesh Pandey Mar 21 '20 at 23:16
  • 1
    Try this (add the single quotation when passing from the cmd line) `'{ "cust_raw_path": "abc/xyx", "cust_feat_path": "pqr/mno", "account_name": "abc", "account_key": "BAFCS" }'`. It is a valid json. You can use here to check if its valid or not https://jsonlint.com/ – AzyCrw4282 Mar 21 '20 at 23:20
  • JSON string is valid. Used singled quote last time as well. I also confirmed it. Still throwing an error. I wrote even a sample code. There as well I am facing the same issue. Actually working on a project where I need to avoid hard coding in python script. – Sarvesh Pandey Mar 21 '20 at 23:56
  • That's weird. Please update the new error trace to your question. – AzyCrw4282 Mar 21 '20 at 23:57
  • Command : python test.py '{"asd": "123"}' Error trace for sample code. File "test.py", line 13, in objec_name= class_name(sys.argv[1:]) File "test.py", line 6, in __init__ data = json.loads(sys.argv[1]) File "C:\Users\Sarvesh Pandey\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 357, in loads return _default_decoder.decode(s) File "C:\Users\Sarvesh Pandey\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) – Sarvesh Pandey Mar 22 '20 at 00:02
  • 1
    Found the solution. [This](https://stackoverflow.com/questions/37190282/json-argument-passed-from-the-command-line-with-python-cant-be-decoded) will definitely solve it :) – AzyCrw4282 Mar 22 '20 at 00:08
  • 1
    Finally, it worked. Now I can access data in dictionary form. {u'cust_feat_path': u'CustomerChurnRiskManagement/FeaturedData', u'account_name': u'enterprisedata'}. That's what I got. The char u is associated that I will see. Rest I'll make it done. You really saved me today. Thanks – Sarvesh Pandey Mar 22 '20 at 00:21