access a dictionary value with dynamic path using in python

Question

I have this piece of code to extract a value from a dictionary object

extracted_value = response_content["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["lastPayment"]["status"]
extracted_value = response_content["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["bill"]["dueDate"]

These are just two samples but I have a dozen of these with different key/path combinations. How can I just call these using a module and do something like this

def get_value_from_content (response_content, my_path):
    # how can I use the value in my_path as the key instead of this hard coded path ?
    extracted_value =  response_content["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["lastPayment"]["status"]
    #extracted_value = response_content using my_path is what I would like to do
    return extracted_value

#I get this from a REST API call but skipping the code here and just hard coding to ask the question here
response_content = {u'retrievePolicyBillingSummariesResponse': {u'billingSummaries': {u'policyBillingSummary': [{u'policy': {u'status': u'A', u'policyNumber': u'xyz123', u'writingCompany': u'FBI', u'renewalFlag': u'false', u'convertedRenewalOffer': u'false', u'termExpirationDate': u'2017-06-26', u'lineOfBusiness': u'PC', u'termEffectiveDate': u'2016-06-26', u'riskState': u'CA', u'insureds': {u'namedInsuredSummary': [{u'preferredPostalAddress': {u'streetAddressLine': u'1 disney', u'cityName': u'palo alto', u'zipCode': u'94100', u'isoRegionCode': u'CA'}, u'name': {u'lastName': u'DOE', u'fullName': u'john doe', u'firstName': u'john'}}]}, u'additionalInterests': {u'additionalInterest': [{u'billTo': u'N', u'name': {u'partyType': u'Organization'}}]}, u'type': u'PA', u'statusDescription': u'Active', u'dataSource': u'from_heaven'}, u'billingSummary': {u'paymentRestriction': u'false', u'nextInstallmentAmount': u'0.00', u'bill': {u'installmentNumber': u'1', u'statementDate': u'2016-06-26', u'paymentPlan': u'Direct', u'installmentAmount': u'12.00', u'totalBillAmountDue': u'1.76', u'previousBalance': u'0.00', u'dueDate': u'2016-06-26', u'billingPlan': u'ANN'}, u'lastPayment': {u'status': u'A'}, u'currentBalance': u'16.66', u'payOffAmount': u'15.66', u'isRestrictedToPay': u'false'}}]}}}

my_path = '["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["lastPayment"]["status"]'
get_extracted_item = get_value_from_content(response_content,my_path)

my_path = '["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["bill"]["dueDate"]'
get_extracted_item = get_value_from_content(response_content,my_path)

what about the module you've proposed doesn't work the way you want it to? — chickity china chinese chicken, May 10 '17 at 01:15
using string path like this is not a good solution since it needs to be parsed or evaluated, when it simply can be changed into list of keys/indexes with iterating through them — Azat Ibrakov, May 10 '17 at 05:28
Keep in mind that [Using eval is a bad practice](http://stackoverflow.com/questions/1832940/is-using-eval-in-python-a-bad-practice). — keepAlive, May 10 '17 at 11:19
Thanks, yes I agree, this is a temp solution and I am moving soon towards the answer suggested by Azat or one of the iterations — Naresh MG, May 10 '17 at 11:37

Azat Ibrakov · Answer 1 · 2017-05-10T05:23:15.747

4

first of all it will be easier to write small utility function like

def extract_from_dictionary(dictionary, *keys_or_indexes):
    value = dictionary
    for key_or_index in keys_or_indexes:
        value = value[key_or_index]
    return value

as we can see from your example there is an object called billingSummary that appears in required paths, so we can avoid boilerplate with

def get_billing_summary(response_content):
    return extract_from_dictionary(
        response_content,
        "retrievePolicyBillingSummariesResponse",
        "billingSummaries",
        "policyBillingSummary",
        0,
        "billingSummary")

then we can simply write

def get_value_from_content(response_content, *keys):
    billing_summary = get_billing_summary(response_content)
    extracted_value = extract_from_dictionary(billing_summary,
                                              *keys)
    return extracted_value

and obtain required objects like

last_payment_status = get_value_from_content(response_content,
                                             "lastPayment",
                                             "status")
bill_due_date = get_value_from_content(response_content,
                                       "bill",
                                       "dueDate")
print("last_payment_status:", last_payment_status)
print("bill_due_date:", bill_due_date)

gives us

last_payment_status: A
bill_due_date: 2016-06-26

edited May 10 '17 at 05:23

answered May 10 '17 at 05:18

Azat Ibrakov

9,998
9
38
50

this looks great, for the next few days I am going to use eval() and am going to move to this route next week or so. hopefully I can come back with questions for clarification on this any once I soak this in -:) – Naresh MG May 10 '17 at 11:39
Is there a way to see the response content in a beautiful way ? I mean easily readable format .. just like json beautifiers ? I cannot see this in a better format using a json beautifier due to the Lists in between .. not needed programmatically but just like that to try and build utilities for other responses ? (The ones with a different structure ) – Naresh MG May 12 '17 at 06:57
@NareshMG: try `from pprint import pprint; pprint(response)` – Azat Ibrakov May 12 '17 at 15:17
and y did I not think of it...had used it in the past a bit but not much and hence forgot...it does the job perfect Sire ! I will now start messing with the magic portion u spilt here and then let u know how it goes... – Naresh MG May 12 '17 at 16:09
one issue which I am seeing is the hard coding of the path , I would like to achieve a method as generic as possible (not sure to what extent it can be achieved)...and then say I just give the names of the key and parent, exactly as you have done..example..."bill", "dueDate" OR "lastPayment","status" OR "billingSummary","paymentRestriction" – Naresh MG May 24 '17 at 21:17

score 2 · Answer 2 · edited May 23 '17 at 12:10

What about using an iterative solution, as follows:

response_content = {u'retrievePolicyBillingSummariesResponse': {u'billingSummaries': {u'policyBillingSummary': [{u'policy': {u'status': u'A', u'policyNumber': u'xyz123', u'writingCompany': u'FBI', u'renewalFlag': u'false', u'convertedRenewalOffer': u'false', u'termExpirationDate': u'2017-06-26', u'lineOfBusiness': u'PC', u'termEffectiveDate': u'2016-06-26', u'riskState': u'CA', u'insureds': {u'namedInsuredSummary': [{u'preferredPostalAddress': {u'streetAddressLine': u'1 disney', u'cityName': u'palo alto', u'zipCode': u'94100', u'isoRegionCode': u'CA'}, u'name': {u'lastName': u'DOE', u'fullName': u'john doe', u'firstName': u'john'}}]}, u'additionalInterests': {u'additionalInterest': [{u'billTo': u'N', u'name': {u'partyType': u'Organization'}}]}, u'type': u'PA', u'statusDescription': u'Active', u'dataSource': u'from_heaven'}, u'billingSummary': {u'paymentRestriction': u'false', u'nextInstallmentAmount': u'0.00', u'bill': {u'installmentNumber': u'1', u'statementDate': u'2016-06-26', u'paymentPlan': u'Direct', u'installmentAmount': u'12.00', u'totalBillAmountDue': u'1.76', u'previousBalance': u'0.00', u'dueDate': u'2016-06-26', u'billingPlan': u'ANN'}, u'lastPayment': {u'status': u'A'}, u'currentBalance': u'16.66', u'payOffAmount': u'15.66', u'isRestrictedToPay': u'false'}}]}}}

my_path = [
    "retrievePolicyBillingSummariesResponse",
    "billingSummaries",
    "policyBillingSummary",
    0,
    "billingSummary",
    "lastPayment",
    "status"
]

def get_value_from_content(extraction, my_path):
    for el in my_path:
        if isinstance(extraction, dict):
            extraction = extraction.get(el, extraction)
        else:
            extraction = extraction[el]
    return extraction

extraction = get_value_from_content(response_content, my_path)
print(extraction)

The function get_value_from_content can even be shorter than before, i.e.

def get_value_from_content(extraction, my_path):
    for el in my_path:
        extraction = extraction[el]
    return extraction

This last version of get_value_from_content is nonetheless more prone to throw exceptions if one has, e.g. misread the chaining of path components. It thus remains to be determined whether the string object my_path is human- or machine-made.

Which, in the two cases, returns "A". Tested in Python 2 and 3. Note also that I favor an iterative solution over a recursive one since the former is usually faster than the latter. Between 20% and 40% faster in the present case.

That being said, this does not address the question, since my_path is originally not a list object, but a string object. To adress the question, one would first convert this string into a list of keys/indexes, and then process it as mentioned above. Or, as @Minji does, one may want to use the python built-in function eval. Even if using this function is said to be a bad practice, I wonder to what extent the use of eval cannot be described in this situation as the best way to go.

mgig · Answer 3 · 2017-05-10T01:46:05.980

0

What about a recursive function?

def get_value(response, index):
    if len(index) > 1:
        return get_value(response[index[0]], index[1:])
    else:
        return response[index[0]]

index = ["retrievePolicyBillingSummariesResponse", "billingSummaries", "policyBillingSummary", 0, "billingSummary", "lastPayment", "status"]

get_value(response_content, index)

edited May 10 '17 at 01:46

answered May 10 '17 at 01:22

mgig

2,395
4
21
36

score -1 · Answer 4 · answered May 10 '17 at 04:45

-1

eval() interprets a string as code

def get_value_from_content (response_content, my_path):
    # string is arguments name
    item = "response_content" + my_path
    return eval(item)

answered May 10 '17 at 04:45

minji

512
4
16

not able to upvote for two answers...but thanks for the answer ! – Naresh MG May 24 '17 at 20:54

access a dictionary value with dynamic path using in python

4 Answers4