How can I get every URL from dictionary in Python?

Question

I have a yaml string representing a menu. After I have a dictionary from it, I try to get URL simply said the key url of Parameter and every first element of list for other element that is not Parameter.

Here what I tried

import yaml
import jmespath
import pprint

setting = """
GeneralSetting:
  - Description: General Setting
MenuSetting:
  - Description: Menu Setting
  - Setting:
    - AutoMenu:
      - Description: Register menu
      - HelpText: This is for auto create menu
      - Value:
        - Sample:
          - Parameter:
            - icon: fa fa-birthday-cake
            - Active: Y
          - Sample: [Sample]
          - Sample Account: [SampleAccount]
        - Multi Journal:
          - Parameter:
            - icon: fas fa-book
            - url: MultiJournal
        - Foreign Exchange:
          - Parameter:
            - icon: fa fa-money-bill-alt
            - url: ForeignExchange
        - Loan Contract:
          - Parameter:
            - icon: fa fa-capsules
            - Active: Y
          - Loan Contract: [LoanContract,1]
          - Loan Report:
            - Loan Detail Listing: [LoanDetailListing,1]
            - Loan Application Detail Listing: [ReportR003,1]

"""

toDict = yaml.load(setting, yaml.SafeLoader)

pp = pprint.PrettyPrinter(indent=2)

# ALL
ALL       = jmespath.search('MenuSetting[].Setting[].AutoMenu[].Value[].*[]', toDict)
ParentURL       = jmespath.search('MenuSetting[].Setting[].AutoMenu[].Value[].*[0][].Parameter[].url', toDict)

NotParameter = {key:value for item in ALL for it in item for key,value in it.items() if key != "Parameter"}

# pp.pprint(NotParameter)

# print ('-----')
SubURL = jmespath.search('*[0]', NotParameter)

SubSubURL = jmespath.search('*[].*[][]', NotParameter)

pp.pprint(ParentURL)
print('---')
pp.pprint(SubURL)
print('---')
pp.pprint(SubSubURL)

From this, I could done well only for the Parent URL which is url under Parameter such as [MultiJournal,ForeignExchange] but not for child and sub-child.

I just want the final result of url as a list of this

[Sample,SampleAccount,MultiJournal,ForeignExchange,LoanContract,LoanDetailListing,Report/R003]

I tried several way but I still could not get the result of that?

Any way that I could get list of value as that? Thanks

Your YAML seems broken to me, you seem to confuse list and dict in YAML. Related: https://stackoverflow.com/questions/30221348/building-an-array-of-dictionary-items-in-yaml — β.εηοιτ.βε, Jan 17 '21 at 20:49

β.εηοιτ.βε · Accepted Answer · 2021-11-22T19:29:02.547

Although I firmly believe that your YAML structure is not the one you should have (more on that lower), here would be a query matching your expected output:

MenuSetting[].Setting[].AutoMenu[2].[Value[0].[Sample[].Sample, Sample[]."Sample Account"], Value[]."Multi Journal"[].Parameter[1].url, Value[]."Foreign Exchange"[].Parameter[1].url, Value[]."Loan Contract"[]."Loan Contract"[0], Value[]."Loan Contract"[]."Loan Report"[0]."Loan Detail Listing"[0], Value[]."Loan Contract"[]."Loan Report"[1]."Loan Application Detail Listing"[0]][][][][]

Based on the JSON equivalent of your YAML:

{
  "GeneralSetting": [
    {
      "Description": "General Setting"
    }
  ],
  "MenuSetting": [
    {
      "Description": "Menu Setting"
    },
    {
      "Setting": [
        {
          "AutoMenu": [
            {
              "Description": "Register menu"
            },
            {
              "HelpText": "This is for auto create menu"
            },
            {
              "Value": [
                {
                  "Sample": [
                    {
                      "Parameter": [
                        {
                          "icon": "fa fa-birthday-cake"
                        },
                        {
                          "Active": "Y"
                        }
                      ]
                    },
                    {
                      "Sample": [
                        "Sample"
                      ]
                    },
                    {
                      "Sample Account": [
                        "SampleAccount"
                      ]
                    }
                  ]
                },
                {
                  "Multi Journal": [
                    {
                      "Parameter": [
                        {
                          "icon": "fas fa-book"
                        },
                        {
                          "url": "MultiJournal"
                        }
                      ]
                    }
                  ]
                },
                {
                  "Foreign Exchange": [
                    {
                      "Parameter": [
                        {
                          "icon": "fa fa-money-bill-alt"
                        },
                        {
                          "url": "ForeignExchange"
                        }
                      ]
                    }
                  ]
                },
                {
                  "Loan Contract": [
                    {
                      "Parameter": [
                        {
                          "icon": "fa fa-capsules"
                        },
                        {
                          "Active": "Y"
                        }
                      ]
                    },
                    {
                      "Loan Contract": [
                        "LoanContract",
                        1
                      ]
                    },
                    {
                      "Loan Report": [
                        {
                          "Loan Detail Listing": [
                            "LoanDetailListing",
                            1
                          ]
                        },
                        {
                          "Loan Application Detail Listing": [
                            "ReportR003",
                            1
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

This query gives:

[
  "Sample",
  "SampleAccount",
  "MultiJournal",
  "ForeignExchange",
  "LoanContract",
  "LoanDetailListing",
  "ReportR003"
]

Now I believe that your YAML has a lot of issue and that you are confusing lists and dictionaries in YAML.

A list will start with a dash:

- carrot
- onions
- potatoes

A dictionary is a set of key/value pair all representing properties of the parent key:

car:
  brand: ferrari
  model: 488
  engine: "3.9 l twin-turbocharged"

You can, of course, have a list of dictionaries:

- name: duck
  sound: quack
  legs: 2
- name: cow
  sound: moo
  legs: 4

So in your YAML, if I focus on the top of it I would believe it should rather be something along the line of:

GeneralSetting:
  Description: General Setting
MenuSetting:
  Description: Menu Setting
  Setting:
    AutoMenu:
      Description: Register menu
      HelpText: This is for auto create menu
      Value:
        Sample:
          Parameter:
            icon: fa fa-birthday-cake
            Active: Y
          Sample: Sample
          Sample Account: SampleAccount
        Multi Journal:
          Parameter:
            icon: fas fa-book
            url: MultiJournal
        Foreign Exchange:
          Parameter:
            icon: fa fa-money-bill-alt
            url: ForeignExchange 
# ...

Which, of course, would totally change and simplify the query, for example, based on this YAML, the resulting JSON would be:

{
  "GeneralSetting": {
    "Description": "General Setting"
  },
  "MenuSetting": {
    "Description": "Menu Setting",
    "Setting": {
      "AutoMenu": {
        "Description": "Register menu",
        "HelpText": "This is for auto create menu",
        "Value": {
          "Sample": {
            "Parameter": {
              "icon": "fa fa-birthday-cake",
              "Active": "Y"
            },
            "Sample": "Sample",
            "Sample Account": "SampleAccount"
          },
          "Multi Journal": {
            "Parameter": {
              "icon": "fas fa-book",
              "url": "MultiJournal"
            }
          },
          "Foreign Exchange": {
            "Parameter": {
              "icon": "fa fa-money-bill-alt",
              "url": "ForeignExchange"
            }
          }
        }
      }
    }
  }
}

A query for the first elements would be:

MenuSetting.Setting.AutoMenu.Value.[Sample.Sample, Sample."Sample Account", *.Parameter.url][]

And this would give the expected:

[
  "Sample",
  "SampleAccount",
  "MultiJournal",
  "ForeignExchange"
]

thank very much, but can I general syntax or wild card like * or [] in the mentioned YAML Syntax instead of only fit value above, since we could modify, add/remove any item other than the above item? Thanks very much :) — Houy Narun, Jan 18 '21 at 01:47
Well, without a clear definition of your YAML structure, this would be plain guessing. In the actual state of it I see several issue preventing you to do so: 1. From times to times you url is in `menu_name[0]`, other times it is under `menu_name[0].Parameter[0].url`. 2. You didn't stated how elements are nested into each other. How deep can they be nested, what really happens when nested? 3. The structure that seems to confuse list and dictionaries as stated already. — β.εηοιτ.βε, Jan 19 '21 at 12:38

How can I get every URL from dictionary in Python?

1 Answers1