0

I need to pull data in from elasticsearch, do some cleaning/munging and export as table/rds.

To do this I have a long list of variable names required to pull from elasticsearch. This list of variables is required for the pull, but the issue is that not all fields may be represented within a given pull, meaning that I need to add the fields after the fact. I can do this using a schema (in nested json format) of the same list of variable names.

To try and [slightly] future proof this work I would ideally like to only maintain the list/schema in one place, and convert from list to schema (or vice-versa).

Is there a way to do this in python? Please see example below of input and desired output.

Small part of schema:

{
  "_source": {
    "filters": {"group": {"filter_value": 0}},
    "user": {
      "email":  "",
      "uid":  ""
    },
    "status": {
      "date": "", 
      "active": True
    }
  }
}

Desired string list output:

[
"_source.filters.group.filter_value",
"_source.user.email",
"_source.user.uid",
"_source.status.date",
"_source.status.active"
]

I believe that schema -> list might be an easier transformation than list -> schema, though am happy for it to be the other way round if that is simpler (though need to ensure the schema variables have the correct type, i.e. str, bool, float).

I have explored the following answers which come close, but I am struggling to understand since none appear to be in python:

FitzKaos
  • 381
  • 3
  • 20

1 Answers1

1

Where d is your json as a dictionary,

def full_search(d):
    arr = []
    def dfs(d, curr):
        if not type(d) == dict or curr[-1] not in d or type(d[curr[-1]]) != dict:
            arr.append(curr)
            return
        for key in d[curr[-1]].keys():
            dfs(d[curr[-1]], curr + [key])
    for key in d.keys():
        dfs(d, [key])
    return ['.'.join(x) for x in arr]

If d is in json form, use

import json
res = full_search(json.loads(d))
Rich Caputo
  • 26
  • 1