Flatten/Denormalize Dict/Json in Python

Question

Hi i am trying to denormalize/flatten a JSON to dictionary, in the following i have a generic way of flattening the JSON, the function below somehow dose not take care of nested JSON. The Goal is to flatten the nested dict/json to a CSV compatible objects.

I've used the following code:

def flatten(s):
    for i in s:
        if not isinstance(s[i], dict):
            yield (i, s[i])
        else:
            for b in flatten(s[i]):
                yield b


new_data = dict(list(flatten(mydict)))

print(new_data)

With this input dictionary

mydict = "{
  'G_TRANSACTIONS': {
    'INVOICE_NUMBER': '31002',
    'TRANSACTION_CLASS': 'Invoice',
    'LIST_G_LINES': {
      'G_LINES': [
        {
          'LN_LINE_NUMBER': '1',
          'LN_LINE_TYPE': 'Line',
          'LN_DESCRIPTION': 'PKG980N-MAIN STREET 16C POCKET PLUGS',
          'LN_UNIT_OF_MEASURE': 'EA',
          'LN_QUANTITY': '3',
          'LN_NET_SELLING_PRICE': '200',
          'LN_NET_EXTENDED_AMOUNT': '600',
          'LIST_G_LINES_ITEM_NUMBER': {
            'G_LINES_ITEM_NUMBER': {
              'ITEM_NUMBER': '136864001'
            }
          }
        },
        {
          'LN_LINE_NUMBER': '1',
          'LN_LINE_TYPE': 'Other Line',
          'LN_DESCRIPTION': 'Johar Town Lahore',
          'LN_UNIT_OF_MEASURE': 'EA',
          'LN_QUANTITY': '3',
          'LN_NET_SELLING_PRICE': '200',
          'LN_NET_EXTENDED_AMOUNT': '999',
          'LIST_G_LINES_ITEM_NUMBER': {
            'G_LINES_ITEM_NUMBER': {
              'ITEM_NUMBER': '99999999'
            }
          }
        }
      ]
    },
    'TR_LN_AMOUNT': '600'
  }
}"

they output i get is:

{
  'INVOICE_NUMBER': '31002',
  'TRANSACTION_CLASS': 'Invoice',
  'G_LINES': [
    {
      'LN_LINE_NUMBER': '1',
      'LN_LINE_TYPE': 'Line',
      'LN_DESCRIPTION': 'PKG980N-MAIN STREET 16C POCKET PLUGS',
      'LN_UNIT_OF_MEASURE': 'EA',
      'LN_QUANTITY': '3',
      'LN_NET_SELLING_PRICE': '200',
      'LN_NET_EXTENDED_AMOUNT': '600',
      'LIST_G_LINES_ITEM_NUMBER': {
        'G_LINES_ITEM_NUMBER': {
          'ITEM_NUMBER': '136864001'
        }
      }
    },
    {
      'LN_LINE_NUMBER': '1',
      'LN_LINE_TYPE': 'Other Line',
      'LN_DESCRIPTION': 'Johar Town Lahore',
      'LN_UNIT_OF_MEASURE': 'EA',
      'LN_QUANTITY': '3',
      'LN_NET_SELLING_PRICE': '200',
      'LN_NET_EXTENDED_AMOUNT': '999',
      'LIST_G_LINES_ITEM_NUMBER': {
        'G_LINES_ITEM_NUMBER': {
          'ITEM_NUMBER': '99999999'
        }
      }
    }
  ],
  'TR_LN_AMOUNT': '600'
}

Desired Output

{
  'INVOICE_NUMBER': '31002',
  'TRANSACTION_CLASS': 'Invoice',
  'LN_LINE_NUMBER': '1',
  'LN_LINE_TYPE': 'Line',
  'LN_DESCRIPTION': 'PKG980N-MAIN STREET 16C POCKET PLUGS',
  'LN_UNIT_OF_MEASURE': 'EA',
  'LN_QUANTITY': '3',
  'LN_NET_SELLING_PRICE': '200',
  'LN_NET_EXTENDED_AMOUNT': '600',
  'ITEM_NUMBER': '136864001',
  'TR_LN_AMOUNT': '600'
}
,
{
'INVOICE_NUMBER': '31002',
'TRANSACTION_CLASS': 'Invoice',
'LN_LINE_NUMBER': '1',
'LN_LINE_TYPE': 'Other Line',
'LN_DESCRIPTION': 'Johar Town Lahore',
'LN_UNIT_OF_MEASURE': 'EA',
'LN_QUANTITY': '3',
'LN_NET_SELLING_PRICE': '200',
'LN_NET_EXTENDED_AMOUNT': '999',
'ITEM_NUMBER': '99999999',
'TR_LN_AMOUNT': '600'
}

Can you show us the code you used to generate the first output? — C.Nivs, Aug 22 '19 at 14:04
@C.Nivs: if you look at the question its there but i guess due to the size of the JSON obj its hidden, i will post in comment, too `def flatten(s): for i in s: if not isinstance(s[i], dict): yield (i, s[i]) else: for b in flatten(s[i]): yield b` — noobie-php, Aug 22 '19 at 14:10
@noobie-php There must be a `json.loads` before calling `flatten(mydict))` right?. Bcz `my_dict` is a JSON string? — Abdul Niyas P M, Aug 22 '19 at 14:12
@AbdulNiyasPM: yes, code here was directed to the problem at hand. — noobie-php, Aug 22 '19 at 14:13

score 1 · Answer 1 · answered Aug 22 '19 at 16:25

You can try this script, it will flatten the mydict to correct form:

mydict = {
  'G_TRANSACTIONS': {
    'INVOICE_NUMBER': '31002',
    'TRANSACTION_CLASS': 'Invoice',
    'LIST_G_LINES': {
      'G_LINES': [
        {
          'LN_LINE_NUMBER': '1',
          'LN_LINE_TYPE': 'Line',
          'LN_DESCRIPTION': 'PKG980N-MAIN STREET 16C POCKET PLUGS',
          'LN_UNIT_OF_MEASURE': 'EA',
          'LN_QUANTITY': '3',
          'LN_NET_SELLING_PRICE': '200',
          'LN_NET_EXTENDED_AMOUNT': '600',
          'LIST_G_LINES_ITEM_NUMBER': {
            'G_LINES_ITEM_NUMBER': {
              'ITEM_NUMBER': '136864001'
            }
          }
        },
        {
          'LN_LINE_NUMBER': '1',
          'LN_LINE_TYPE': 'Other Line',
          'LN_DESCRIPTION': 'Johar Town Lahore',
          'LN_UNIT_OF_MEASURE': 'EA',
          'LN_QUANTITY': '3',
          'LN_NET_SELLING_PRICE': '200',
          'LN_NET_EXTENDED_AMOUNT': '999',
          'LIST_G_LINES_ITEM_NUMBER': {
            'G_LINES_ITEM_NUMBER': {
              'ITEM_NUMBER': '99999999'
            }
          }
        }
      ]
    },
    'TR_LN_AMOUNT': '600'
  }
}


def flatten(d, depth=0):
    rv = [({}, depth)]
    if isinstance(d, dict):
        for k, v in d.items():
            if not isinstance(v, dict) and not isinstance(v, list):
                for i in rv:
                    i[0][k] = v
            else:
                for (vv, _depth) in flatten(v,depth+1):
                    rv.append((rv[-1][0].copy(), _depth))
                    for kkk, vvv in vv.items():
                        rv[-1][0][kkk] = vvv
    elif isinstance(d, list):
        for v in d:
            rv.append((rv[-1][0].copy(), depth+1))
            for (vv, _) in flatten(v,depth+1):
                for kkk, vvv in vv.items():
                    rv[-1][0][kkk] = vvv
    for i, _depth in rv:
        yield i, _depth

from itertools import groupby

out = []
for v, g in groupby(sorted(flatten(mydict), key=lambda k: -k[1]), lambda k: k[1]):
    out.extend(i[0] for i in g)
    break

from pprint import pprint
for d in out:
    pprint(d)
    print('-' * 80)

Prints:

{'INVOICE_NUMBER': '31002',
 'ITEM_NUMBER': '136864001',
 'LN_DESCRIPTION': 'PKG980N-MAIN STREET 16C POCKET PLUGS',
 'LN_LINE_NUMBER': '1',
 'LN_LINE_TYPE': 'Line',
 'LN_NET_EXTENDED_AMOUNT': '600',
 'LN_NET_SELLING_PRICE': '200',
 'LN_QUANTITY': '3',
 'LN_UNIT_OF_MEASURE': 'EA',
 'TRANSACTION_CLASS': 'Invoice',
 'TR_LN_AMOUNT': '600'}
--------------------------------------------------------------------------------
{'INVOICE_NUMBER': '31002',
 'ITEM_NUMBER': '99999999',
 'LN_DESCRIPTION': 'Johar Town Lahore',
 'LN_LINE_NUMBER': '1',
 'LN_LINE_TYPE': 'Other Line',
 'LN_NET_EXTENDED_AMOUNT': '999',
 'LN_NET_SELLING_PRICE': '200',
 'LN_QUANTITY': '3',
 'LN_UNIT_OF_MEASURE': 'EA',
 'TRANSACTION_CLASS': 'Invoice',
 'TR_LN_AMOUNT': '600'}
--------------------------------------------------------------------------------

Flatten/Denormalize Dict/Json in Python

1 Answers1