Parsing a nested json in python

Question

I am trying to parse the a small extract of a json to search and retrieve the items securityId, enabled, productType, expiry and lastTradingDate from _source.

{ "took": 3, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 36450, "max_score": 5.2289367, "hits": [ { "_index": "securitylisting", "_type": "securitylistingtype", "_id": "1238346721692487680", "_score": 5.2289367, "_source": { "id": 1238346721692487700, "assetType": "FUTURE", "exchangeCode": "CME", "securityCode": "NG", "securityId": "NGG2", "enabled": true, "createdTimestampUtc": "2022-03-09T02:11:19.898Z", "updatedTimestampUtc": "2022-04-14T23:56:51.077Z", "productType": "OUTRIGHT", "currencyCode": "USD", "description": "Henry Hub Natural Gas Futures", "divisor": "3", "tickSize": "1", "tickValue": "10.0", "expiry": "2032-02-28", "lastTradingDate": "2032-01-28", "isFractionalContract": "false", "displayFactor": "0.001", "exchangeId": "2224171651002540032", "aliases": [ { "securityIdSource": "TT", "securityCode": "NG", "securityId": "NGG32" }, { "securityIdSource": "EXCHANGE", "securityCode": "NG", "securityId": "NGG2" } ], "protobuf": "CICggLqEud+XERABGgNDTUUiAk5HKgROR0cyShcaC3Byb2R1Y3RUeXBlIghPVVRSSUdIVEoTGgxj\r\ndXJyZW5jeUNvZGUiA1VTREosGgtkZXNjcmlwdGlvbiIdSGVucnkgSHViIE5hdHVyYWwgR2FzIEZ1\r\ndHVyZXNKDBoHZGl2aXNvciIBM0oNGgh0aWNrU2l6ZSIBMUoRGgl0aWNrVmFsdWUiBDEwLjBKFBoG\r\nZXhwaXJ5IgoyMDMyLTAyLTI4Sh0aD2xhc3RUcmFkaW5nRGF0ZSIKMjAzMi0wMS0yOEodGhRpc0Zy\r\nYWN0aW9uYWxDb250cmFjdCIFZmFsc2VKFhoNZGlzcGxheUZhY3RvciIFMC4wMDFKIRoKZXhjaGFu\r\nZ2VJZCITMjIyNDE3MTY1MTAwMjU0MDAzMlIPIgJUVCoCTkcyBU5HRzMyUhQiCEVYQ0hBTkdFKgJO\r\nRzIETkdHMg==\r\n" } }, { "_index": "securitylisting", "_type": "securitylistingtype", "_id": "1230533889064177664", "_score": 5.2289367, "_source": { "id": 1230533889064177700, "assetType": "FUTURE", "exchangeCode": "CME", "securityCode": "NG", "securityId": "NGN34", "enabled": true, "createdTimestampUtc": "2022-02-15T12:45:55.444Z", "updatedTimestampUtc": "2022-04-17T05:48:48.645Z", "productType": "OUTRIGHT", "currencyCode": "USD", "description": "Henry Hub Natural Gas Futures", "divisor": "3", "tickSize": "1", "tickValue": "10.0", "expiry": "2034-07-28", "lastTradingDate": "2034-06-28", "isFractionalContract": "false", "displayFactor": "0.001", "exchangeId": "2224171651002540032", "aliases": [ { "securityIdSource": "TT", "securityCode": "NG", "securityId": "NGN34" }, { "securityIdSource": "EXCHANGE", "securityCode": "NG", "securityId": "NGN34" } ], "protobuf": "CICggO7Wge+JERABGgNDTUUiAk5HKgVOR04zNEoXGgtwcm9kdWN0VHlwZSIIT1VUUklHSFRKExoM\r\nY3VycmVuY3lDb2RlIgNVU0RKLBoLZGVzY3JpcHRpb24iHUhlbnJ5IEh1YiBOYXR1cmFsIEdhcyBG\r\ndXR1cmVzSgwaB2Rpdmlzb3IiATNKDRoIdGlja1NpemUiATFKERoJdGlja1ZhbHVlIgQxMC4wShQa\r\nBmV4cGlyeSIKMjAzNC0wNy0yOEodGg9sYXN0VHJhZGluZ0RhdGUiCjIwMzQtMDYtMjhKHRoUaXNG\r\ncmFjdGlvbmFsQ29udHJhY3QiBWZhbHNlShYaDWRpc3BsYXlGYWN0b3IiBTAuMDAxSiEaCmV4Y2hh\r\nbmdlSWQiEzIyMjQxNzE2NTEwMDI1NDAwMzJSDyICVFQqAk5HMgVOR04zNFIVIghFWENIQU5HRSoC\r\nTkcyBU5HTjM0\r\n" } }, { "_index": "securitylisting", "_type": "securitylistingtype", "_id": "1185544030478405632", "_score": 5.2289367, "_source": { "id": 1185544030478405600, "assetType": "FUTURE", "exchangeCode": "CME", "securityCode": "NG", "securityId": "NGH6", "enabled": true, "createdTimestampUtc": "2021-10-14T09:12:17.286Z", "updatedTimestampUtc": "2022-04-17T05:48:49.130Z", "productType": "OUTRIGHT", "currencyCode": "USD", "description": "Henry Hub Natural Gas Futures", "divisor": "3", "tickSize": "1", "tickValue": "10.0", "expiry": "2026-03-25", "lastTradingDate": "2026-02-25", "isFractionalContract": "false", "displayFactor": "0.001", "exchangeId": "2224171651002540032", "aliases": [ { "securityIdSource": "TT", "securityCode": "NG", "securityId": "NGH26" }, { "securityIdSource": "EXCHANGE", "securityCode": "NG", "securityId": "NGH6" } ], "protobuf": "CICggJK7wPm5EBABGgNDTUUiAk5HKgROR0g2ShcaC3Byb2R1Y3RUeXBlIghPVVRSSUdIVEoTGgxj\r\ndXJyZW5jeUNvZGUiA1VTREosGgtkZXNjcmlwdGlvbiIdSGVucnkgSHViIE5hdHVyYWwgR2FzIEZ1\r\ndHVyZXNKDBoHZGl2aXNvciIBM0oNGgh0aWNrU2l6ZSIBMUoRGgl0aWNrVmFsdWUiBDEwLjBKFBoG\r\nZXhwaXJ5IgoyMDI2LTAzLTI1Sh0aD2xhc3RUcmFkaW5nRGF0ZSIKMjAyNi0wMi0yNUodGhRpc0Zy\r\nYWN0aW9uYWxDb250cmFjdCIFZmFsc2VKFhoNZGlzcGxheUZhY3RvciIFMC4wMDFKIRoKZXhjaGFu\r\nZ2VJZCITMjIyNDE3MTY1MTAwMjU0MDAzMlIPIgJUVCoCTkcyBU5HSDI2UhQiCEVYQ0hBTkdFKgJO\r\nRzIETkdINg==\r\n" } } ] } }

I am unable to work out how to parse it correctly due to the nested sections. Any guidance is appreciated. I have checked other replies on similar posts but not been able to work it out using json_data['output'][index']

import json

product = '{"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":36450,"max_score":5.2289367,"hits":[{"_index":"securitylisting","_type":"securitylistingtype","_id":"1238346721692487680","_score":5.2289367,"_source":{"id":1238346721692487680,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGG2","enabled":true,"createdTimestampUtc":"2022-03-09T02:11:19.898Z","updatedTimestampUtc":"2022-04-14T23:56:51.077Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2032-02-28","lastTradingDate":"2032-01-28","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGG32"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGG2"}],"protobuf":"CICggLqEud+XERABGgNDTUUiAk5HKgROR0cyShcaC3Byb2R1Y3RUeXBlIghPVVRSSUdIVEoTGgxj\r\ndXJyZW5jeUNvZGUiA1VTREosGgtkZXNjcmlwdGlvbiIdSGVucnkgSHViIE5hdHVyYWwgR2FzIEZ1\r\ndHVyZXNKDBoHZGl2aXNvciIBM0oNGgh0aWNrU2l6ZSIBMUoRGgl0aWNrVmFsdWUiBDEwLjBKFBoG\r\nZXhwaXJ5IgoyMDMyLTAyLTI4Sh0aD2xhc3RUcmFkaW5nRGF0ZSIKMjAzMi0wMS0yOEodGhRpc0Zy\r\nYWN0aW9uYWxDb250cmFjdCIFZmFsc2VKFhoNZGlzcGxheUZhY3RvciIFMC4wMDFKIRoKZXhjaGFu\r\nZ2VJZCITMjIyNDE3MTY1MTAwMjU0MDAzMlIPIgJUVCoCTkcyBU5HRzMyUhQiCEVYQ0hBTkdFKgJO\r\nRzIETkdHMg==\r\n"}},{"_index":"securitylisting","_type":"securitylistingtype","_id":"1230533889064177664","_score":5.2289367,"_source":{"id":1230533889064177664,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGN34","enabled":true,"createdTimestampUtc":"2022-02-15T12:45:55.444Z","updatedTimestampUtc":"2022-04-17T05:48:48.645Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2034-07-28","lastTradingDate":"2034-06-28","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGN34"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGN34"}],"protobuf":"CICggO7Wge+JERABGgNDTUUiAk5HKgVOR04zNEoXGgtwcm9kdWN0VHlwZSIIT1VUUklHSFRKExoM\r\nY3VycmVuY3lDb2RlIgNVU0RKLBoLZGVzY3JpcHRpb24iHUhlbnJ5IEh1YiBOYXR1cmFsIEdhcyBG\r\ndXR1cmVzSgwaB2Rpdmlzb3IiATNKDRoIdGlja1NpemUiATFKERoJdGlja1ZhbHVlIgQxMC4wShQa\r\nBmV4cGlyeSIKMjAzNC0wNy0yOEodGg9sYXN0VHJhZGluZ0RhdGUiCjIwMzQtMDYtMjhKHRoUaXNG\r\ncmFjdGlvbmFsQ29udHJhY3QiBWZhbHNlShYaDWRpc3BsYXlGYWN0b3IiBTAuMDAxSiEaCmV4Y2hh\r\nbmdlSWQiEzIyMjQxNzE2NTEwMDI1NDAwMzJSDyICVFQqAk5HMgVOR04zNFIVIghFWENIQU5HRSoC\r\nTkcyBU5HTjM0\r\n"}},{"_index":"securitylisting","_type":"securitylistingtype","_id":"1185544030478405632","_score":5.2289367,"_source":{"id":1185544030478405632,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGH6","enabled":true,"createdTimestampUtc":"2021-10-14T09:12:17.286Z","updatedTimestampUtc":"2022-04-17T05:48:49.130Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2026-03-25","lastTradingDate":"2026-02-25","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGH26"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGH6"}],"protobuf":"CICggJK7wPm5EBABGgNDTUUiAk5HKgROR0g2ShcaC3Byb2R1Y3RUeXBlIghPVVRSSUdIVEoTGgxj\r\ndXJyZW5jeUNvZGUiA1VTREosGgtkZXNjcmlwdGlvbiIdSGVucnkgSHViIE5hdHVyYWwgR2FzIEZ1\r\ndHVyZXNKDBoHZGl2aXNvciIBM0oNGgh0aWNrU2l6ZSIBMUoRGgl0aWNrVmFsdWUiBDEwLjBKFBoG\r\nZXhwaXJ5IgoyMDI2LTAzLTI1Sh0aD2xhc3RUcmFkaW5nRGF0ZSIKMjAyNi0wMi0yNUodGhRpc0Zy\r\nYWN0aW9uYWxDb250cmFjdCIFZmFsc2VKFhoNZGlzcGxheUZhY3RvciIFMC4wMDFKIRoKZXhjaGFu\r\nZ2VJZCITMjIyNDE3MTY1MTAwMjU0MDAzMlIPIgJUVCoCTkcyBU5HSDI2UhQiCEVYQ0hBTkdFKgJO\r\nRzIETkdINg==\r\n"}},{"_index":"securitylisting","_type":"securitylistingtype","_id":"1185544029807316993","_score":5.2289367,"_source":{"id":1185544029807316993,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGV7","enabled":true,"createdTimestampUtc":"2021-10-14T09:12:17.126Z","updatedTimestampUtc":"2022-04-17T05:48:44.373Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2027-10-28","lastTradingDate":"2027-09-28","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGV27"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGV7"}],"protobuf":"CIGggNK4wPm5EBABGgNDTUUiAk5HKgROR1Y3ShcaC3Byb2R1Y3RUeXBlIghPVVRSSUdIVEoTGgxj\r\ndXJyZW5jeUNvZGUiA1VTREosGgtkZXNjcmlwdGlvbiIdSGVucnkgSHViIE5hdHVyYWwgR2FzIEZ1\r\ndHVyZXNKDBoHZGl2aXNvciIBM0oNGgh0aWNrU2l6ZSIBMUoRGgl0aWNrVmFsdWUiBDEwLjBKFBoG\r\nZXhwaXJ5IgoyMDI3LTEwLTI4Sh0aD2xhc3RUcmFkaW5nRGF0ZSIKMjAyNy0wOS0yOEodGhRpc0Zy\r\nYWN0aW9uYWxDb250cmFjdCIFZmFsc2VKFhoNZGlzcGxheUZhY3RvciIFMC4wMDFKIRoKZXhjaGFu\r\nZ2VJZCITMjIyNDE3MTY1MTAwMjU0MDAzMlIPIgJUVCoCTkcyBU5HVjI3UhQiCEVYQ0hBTkdFKgJO\r\nRzIETkdWNw==\r\n"}},{"_index":"securitylisting","_type":"securitylistingtype","_id":"1153026101493108737","_score":5.2289367,"_source":{"id":1153026101493108737,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGK4","enabled":true,"createdTimestampUtc":"2021-07-16T15:37:38.862Z","updatedTimestampUtc":"2022-04-17T05:48:44.852Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2024-05-26","lastTradingDate":"2024-04-26","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGK4"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGK4"}],"protobuf":"CIGggOKV5JeAEBABGgNDTUUiAk5HKgROR0s0ShcaC3Byb2R1Y3RUeXBlIghPVVRSSUdIVEoTGgxj\r\ndXJyZW5jeUNvZGUiA1VTREosGgtkZXNjcmlwdGlvbiIdSGVucnkgSHViIE5hdHVyYWwgR2FzIEZ1\r\ndHVyZXNKDBoHZGl2aXNvciIBM0oNGgh0aWNrU2l6ZSIBMUoRGgl0aWNrVmFsdWUiBDEwLjBKFBoG\r\nZXhwaXJ5IgoyMDI0LTA1LTI2Sh0aD2xhc3RUcmFkaW5nRGF0ZSIKMjAyNC0wNC0yNkodGhRpc0Zy\r\nYWN0aW9uYWxDb250cmFjdCIFZmFsc2VKFhoNZGlzcGxheUZhY3RvciIFMC4wMDFKIRoKZXhjaGFu\r\nZ2VJZCITMjIyNDE3MTY1MTAwMjU0MDAzMlIOIgJUVCoCTkcyBE5HSzRSFCIIRVhDSEFOR0UqAk5H\r\nMgROR0s0\r\n"}},{"_index":"securitylisting","_type":"securitylistingtype","_id":"1185544029245280256","_score":5.2289367,"_source":{"id":1185544029245280256,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGH1","enabled":true,"createdTimestampUtc":"2021-10-14T09:12:16.992Z","updatedTimestampUtc":"2022-04-17T05:48:39.077Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2031-03-26","lastTradingDate":"2031-02-26","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGH31"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGH1"}],"protobuf":"CICggMa2wPm5EBABGgNDTUUiAk5HKgROR0gxShcaC3Byb2R1Y3RUeXBlIghPVVRSSUdIVEoTGgxj\r\ndXJyZW5jeUNvZGUiA1VTREosGgtkZXNjcmlwdGlvbiIdSGVucnkgSHViIE5hdHVyYWwgR2FzIEZ1\r\ndHVyZXNKDBoHZGl2aXNvciIBM0oNGgh0aWNrU2l6ZSIBMUoRGgl0aWNrVmFsdWUiBDEwLjBKFBoG\r\nZXhwaXJ5IgoyMDMxLTAzLTI2Sh0aD2xhc3RUcmFkaW5nRGF0ZSIKMjAzMS0wMi0yNkodGhRpc0Zy\r\nYWN0aW9uYWxDb250cmFjdCIFZmFsc2VKFhoNZGlzcGxheUZhY3RvciIFMC4wMDFKIRoKZXhjaGFu\r\nZ2VJZCITMjIyNDE3MTY1MTAwMjU0MDAzMlIPIgJUVCoCTkcyBU5HSDMxUhQiCEVYQ0hBTkdFKgJO\r\nRzIETkdIMQ==\r\n"}},{"_index":"securitylisting","_type":"securitylistingtype","_id":"1185544029245280257","_score":5.2289367,"_source":{"id":1185544029245280257,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGU0","enabled":true,"createdTimestampUtc":"2021-10-14T09:12:16.992Z","updatedTimestampUtc":"2022-04-17T05:48:39.564Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2030-09-28","lastTradingDate":"2030-08-28","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGU30"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGU0"}],"protobuf":"CIGggMa2wPm5EBABGgNDTUUiAk5HKgROR1UwShcaC3Byb2R1Y3RUeXBlIghPVVRSSUdIVEoTGgxj\r\ndXJyZW5jeUNvZGUiA1VTREosGgtkZXNjcmlwdGlvbiIdSGVucnkgSHViIE5hdHVyYWwgR2FzIEZ1\r\ndHVyZXNKDBoHZGl2aXNvciIBM0oNGgh0aWNrU2l6ZSIBMUoRGgl0aWNrVmFsdWUiBDEwLjBKFBoG\r\nZXhwaXJ5IgoyMDMwLTA5LTI4Sh0aD2xhc3RUcmFkaW5nRGF0ZSIKMjAzMC0wOC0yOEodGhRpc0Zy\r\nYWN0aW9uYWxDb250cmFjdCIFZmFsc2VKFhoNZGlzcGxheUZhY3RvciIFMC4wMDFKIRoKZXhjaGFu\r\nZ2VJZCITMjIyNDE3MTY1MTAwMjU0MDAzMlIPIgJUVCoCTkcyBU5HVTMwUhQiCEVYQ0hBTkdFKgJO\r\nRzIETkdVMA==\r\n"}},{"_index":"securitylisting","_type":"securitylistingtype","_id":"1230533848052273153","_score":5.2289367,"_source":{"id":1230533848052273153,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGF33","enabled":true,"createdTimestampUtc":"2022-02-15T12:45:45.666Z","updatedTimestampUtc":"2022-04-17T05:48:30.908Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2033-01-29","lastTradingDate":"2032-12-29","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGF33"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGF33"}],"protobuf":"CIGggIq+gO+JERABGgNDTUUiAk5HKgVOR0YzM0oXGgtwcm9kdWN0VHlwZSIIT1VUUklHSFRKExoM\r\nY3VycmVuY3lDb2RlIgNVU0RKLBoLZGVzY3JpcHRpb24iHUhlbnJ5IEh1YiBOYXR1cmFsIEdhcyBG\r\ndXR1cmVzSgwaB2Rpdmlzb3IiATNKDRoIdGlja1NpemUiATFKERoJdGlja1ZhbHVlIgQxMC4wShQa\r\nBmV4cGlyeSIKMjAzMy0wMS0yOUodGg9sYXN0VHJhZGluZ0RhdGUiCjIwMzItMTItMjlKHRoUaXNG\r\ncmFjdGlvbmFsQ29udHJhY3QiBWZhbHNlShYaDWRpc3BsYXlGYWN0b3IiBTAuMDAxSiEaCmV4Y2hh\r\nbmdlSWQiEzIyMjQxNzE2NTEwMDI1NDAwMzJSDyICVFQqAk5HMgVOR0YzM1IVIghFWENIQU5HRSoC\r\nTkcyBU5HRjMz\r\n"}},{"_index":"securitylisting","_type":"securitylistingtype","_id":"1230533848043884545","_score":5.2289367,"_source":{"id":1230533848043884545,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGZ33","enabled":true,"createdTimestampUtc":"2022-02-15T12:45:45.664Z","updatedTimestampUtc":"2022-04-17T05:48:29.489Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2033-12-28","lastTradingDate":"2033-11-28","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGZ33"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGZ33"}],"protobuf":"CIGggIa+gO+JERABGgNDTUUiAk5HKgVOR1ozM0oXGgtwcm9kdWN0VHlwZSIIT1VUUklHSFRKExoM\r\nY3VycmVuY3lDb2RlIgNVU0RKLBoLZGVzY3JpcHRpb24iHUhlbnJ5IEh1YiBOYXR1cmFsIEdhcyBG\r\ndXR1cmVzSgwaB2Rpdmlzb3IiATNKDRoIdGlja1NpemUiATFKERoJdGlja1ZhbHVlIgQxMC4wShQa\r\nBmV4cGlyeSIKMjAzMy0xMi0yOEodGg9sYXN0VHJhZGluZ0RhdGUiCjIwMzMtMTEtMjhKHRoUaXNG\r\ncmFjdGlvbmFsQ29udHJhY3QiBWZhbHNlShYaDWRpc3BsYXlGYWN0b3IiBTAuMDAxSiEaCmV4Y2hh\r\nbmdlSWQiEzIyMjQxNzE2NTEwMDI1NDAwMzJSDyICVFQqAk5HMgVOR1ozM1IVIghFWENIQU5HRSoC\r\nTkcyBU5HWjMz\r\n"}},{"_index":"securitylisting","_type":"securitylistingtype","_id":"1230533848048078848","_score":5.2289367,"_source":{"id":1230533848048078848,"assetType":"FUTURE","exchangeCode":"CME","securityCode":"NG","securityId":"NGN33","enabled":true,"createdTimestampUtc":"2022-02-15T12:45:45.665Z","updatedTimestampUtc":"2022-04-17T05:48:29.958Z","productType":"OUTRIGHT","currencyCode":"USD","description":"Henry Hub Natural Gas Futures","divisor":"3","tickSize":"1","tickValue":"10.0","expiry":"2033-07-28","lastTradingDate":"2033-06-28","isFractionalContract":"false","displayFactor":"0.001","exchangeId":"2224171651002540032","aliases":[{"securityIdSource":"TT","securityCode":"NG","securityId":"NGN33"},{"securityIdSource":"EXCHANGE","securityCode":"NG","securityId":"NGN33"}],"protobuf":"CICggIi+gO+JERABGgNDTUUiAk5HKgVOR04zM0oXGgtwcm9kdWN0VHlwZSIIT1VUUklHSFRKExoM\r\nY3VycmVuY3lDb2RlIgNVU0RKLBoLZGVzY3JpcHRpb24iHUhlbnJ5IEh1YiBOYXR1cmFsIEdhcyBG\r\ndXR1cmVzSgwaB2Rpdmlzb3IiATNKDRoIdGlja1NpemUiATFKERoJdGlja1ZhbHVlIgQxMC4wShQa\r\nBmV4cGlyeSIKMjAzMy0wNy0yOEodGg9sYXN0VHJhZGluZ0RhdGUiCjIwMzMtMDYtMjhKHRoUaXNG\r\ncmFjdGlvbmFsQ29udHJhY3QiBWZhbHNlShYaDWRpc3BsYXlGYWN0b3IiBTAuMDAxSiEaCmV4Y2hh\r\nbmdlSWQiEzIyMjQxNzE2NTEwMDI1NDAwMzJSDyICVFQqAk5HMgVOR04zM1IVIghFWENIQU5HRSoC\r\nTkcyBU5HTjMz\r\n"}}]}} '

# Convert string to python dict

product_dict = json.loads(str(product)

product_dict['hits']['hits'][0]['_source']['securityId']

Traceback (most recent call last):
  File ".prog.py", line 9, in <module>
  File "usr/lib/python3.8/json/_init_.py", line 357, in loads
    return _default_decoder.decode(s)
  File "usr/lib/python3.8/json/decode.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "usr/lib/python3.8/json/decode.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 1 column 1000 (char999)

What is the problem exactly? Do you get any error message you can post? There are various errors I can see immediately now I've formatted your code, but I'm unsure whether this is your real code, or whether you had some issues with pasting it. — quamrana, Apr 20 '22 at 13:27
i am unable to work out how to use the json output from my curl command and create a way to search the data using python. All i need to pull out of the json is securityId', 'enabled', 'productType', 'expiry' and 'lastTradingDate — Aphunkie Monkey, Apr 20 '22 at 13:32
The json you have posted is polluted with rogue `"\r\n"` characters inside protobuf strings (I think). You need to get pure json to be able to decode with the `json` library. — quamrana, Apr 20 '22 at 13:35
Please update the text of the question with your error traceback. — quamrana, Apr 20 '22 at 13:37
maybe like `product_dict['hits']['hits'][0]['_source']['securityId']`? FYI your data is severely nested, not sure if that's expected or not. — rv.kvetch, Apr 20 '22 at 13:38
"Parse json" refers to what `json.loads` does. Once you have the Python object, the question is now one of *indexing*. — chepner, Apr 20 '22 at 13:38
Did you notice that json.loads failed with an exception? It certainly will do given the data as shown in your question. — DarkKnight, Apr 20 '22 at 13:44
I think the OP realises that now. I'm hoping they are sorting something out right now. — quamrana, Apr 20 '22 at 13:46
FYI your data is severely nested, not sure if that's expected or not - Yes this is expected — Aphunkie Monkey, Apr 20 '22 at 13:50
btw nesting is not really a problem for the `json` data format. — quamrana, Apr 20 '22 at 13:51

score 0 · Accepted Answer · answered Apr 20 '22 at 13:52

0

From the error message you need to sanitize your json as I said in my comment.

You may get some help from the source of the json, but if not, it may be possible to just replace all the control characters with empty strings and get some success, but leave yourself open to future problems with the source.

So for now you can use:

product = product.replace('\n', '').replace('\r', '')

and then it becomes an indexing problem.

answered Apr 20 '22 at 13:52

quamrana

37,849
12
53
71

easier to make it a raw string, just adds an `r`. i.e. `product = r'...'` – rv.kvetch Apr 20 '22 at 14:01
tried both product = product.replace(r'\r', '').replace(r'\n', '') and product = product.replace('\r', '').replace('\n', '') based on previous question https://stackoverflow.com/questions/16566268/remove-all-line-breaks-from-a-long-string-of-text to no avail – Aphunkie Monkey Apr 20 '22 at 14:13
is there a way to remove the protobuf altogether? i dont need it – Aphunkie Monkey Apr 20 '22 at 14:15
or ignoring protobuf? – Aphunkie Monkey Apr 20 '22 at 14:29
What I'm saying in my answer is that when I sanitize the `product` variable, `json` can convert it to `product_dict`. At that point your problem becomes one of indexing a hierarchical structure. (I get: `KeyError: 'securityId'` which is a different question altogether). – quamrana Apr 20 '22 at 14:39

Parsing a nested json in python

1 Answers1