Parse data from Json file using python script

Question

I created the json from a python script, and here is the code what I wrote to get the json data:

import requests
import json
import ConfigParser

url = "xxx"

payload = "items.find({ \"repo\": {\"$match\" : \"nswps-*\"}}).include(\"name\",\"repo\",\"path\")\r\n"
headers = {
    'Content-Type': "text/plain",
    'Authorization': "Basic xxxxxxxxx",
    'Accept': "*/*",
    'Cache-Control': "no-cache",
    'Host': "xxxxxx.com",
    'accept-encoding': "gzip, deflate",
    'content-length': "77",
    'Connection': "keep-alive",
    'cache-control': "no-cache"
    }

response = requests.request("POST", url, data=payload, headers=headers)

print(response.text)

the above code gives me the json file which is a huge file of multiple objects. Due to some limitation in artifactory I am unable to get the repos starting with nswps but rather the result is all the repository names. The json file has data like this:

"repo" : "npm-remote-cache",
  "path" : "zrender/-",
  "name" : "zrender-4.0.7.tgz"
},{
  "repo" : "npm-remote-cache",
  "path" : "ztree/-",
  "name" : "ztree-3.5.24.tgz"
},{
  "repo" : "nswps-docker-inprogress-local",
  "path" : "ace/core/latest",
  "name" : "manifest.json"
},{
  "repo" : "nswps-docker-inprogress-local",
  "path" : "ace/core/latest",
  "name" : "sha256__0a381222a179dbaef7d1f50914549a84e922162a772ca5346b5f6147d0e5aab4"
},{
.........

Now I need to create a python script which fetches out the objects in which only the object that has value of nswps , lets say from the above json I need data like this:

{
  "repo" : "nswps-docker-inprogress-local",
  "path" : "ace/core/latest",
  "name" : "manifest.json"
},{
  "repo" : "nswps-docker-inprogress-local",
  "path" : "ace/core/latest",
  "name" : "sha256__0a381222a179dbaef7d1f50914549a84e922162a772ca5346b5f6147d0e5aab4"
}

Possible duplicate of [Convert string to JSON using Python](https://stackoverflow.com/questions/4528099/convert-string-to-json-using-python) — Sayse, May 01 '19 at 11:00
`resonse.json` gives data converted to Python's list and you can use `for` loop to get only some of the items. — furas, May 01 '19 at 11:13
I think its not a duplicate of Convert string to JSON using Python. Mine issue is to print only value starting from nswps — user3403309, May 01 '19 at 11:14
Possible duplicate of [HTTP requests and JSON parsing in Python](https://stackoverflow.com/questions/6386308/http-requests-and-json-parsing-in-python) — Jared Smith, May 01 '19 at 11:48

score 1 · Answer 1 · answered May 03 '19 at 06:29

I could successfully do it with the help of @furas, once again thanks! The solution code is below:

response = requests.request("POST", url, data=payload, headers=headers)
 list = response.text
 data = response.json()['results']
 print("Line#24")
output_dict = [items for items in data if 'nswps' in items['repo']]

furas · Answer 2 · 2019-05-01T11:45:58.600

0

Using

data = response.json()

you should get JSON converted to Python's list

Let's say you have data

data = [
    {"repo" : "npm-remote-cache",
      "path" : "zrender/-",
      "name" : "zrender-4.0.7.tgz"
    },{
      "repo" : "npm-remote-cache",
      "path" : "ztree/-",
      "name" : "ztree-3.5.24.tgz"
    },{
      "repo" : "nswps-docker-inprogress-local",
      "path" : "ace/core/latest",
      "name" : "manifest.json"
    },{
      "repo" : "nswps-docker-inprogress-local",
      "path" : "ace/core/latest",
      "name" : "sha256__0a381222a179dbaef7d1f50914549a84e922162a772ca5346b5f6147d0e5aab4"
    }
]

then you can use for loop to check every item on the list and select only some of them

for item in data:
    if item['repo'].startswith('nswps'):
        print(item)

and as list comprehension

selected = [item for item in data if item['repo'].startswith('nswps')]

edited May 01 '19 at 11:45

answered May 01 '19 at 11:17

furas

134,197
12
106
148

Thanks SO much furas. It is good to know that. I am facing this error, "if item['repo'].startswith('nswps'): TypeError: string indices must be integers" – user3403309 May 01 '19 at 14:17
it means your JSON has more complex structure. It can be external list with internal list which has dictionares. My example shows only list with dictionares - without external list. But your data may have even more complex structure. At start use `print( item , type(item) )` to see what you have in item. – furas May 01 '19 at 17:06
Thanks furas for looking at it. Here is what my file has ( just a snippet) {u'range': {u'total': 123812, u'end_pos': 123812, u'start_pos': 0}, u'results': [{u'repo': u'jcenter-cache', u'path': u"'com/virtualightning'/stateframework/0.1.1", u'name': u'stateframework-0.1.1-sources.jar'}, {u'repo': u'jcenter-cache', u'path': u"'com/virtualightning'/stateframework/0.1.1", u'name': u'stateframework-0.1.1.aar'}, {u'repo': u'jcenter-cache', u'path': u'.', u'name': u'sdk-release'}, {u'repo': u'jcenter-cache', u'path': u'abbot/abbot/0.12.3', u'name': u'abbot-0.12.3.jar'} – user3403309 May 01 '19 at 17:42
did you use `response.json()` ? `string indices must be integers"` may means you used `response.text`. With snippet it can be `data = response.json()['results']` – furas May 01 '19 at 18:58
Thanks So much furas for your help, I could achieve the task. To conclude , my input was the json which I received from artifactory using rest API. And then I cut the size of the file by selecting only what I want. Here is the code: response = requests.request("POST", url, data=payload, headers=headers) list = response.text output_dict = [items for items in data if 'nswps' in items['repo']] Once again Thanks so much for helping me out! – user3403309 May 03 '19 at 06:25

Parse data from Json file using python script

2 Answers2