Separating the data from a string filled with multiple data types

Question

I want to turn this string into a list of the dictionaries within it:

”[{‘id’: ‘x’, ‘name’: ‘y’}, {‘id’: ‘p’, ‘name’: ‘q’}]”

I have multiple columns in a pandas dataframe composed of similar strings to this. Some are only ”[]” or ”[{‘id’: ‘x’, ‘name’: ‘y’}]”, and others have many stringified dictionaries within them.

I have tried

import json
z = ”[{‘id’: ‘x’, ‘name’: ‘y’}, {‘id’: ‘p’, ‘name’: ‘q’}]”
list(json.loads(z[1:-1]))

And this works fine for when there is only one stringified dictionary (”[{‘id’: ‘x’, ‘name’: ‘y’}]”) but because the dictionaries have shared keys, they cannot simply be jsonified.

Once this is done, I’ll retrieve the value from name of each and create a list of those.

ANSWERED Thanks @Tenacious B

To turn the string into the list of the dicts:

import json
z = ”[{‘id’: ‘x’, ‘name’: ‘y’}, {‘id’: ‘p’, ‘name’: ‘q’}]”
data = json.loads(z.replace("'", ""))]) # this will be a list of dicts

Output:

[{'id': 'x', 'name': 'y'}, {'id': 'p', 'name': 'q'}]

And to apply to a pandas dataframe having a column with entries as the above, retrieving only the value from the desired key:

import json
df['col'] = df['col'].apply(lambda x: [i['name'] for i in json.loads(x.replace("'", ""))])

your json is invalid in the example, starts with a list at the root node but ends with a curly bracket — bherbruck, May 29 '20 at 14:20
@TenaciousB that looks like a typo. It ends with a `]` in the first example. — timgeb, May 29 '20 at 14:21

score 1 · Answer 1 · answered May 29 '20 at 14:20

1

After I searched up how to convert a string list to list object I came across this: How to convert string representation of list to a list?

using ast.literal_eval can solve you problem.

answered May 29 '20 at 14:20

tygzy

698
1
6
23

Also an excellent solution! I chose @tenacious-b solution based on its extension of my proposed method. I will keep this method in mind in the future, it's very slick – MaxPi May 29 '20 at 14:52

score 1 · Accepted Answer · answered May 29 '20 at 14:21

1

You can have the same keys in different dicts/json objects, also to loads() you need to replace the ' with ":

import json
z = "[{'id': 'x', 'name': 'y'}, {'id': 'p', 'name': 'q'}]"
data = json.loads(z.replace("'", '"')) # this will be a list of dicts
print(data)

Output:

[{'id': 'x', 'name': 'y'}, {'id': 'p', 'name': 'q'}]

answered May 29 '20 at 14:21

bherbruck

2,167
1
6
17

1

Thanks! The issue with the shared keys was related to combining all of the dicts into a single dict without an index identifying the different dicts; I DO want exactly that, a list with dicts that have the same keys. Excellent! Thank you very much! – MaxPi May 29 '20 at 14:36

Separating the data from a string filled with multiple data types

2 Answers2