Convert string list (with spaces instead of commas) into a list of lists

Question

I have the following string:

'[[ 0.03694573  0.01084746  0.01306414  0.00499198  0.00307188]\r\n [ 0.03780498  0.02610241  0.00967986  0.02228694 -0.01153102]\r\n [ 0.03837708  0.02111896  0.01370878 -0.00136839  0.01906253]\r\n [ 0.03490672 -0.02791057  0.08824896 -0.01991765  0.01964361]\r\n [ 0.0811892   0.28361901  0.21539196 -0.00259571  0.28737829]\r\n [ 0.20022041  0.16184418  0.25939959  0.00546446  0.36415219]\r\n [ 0.22920615  0.27439767  0.36991198  0.00624375  0.39911759]]'

As you can see, it comprises of a list of lists. However, each element within the lists are separated by tabs and not commas, and therefore ast.literal_eval() does not work. Does anyone have any ideas how best to convert this into a list of lists?

BJR

Why do you have it in that format? If you're reading it from a file, can you not just store the numbers in a 2D grid — sshashank124, Jan 14 '20 at 10:14
The numbers are the raw Permutation Importance indices stored from SciKit Learn. I will try and alter how they are saved, but for now I was just interested how to solve this problem. — BillyJo_rambler, Jan 14 '20 at 10:21

score 2 · Accepted Answer · answered Jan 14 '20 at 10:17

2

Here's one way using re.sub:

import re
literal_eval(','.join(re.sub(r'(?<=\d)(\s+)(?=-?\d)', ',', s).splitlines()))

[[0.03694573, 0.01084746, 0.01306414, 0.00499198, 0.00307188],
 [0.03780498, 0.02610241, 0.00967986, 0.02228694, -0.01153102],
 [0.03837708, 0.02111896, 0.01370878, -0.00136839, 0.01906253],
 [0.03490672, -0.02791057, 0.08824896, -0.01991765, 0.01964361],
 [0.0811892, 0.28361901, 0.21539196, -0.00259571, 0.28737829],
 [0.20022041, 0.16184418, 0.25939959, 0.00546446, 0.36415219],
 [0.22920615, 0.27439767, 0.36991198, 0.00624375, 0.39911759]]

answered Jan 14 '20 at 10:17

yatu

86,083
12
84
139

1

That works well. Now I just need to decode what all that regex means. – BillyJo_rambler Jan 14 '20 at 10:22
Meaning: Replace every block of whitespace that is preceded by a digit and not followed by either a "-" or a digit with one ",". – Matthias Jan 14 '20 at 10:26

score 1 · Answer 2 · answered Jan 14 '20 at 10:21

1

Modify the string until it can be interpreted by literal_eval. Basically this means adding a "," where needed.

If your string is called data:

import ast
data = data.replace('  ', ',').replace('\r\n ', ',')
result = ast.literal_eval(dat2)

Now result will contain your list of lists.

I'm not sure if you have spaces or tabs there. If you have tabs you'll have to replace ' ' by '\t\t'.

answered Jan 14 '20 at 10:21

Matthias

12,873
6
42
48

`ast.literal_eval` is dubbed **dangerous**. – amirouche Jan 14 '20 at 10:26
2

@amirouche Do you confuse `eval` and `literal_eval`? Read https://stackoverflow.com/questions/4710247/python-3-are-there-any-known-security-holes-in-ast-literal-evalnode-or-string. – Matthias Jan 14 '20 at 10:27
Sorry for the noise! – amirouche Jan 14 '20 at 21:12

score 0 · Answer 3 · answered Jan 14 '20 at 10:38

This is a rough idea. Can be made smaller.

input = '[[ 0.03694573  0.01084746  0.01306414  0.00499198  0.00307188]\r\n [ 0.03780498  0.02610241  0.00967986  0.02228694 -0.01153102]\r\n [ 0.03837708  0.02111896  0.01370878 -0.00136839  0.01906253]\r\n [ 0.03490672 -0.02791057  0.08824896 -0.01991765  0.01964361]\r\n [ 0.0811892   0.28361901  0.21539196 -0.00259571  0.28737829]\r\n [ 0.20022041  0.16184418  0.25939959  0.00546446  0.36415219]\r\n [ 0.22920615  0.27439767  0.36991198  0.00624375  0.39911759]]'

input = input.replace('\r\n', ',')

array_of_arrays = input.split(',')
array_of_strings = []
for arr in array_of_arrays:
     _arr = arr.replace('[', '')
     _arr = _arr.replace(']', '')
     array_of_strings.append(_arr)
result = []
for arr in array_of_strings:
     num_strings = arr.split()
     num_arr = [float(x) for x in num_strings]
     result.append(num_arr)

print(result)

Convert string list (with spaces instead of commas) into a list of lists

3 Answers3