convert list with arrays from csv file into floats

Question

I have a csv file with list of samples of various parameters of accelerometer and gyroscopic data. It contains floats(IQR,Range,MAD,...) and arrays(ZCD values). When I read it back, they are returned as strings and for my processing,I need numbers (int/float). I tried

    results = map(int, results)

and

results = [int(i) for i in results]

I got them from here, but I am having a problem with the arrays.

Traceback (most recent call last):
 File "/home/pi/Desktop/imu/mpu_v16.py", line 458, in <module>
  main()
 File "/home/pi/Desktop/imu/mpu_v16.py", line 439, in main
  compare()
 File "/home/pi/Desktop/imu/mpu_v16.py", line 305, in compare
  nrow = [float(i) for i in nrow]
ValueError: could not convert string to float: [ 53  73  79 139]

How can I solve this?

I am adding a sample of my data:

    C,3.3440452548950501e-17,0.99999999999999645,0.89244290726827058,1.7947499048650259,3.3651560716219242,[108 123 149 220 235],0.99999999999999822,110223.99999999971,331.99999999999915,-5.9524005537131887e-17,0.99999999999999989,0.81051947660768831,1.4756039753111405,4.4268119259334213,[103 122 160 205 212],0.99999999999999989,110224.00000000001,332.00000000000006,-6.4540073419474463e-17,0.999999999999999,0.74198651253618131,0.63512619216067612,4.256170326687128,[106 164 192 226],0.99999999999999933,110224.00000000001,332.00000000000006,2.083131190971185e-16,1.0000000000000009,0.66659374901400581,0.52759419283475883,4.5104130995285256,[  7  14  45  56 150 327],1.0000000000000002,110223.99999999994,331.99999999999983,-3.0890618042093025e-17,0.99999999999999667,0.58289607514346964,0.21669963911591134,4.7919240951669444,[ 82 149 208],0.99999999999999822,110223.99999999942,331.99999999999824,-3.2771643497971487e-16,1.0000000000000009,0.58746356061392535,0.29681486739557372,5.2741718744905794,[ 26  48  59  66 114 171 231 242],1.0000000000000002,110224.00000000036,332.00000000000108,-0.57536274345915739,0.147595080030029,0.13018399571123057

the problem I see in the error you posted is that you are trying to convert an array of strings into float, whereas you should apply int() one by one. As a quick and dirty solution, apply another for loop. — giosans, Jul 07 '16 at 13:59
Your csv does not contain numbers, it contains space separated sequences of numbers. — MisterMiyagi, Jul 07 '16 at 14:06
Looks like you wrote numpy array to the file as strings.you will need to strip the [] and then split each line on whitespace and cast each element — Padraic Cunningham, Jul 07 '16 at 14:08
post your input file data, and the code you wrote to read it — joel goldstick, Jul 07 '16 at 14:12
You should give a sample of your data if you are having conversion problems. — MisterMiyagi, Jul 07 '16 at 14:12
Added a sample of data. Also,I am working on the answer given by @MisterMiyagi — Kranthi Kumar, Jul 08 '16 at 03:54

MisterMiyagi · Answer 1 · 2016-07-07T14:11:56.837

Your csv is malformed, or you are reading it incorrectly. What you have extracted aren't individual numbers, it's multiple numbers boxed into brackets. See the error message:

ValueError: could not convert string to float: [ 53  73  79 139]
            ###              This is your data ^               ^

So you have a list like ["[ 53 73 79 139]", "[ 123 12 11]"] when you should actually have [["53", "73", "79", "139"], ["123", "12", "11"]] or ["53", "73", "79", "139", "123", "12", "11"].

If your csv is malformed, you can correct this on the fly by splitting the erroneous values. This will give you a flat list like [53, 73, 79, 139]:

nrow = [float(i) for elem in nrow for i in elem.strip('[] ').split()]
#                ^              ^ extract every sequence of numbers
#                                          ^              ^ remove [] and whitespace around sequences
#                                                            ^     ^ split the actual numbers

This will give you a list of lists like [[ 53, 73, 79, 139], ...]:

    nrow = [[float(i) for i in elem.strip('[] ').split()] for elem in nrow ]

score 1 · Accepted Answer · edited May 23 '17 at 11:58

Because your input consists of strings of floats AND strings of arrays, you need a way of differentiating between the two. One way is to perhaps check the string for spaces? Then, for float strings, simply use float(i) like you have been doing. Then for array strings, try splitting on the spaces to separate the array elements and then running float on each individual string float (convert a string to an array).

Arbitrary float input: "54.6"
Check for spaces: False
Call: float("54.6")

Arbitrary array input: "[ 53 73 79 139]"
Check for spaces: True
Remove brackets: " 53 73 79 139"
Split on spaces and store in an array: ["53","73","79","139"]
Call float on each element: [53,73,79,139]

,Your solution not exactly solved my problem, but helped me somewhat.I had changed my storing method to overcome this array problem. — Kranthi Kumar, Jul 08 '16 at 09:10

convert list with arrays from csv file into floats

2 Answers2