Python Load in multiple files - nested loop

Question

I am struggeling with a problem since a few days. I have 100 different time folders, every folder contains 11 sensor1.dat, sensor2.dat etc.. files.

Example for sensor1.dat

x1,x2,x3
1,2,3
4,5,6
7,8,9

I am trying to loop all over the timefolders and load in for every timestep the last row of the sensor files

Results could be a nested list: Outer list contains the time data - Inner List for every time data all sensor_data, which means here: Outer List 100 rows, inner list 11 rows for each outer row.

Therefore I tried following:

def getData():
for TIME in TIMES:   
    cd_string = TIME
    cd_string = string.replace(cd_string,'\n','') 
    os.chdir(cd_string)

    print(TIME)
    i_sensors=11
    data=[]
    for i in xrange(1,i_sensors+1):
        #print i
        sensor_string='../../sets/'+cd_string+'/w_z_sensor_'+str(i)+'_U.dat'
        data_input = np.genfromtxt(sensor_string, delimiter="," , skip_header=1)
        data.append(data_input[-1,:])  

    os.chdir('../') 
    return data



######MAIN####
TIMES = os.popen("ls -d 0.31* ").readlines()

zw=[[]] #*len(TIMES)

for i in xrange(0,len(TIMES)):
   zw[i].append(getData())

The function getData works fine and I get all the data I need for the sensor files for each time step,

The problem is the main part. Here I amt trying to append the sensor data for every time folder. This does not work. With the uncommented z=[[]]*len(TIMES) it just only copy the data in 100 rows

Where is my problem?

Any advice and thanks in advance?

See http://stackoverflow.com/q/240178/3001761 for the core problem you're having. — jonrsharpe, Jun 24 '15 at 10:58
In addition you seem to append a list to a list, this means that the list will contain another list. I think what you should do is to do `zw.append(getData())` instead and just initialize `zw` as an empty list. — skyking, Jun 24 '15 at 11:12

score 0 · Accepted Answer · answered Jun 24 '15 at 12:02

Like jonrsharpe mentioned, your issue is that when you are creating lists using * operator, internally it is actually creating that much amount of list positions and it copies the elements from the list you provide to the new list indexes. If the list contains a reference object, it copies the reference rather than doing a copying the content of the reference, hence if you make any change to any reference in one position, it will reflect in all other positions.

Example -

>>> lst = [[1]] * 10
>>> lst
[[1], [1], [1], [1], [1], [1], [1], [1], [1], [1]]
>>> lst[0][0] = 5
>>> lst
[[5], [5], [5], [5], [5], [5], [5], [5], [5], [5]]

You should try initializing the list using list comprehension -

Example -

>>> lst = [[1] for _ in range(10)]
>>> lst
[[1], [1], [1], [1], [1], [1], [1], [1], [1], [1]]
>>> lst[0][0] = 5
>>> lst
[[5], [1], [1], [1], [1], [1], [1], [1], [1], [1]]

In your case -

zw=[[] for _ in range(len(TIMES))]

Python Load in multiple files - nested loop

1 Answers1