1

I am reading data from a file and extracting the information that I want from it, as floats. I am then storing this data into a temporary list and using vstack to try and put the data into an array with each row being new data that is being processed.

for line in lines:
    if line.find('GPS')!=-1:
        funcGPS(line)
    if line.find('-----')!=-1:
        MasterArray = numpy.vstack(temp)
        temp = []
        #print MasterArray
    if line.find('SERVO')!=-1:
        funcSERVO(line)

This is how I am trying to copy the data to the array. I am successfully extracting the data and after extracting the data obtained I want to add the data to the array. Currently, it is copying the data over the previous data when I build onto MasterArray. Is there anyway to do this without specifically stating what the size of MasterArray is? I don't want to limit the amount of data that can be obtained.

Thanks!

Rubedo
  • 11
  • 2
  • 1
    Where does `temp` come from? You can use `np.append` instead of `np.vstack` inside your loop, or else you can keep adding it to a list and vstack that list _after_ your loop, but you might be better off loading all your data in at once, with `np.genfromtxt`, and then editing it. – askewchan Sep 10 '13 at 16:25
  • 1
    We probably can't help unless we know what `temp` is and how it's constructed. `vstack` accepts a sequence of arrays, and joins them together. So `temp` should be a tuple or list of arrays to join; if you mean to append to `MasterArray`, then your definition of `temp` will need to look something like `temp = (MasterArray, newrow)`. This isn't the best approach to dynamic array creation though; see [here](http://stackoverflow.com/questions/2641691/building-up-an-array-in-numpy-scipy-by-iteration-in-python) for more information. – senderle Sep 10 '13 at 16:38
  • temp is a 1xn matrix ( currently 1x20 ). Append works but it adds it to the end of the column and I want it to be a new row. temp comes from this line of code which is extracting the data from the file. for element in listline: value = element.split(':') value=value[1].split('}') float(value[0]) #print value[0] temp.append(float(value[0])) This is done twice before being placed into the MasterArray. Thank you for the help! – Rubedo Sep 10 '13 at 19:05

1 Answers1

2

Growing an array in a loop is considered bad practice. If you know the final size, or a reasonable upper limit, of the array it's best to pre-allocate the array and then fill it. If you don't know the final size you can often use a temporary list and convert it to an array after the loop. This is not unique to numpy, you'll see similar usage patterns in matlab and c programs.

Do something like this if you know the final size of the array or a reasonable upper limit.

N = 100
array = np.zeros(100)
count = 0
for line in file:
    array[count] = ...
    count += 1
 array = array[:count] # Or maybe array[:count].copy()

or something like this if you don't.

temp = []
for line in file:
    temp.append(...)
array = np.array(temp)
Bi Rico
  • 25,283
  • 3
  • 52
  • 75
  • Thanks! I don't know the size, but its going to be extremely large. I am appending the data to temp and then I wanted to stack it into another array so all the data is in one matrix. – Rubedo Sep 10 '13 at 19:09