Adding records to a numpy record array

Question

Let's say I define a record array

>>> y=np.zeros(4,dtype=('a4,int32,float64'))

and then I proceed to fill up the 4 records available. Now I get more data, something like

>>> c=('a',7,'24.5')

and I want to add this record to y. I can't figure out a clean way to do it. The best I have seen in np.concatenate(), but that would require turning c into an record array in and of itself. Is there any simple way to tack my tuple c onto y? This seems like it should be really straightforward and widely documented. Apologies if it is. I haven't been able to find it.

Answer 1

You can use numpy.append(), but as you need to convert the new data into a record array also:

import numpy as np
y = np.zeros(4,dtype=('a4,int32,float64'))
y = np.append(y, np.array([("0",7,24.5)], dtype=y.dtype))

Since ndarray can't dynamic change it's size, you need to copy all the data when you want to append some new data. You can create a class that reduce the resize frequency:

import numpy as np

class DynamicRecArray(object):
    def __init__(self, dtype):
        self.dtype = np.dtype(dtype)
        self.length = 0
        self.size = 10
        self._data = np.empty(self.size, dtype=self.dtype)

    def __len__(self):
        return self.length

    def append(self, rec):
        if self.length == self.size:
            self.size = int(1.5*self.size)
            self._data = np.resize(self._data, self.size)
        self._data[self.length] = rec
        self.length += 1

    def extend(self, recs):
        for rec in recs:
            self.append(rec)

    @property
    def data(self):
        return self._data[:self.length]

y = DynamicRecArray(('a4,int32,float64'))
y.extend([("xyz", 12, 3.2), ("abc", 100, 0.2)])
y.append(("123", 1000, 0))
print y.data
for i in xrange(100):
    y.append((str(i), i, i+0.1))

Answer 2

This is because concatenating numpy arrays is typically avoided as it requires reallocation of contiguous memory space. Size your array with room to spare, and then concatenate in large chunks if needed. This post may be of some help.