0

I'm writing a python script which takes signals from various different components, such as an accelerometer, GPS position data etc and then saves all of this data in one class (i'm calling a signal class). This way, it doesnt matter where the acceleration data is coming from, because it will all be processed into the same format. Every time there is a new piece of data, I currently add this new value to an expanding list. I chose a list as I belive it is dynamic, so you can add data without to much computational cost. Comapred to, numpy arrays which are static.

However, I need to also perform mathematical operations on these datasets in near live-time. Would it be faster to:

  • Store the data initially as a numpy array, and expand it as data is added
  • Store the data as an expanding list, and every time some math needs to be performed on the data convert what is needed into a numpy array and then use numpy functions
  • Keep all of the data as lists, and write custom functions to perform the math.
  • Some other method, that I dont know about?

The update times vary, depending on where the data comes from, from anywhere between 1Hz to 1000Hz.

Tom McLean
  • 5,583
  • 1
  • 11
  • 36
  • 1
    It would be best to measure the performance of various approaches. BTW, you could create a big numpy array and then not use the whole array, and increase its size only after it is filled up. That is what a list does internally. – zvone Jan 15 '21 at 15:22
  • Also, maybe this has some useful info: https://stackoverflow.com/questions/7133885/fastest-way-to-grow-a-numpy-numeric-array – zvone Jan 15 '21 at 15:22
  • Are you needing to perform these mathematical operations across "all" of the data or just the last little bit? If it's just the last little bit then just take the last required part of the list. Turn it into a numpy array and do your math there. – NanoBennett Jan 15 '21 at 16:23
  • There's not a single approach that works for all cases. It depends on the specific math you want to do, how long your arrays would be, how fast you need your calculations to be, do you know the max size ahead of time, etc. People often make tradeoffs amongst the various requirements. The fastest approach is usually to per-allocate the memory and structure the calculations so they only need the most recent data point and intermediate accumulated values (like if you need an average, as intermediate values keep the sum and N). – tom10 Jan 15 '21 at 16:39

0 Answers0