I have large set of short time series data (average length of short time series = 20). The total size of the data is around 6 GB. Current system working in following way: 1) Load 6 GB data into RAM. 2) Process the data. 3) Put the forecast value corresponding to each time series in excel.
The problem is every time i run the above system it is taking nearly 1 hour in my 8 GB RAM PC. Please suggest a better way to reduce my time.
Use a faster programming language. For example, you can prefer using Julia or C++ instead of MATLAB or Python.
Try to make your code more efficient. For example, instead of passing data to your functions by copying them (pass by value), try to pass them as reference (What's the difference between passing by reference vs. passing by value?). Use more efficient data structures.
Divide your dataset into smaller parts. Work on each small part separately. Then, merge the outputs at the end.
