-1

I have a Rails application which is mainly used to show data to the users through dashboards.

  • The application is currently running on a PostgreSQL database.
  • All the history values stored so far are in the form [date,variable,value].
  • The records can be stored every 5 minutes, and the number of variables I'm tracking is high, aka, millions of records in the history table.

Now I need to do operations in between variables. So let's say I have a variable A and a variable B, I would need:

  • the value of A/B.
  • the history of A/B calculated every t (which can be 5 minutes/1 day/1 week).

Currently I am doing everything within the Rails environment, but I know that Ruby isn't a fast language so I'm planning a change.

In the near future I would also need to calculate derivatives and integrals so I was thinking that I might need a dedicated/decoupled services that only does data crunching.

What would be a good tool/language that allows me to do complex math operations on large set of data?

Has QUIT--Anony-Mousse
  • 76,138
  • 12
  • 138
  • 194
Francesco Meli
  • 2,484
  • 2
  • 21
  • 52

2 Answers2

1

What would be a good tool/language that allows me to do complex math operations on large set of data?

Python and numpy/scipy are what you need! I'm not trying to sell some python to a ruby afficionado, but I can vouch for these packages, they're awesome.

It's close to matlab where you manipulate native vector/matrix data types, using functions that support vectorization. It can use openmp for parallelism, there is also numexpr. It is both very fast and very easy to use, plus the library is enormous, here is an example.

bobflux
  • 11,123
  • 3
  • 27
  • 27
0

Thanks to peufeu answer, I could find these guys: http://sciruby.com/

Francesco Meli
  • 2,484
  • 2
  • 21
  • 52