2

Did anyone come across using kdb’s matrix function? I found it quite slow compared to other tools.

For inverse matrix function inv on a 1000 by 1000 float matrix , It took kdb+ 166,682 milliseconds for 10 executions. For matrix multiplication, it took 3,380 milliseconds, where the two matrices share the same dimension as 1000 by 1000.

I also tried the same experiment on DolphinDB, a similar time series database with built-in analytics features (DolphinDB ). DolphinDB’s inv function is about 17 times faster and matrix multiplication is about 6 times faster.

I would think matrix operation optimization should be something well established. Can any kdb expert explain the reason behind it or if there are any ways to improve it?

I used KDB+ 4.0 64 bit version and DolphinDB_Linux_V2.00.7(DolphinDB community version: 2 cores and 8GB memory). Both experiments are conducted using 2 cores of CPU.

KDB implementation

// Start the server
rlwrap -r taskset -c 0,1 ./l64/q -p 5002 -s 2

// The code
ma:1000 cut (-5.0+ 1000000?10.0)
mb:1000 cut (-5.0+ 1000000?10.0)

\t do[10;inv ma]
16682

\t do[10;ma mmu mb]
3380

DolphinDB implementation

// Start the server
rlwrap -r ./dolphindb -localSite localhost:5002:local5002 -localExecutors 1

// The code
ma=(-5.0+ rand(10.0,1000000))$1000:1000;
mb=(-5.0+ rand(10.0,1000000))$1000:1000;
timer(10) ma.inv();
Time elapsed: 975.349 ms

timer(10) ma**mb;
Time elapsed: 581.618 ms
Polly
  • 603
  • 3
  • 13

2 Answers2

3

DolphinDB uses openblas for matrix operations like inv while kdb+ doesn't.

Davis Zhou
  • 353
  • 4
  • 6
2

Nobody here is privy to what kdb and dolphin are doing under the covers for these operations, other than knowing that inv in kdb uses LU decomposition (previously it used Cholesky decomposition as well).

In general kdb is not optimised for matrix operations, it is optimised for vector operations. Case in point - if you have enough worker threads and use peach:

q)\t do[10;ma mmu mb]
2182
q)\t do[10;mmu[;mb]peach ma]
375

If you wanted to do serious matrix operations in kdb you would load C libraries to carry out the heavy lifting. For example using the Q math library (http://althenia.net/qml):

q)\t inv ma
1588
q)\t .qml.minv ma
689
terrylynch
  • 11,844
  • 13
  • 21