2

I have a NiFi node with 40 CPU's and 12 cores per CPU. I have set the 'Maximum Timer Driven Thread Count' to 30. Still it seems some processors are painfully slow.

Can someone please tell me what the optimal value for this variable is based on the CPU's? This information is not available anywhere. Thanks!

Amardeep Flora
  • 1,255
  • 6
  • 13
  • 29
  • How many threads defined in the nifi processors for the whole flow in total? Check this: https://stackoverflow.com/questions/763579/how-many-threads-can-a-java-vm-support – daggett Mar 31 '19 at 04:37
  • Also, what processors exactly do yo uuse? Is this node a VM? or a physical machine? NiFi is IO-intensive, what kind of disk do you use? Is it mounted directly to your machine? Are all of the repositories placed on the same disk? Please note that it could be one of these, but it could also be the answer I've already given. – Ben Yaakobi Mar 31 '19 at 12:52
  • The total threads are around 30 so I am maxing out....based on the link I can go for 40 *12 * 3 for Max Thread Count if I am not wrong? – Amardeep Flora Mar 31 '19 at 21:38
  • It really depends on your data flow – Ben Yaakobi Mar 31 '19 at 22:18
  • You can increase the Run Duration variable and it will increase the throughput. – Darshan Oct 14 '20 at 13:36

1 Answers1

2

The processor itself might be slow.. Increasing the Maximum Timer Driven Thread Count is not enough(although you can probably increase it even more.)

You should increase the Concurrent Tasks for the slow processors. That will make those processors run on multiple threads.

Ben Yaakobi
  • 1,620
  • 8
  • 22
  • The processors are out of the box processors and I am just wondering if Max Tread count should be 30 or I can go for a bigger number like 500-600? – Amardeep Flora Mar 31 '19 at 21:37
  • 1
    I am pretty sure you can. I believe, however, it depends on your flow – Ben Yaakobi Mar 31 '19 at 22:19
  • 1
    If you have 480 cores (40 cpus x 12 cores each), then you can definitely have more than 30 threads :) – Bryan Bende Apr 01 '19 at 13:38
  • @AmarSingh Did you went for 500-600 thread count and is it working good.? In my case I have 4 core CPU and thread count I have is 10. It's working good for me. – nbalodi Apr 21 '22 at 07:32