2

In my application there are three erlang processes which run periodically at an interval of 10ms, 1s and 5s, let's call them as process A, B and C respectively; and there are two c-nodes, X and Y which perform some low level hardware access.

The c-nodes has a fixed pool of linux thread to serve the queries received from the erlang process simultaneously. One erlang process can send only one query to a c-node as the calls are blocking, but the c-nodes can receive multiple calls from different erlang process and serve them simultaneously through it's fixed pool of threads.

In my application process A and C are using the service of c-node X, and process is B using the service of c-node Y.

We observed that the execution time of all the three erlang processes are varying drastically over time, i.e. to start with process A, B and C is completing their task in 3ms, 100ms and 250ms respectively, but over a period of an hour it is increasing to 200ms, 600ms, 1300ms, then again it is coming down slowly to 3ms, 100ms and 250ms, and as we continue to run we are seeing this pattern repetitively.

I have also implemented the same erlang process logic in C application, there I do not see this issues.

What could be the possible explanation behind this slow down pattern, and what would be the right way to benchmark or profile this behaviour?

Greg
  • 369
  • 1
  • 6
  • What are you using to schedule intervals? – Wintermute Jan 13 '17 at 06:21
  • I am using monotonic_time as in the thread [link] (http://stackoverflow.com/questions/41378523/how-to-run-a-erlang-process-periodically-with-precise-time-i-e-10ms) reply by Hynek -Pichi- Vychodil – Greg Jan 13 '17 at 08:51
  • Are the erlang processes synchronously communicating with the c-nodes? – rvirding Jan 13 '17 at 10:42
  • Yes, the erlang processes are synchronously communicating with the c-nodes – Greg Jan 13 '17 at 15:50
  • 2
    The problem might be in time warp mode you are using. Try passing +C multi_time_warp to erl and measure again. http://erlang.org/doc/apps/erts/time_correction.html – Wintermute Jan 16 '17 at 06:12
  • Thanks @Stan for your input, after using multi_time_warp the execution time has reduced considerably; however, there is still a large difference between the min and max execution time, i.e. the min time earlier was 3ms and max was 128ms, the max is now reduced to 48ms, but I need to make the max under 8ms! – Greg Jan 16 '17 at 15:49
  • Well, I'm not 100% sure, but as far as I know, further "timer precision optimizations" are going to be manual adjustments of timers you set based on messages sent by erlang:monitor(time_offset, clock_service). It's rather tedious and I wouldn't guarantee its effectiveness (you can read more about that here: http://erlang.org/doc/apps/erts/time_correction.html under 'New Erlang Monotonic Time' label). I feel like you are quite close to the best precision Erlang can give you in your specific case. I'd really like to hear from someone who knows how to achieve better results – Wintermute Jan 16 '17 at 16:32
  • Oh, by the way, another option might be writing a port driver, thus coding your time-critical parts in C - which, by your own experiment, is far better – Wintermute Jan 16 '17 at 16:34

0 Answers0