5

I am experimenting with akka / scala 2.10.3 on an Ubuntu 12.04 system running an AMD Quad-Core processor.

I have written a server that should perform some computation when it receives a request. A client will send some data (list of strings) using an actor message. When the actor on the server receives the list, it splits it into four lists and has four child actors sort the list.

So far so good: the program works and I could verify that the client receives a correctly sorted list as a result.

There are however two things I do not understand in the behaviour of the server:

  1. As soon as I start the server, even though no actor is doing any processing (they are all are waiting for messages), all the four cores of my processor go up to almost 100% usage. How is this possible? Shouldn't the cores only be used when some actors are scheduled to process some messages?
  2. I had expected that having the work done by four actors would automatically imply that the sorting would be sped up by a factor of four: each actor is run in a separate thread, and each thread is run by a different core. However, this is not the case: no speed-up observed or, even, the actor-based sorting is much slower than a plain single-threaded sort. Can the cause be my dispatcher configuration, i.e. the default configuration does not exploit multiple cores automatically?

Edit Following Dan Simon's suggestions, I have looked into problem 1 using visualvm.

visualvm reports several dispatcher threads that are waiting most of the time (so they do not seem to use much CPU time). visualvm also shows several other threads that are running all the time, even though the application is not doing anything; at least, none of my code is being executed. These busy threads are named New I/O worker #1, #2, #4, #5, New I/O boss #3, New I/O server boss, Signal Dispatcher, RMI TCP Connection(2)-127.0.0.1, Attach Listener, and RMI TCP Accept-0.

I have some experience programming akka but almost no experience configuring it, so I cannot make much sense out the above information.

Giorgio
  • 5,023
  • 6
  • 41
  • 71
  • The default dispatcher uses a fork-join scheduler and will have multiple threads corresponding to the number of physical cores (I believe it's 2 per core), so I suspect there is another issue going on. I've noticed fork-join does a lot of busy-waiting sometimes, but not like what you describe. Can you post some code? – Dan Simon Mar 26 '14 at 22:15
  • @Dan Simon: Thanks for the feedback. The code is not small enough to be posted here, I could upload it somewhere and post the link here. Or what parts of the code would you like to see? – Giorgio Mar 26 '14 at 22:17
  • Not really sure, do your actors do anything when constructed or in their `preStart` method? Do the actors have any kind of initialization messages or perhaps some kind of periodic polling for work? Also, have you tried using a profiler like visualvm to see where the CPU is being used? – Dan Simon Mar 26 '14 at 22:23
  • @Dan Simon: Some actors do have a `preStart` method in which they create child actors and / or send a few messages. Actors also have some initialization but, after that, they move into a state in which they just wait for incoming messages. As far as I know, this should not imply polling. I have to take some sleep now, but tomorrow I will try using a profiler and see if I can find out more. Thanks for the suggestion. – Giorgio Mar 26 '14 at 22:27

1 Answers1

0

"They move into a state in which they just wait for incoming messages"

Are the actors notified when a messages arrives via a callback method or are the actors themselves polling some queue to see if there are new messages?

The latter can cause 100% cpu usage if you are polling, finding no work and then going right back to poll again (essentially busy spinning). The fix is wait (Thread.yield(), LockSupport.parkNanos(1) etc) for a bit before polling again.

CaptainHastings
  • 1,557
  • 1
  • 15
  • 32
  • We are facing the same issue. Our Utilization is rocketing to 800 % with majority of akka actors with this stack strace. "default-akka.actor.default-dispatcher-10": awaiting notification on [0x000000074e491db0]. Could you suggest how should we proceed ? – prit kalra Jul 17 '19 at 13:45
  • @prit kalra Try to find if that thread is busy spinning as in polling, returning false as nothing was polled & then goes right back to polling. Fix is to put in a wait strategy if poll() returns false. You could try something like LockSupport.parkNanos(1) – CaptainHastings Jul 23 '19 at 22:04
  • Hastings Thanks, found the solution. Actually default dispatcher and scheduler were being created apart from a custom dispatcher. Default dispatcher can create unneccesary threads which were leading to thread starvation and using all the resources. – prit kalra Aug 19 '19 at 08:50