Scala server with Spray Akka overthreading

Question

I'm using spray.io and akka.io on my freebsd 1CPU/2GB server and I'm facing threading problems. I've started to notice it when I got OutOfMemory exception because of "can't create native thread".

I check Thread.activeCount() usually and I see it grows enormously. Currently I use these settings:

myapp-namespace-akka {
  akka {
    loggers = ["akka.event.Logging$DefaultLogger"]
    loglevel = "DEBUG"
    stdout-loglevel = "DEBUG"

    actor {
      deployment {
        default {
          dispatcher = "nio-dispatcher"
          router = "round-robin"
          nr-of-instances = 1
        }
      }

      debug {
        receive = on
        autoreceive = on
        lifecycle = on
      }

      nio-dispatcher {
        type = "Dispatcher"
        executor = "fork-join-executor"

        fork-join-executor {
          parallelism-min = 8
          parallelism-factor = 1.0
          parallelism-max = 16
          task-peeking-mode = "FIFO"
        }

        shutdown-timeout = 4s
        throughput = 4
        throughput-deadline-time = 0ms
        attempt-teamwork = off
        mailbox-requirement = ""
      }

      aside-dispatcher {
        type = "Dispatcher"
        executor = "fork-join-executor"

        fork-join-executor {
          parallelism-min = 8
          parallelism-factor = 1.0
          parallelism-max = 32
          task-peeking-mode = "FIFO"
        }

        shutdown-timeout = 4s
        throughput = 4
        throughput-deadline-time = 0ms
        attempt-teamwork = on
        mailbox-requirement = ""
      }
    }
  }
}

I want nio-dispatcher to be my default non blocking (lets say single thread) dispatcher. And I execute all my futures (db, network queries) on aside-dispatcher.

I get my contexts through my application as follows:

trait Contexts {
  def system: ActorSystem
  def nio: ExecutionContext
  def aside: ExecutionContext
}

object Contexts {
  val Scope = "myapp-namespace-akka"
}

class ContextsImpl(settings: Config) extends Contexts {
  val System = "myapp-namespace-akka"
  val NioDispatcher = "akka.actor.nio-dispatcher"
  val AsideDispatcher = "akka.actor.aside-dispatcher"
  val Settings = settings.getConfig(Contexts.Scope)

  override val system: ActorSystem = ActorSystem(System, Settings)
  override val nio: ExecutionContext = system.dispatchers.lookup(NioDispatcher)
  override val aside: ExecutionContext = system.dispatchers.lookup(AsideDispatcher)
}

// Spray trait mixed to service actors
trait ImplicitAsideContext {
  this: EnvActor =>
  implicit val aside = env.contexts.aside
}

I think that I did mess up with configs or implementations. Help me out here. Usually I see thousands of threads for now on my app until it crashes (I set freebsd pre process limit to 5000).

score 3 · Accepted Answer · edited May 23 '17 at 12:19

3

If your app indeed starts so many threads this can be usually tracked back to blocking behaviours inside ForkJoinPools (bad bad thing to do!), I explained the issue in detail in the answer here: blocking blocks, so you may want to read up on it there and verify what threads are being created in your app and why – the ForkJoinPool does not have a static upper limit.

edited May 23 '17 at 12:19

Community

1
1

answered Mar 06 '16 at 11:52

Konrad 'ktoso' Malawski

13,102
3
47
52

But why ForkJoinPool by default (in PlayFramework for example), any advantage over ThreadPool if I have primary I/O blocking logic? – selfsx Oct 31 '16 at 11:08
It's the _default_ pool for all operations, which should be reactive - e.g. all actors unless you specify a different dispatcher for them. Blocking should be managed and not just hands-in-the-air-white-flag saying everything is blocking :-) – Konrad 'ktoso' Malawski Nov 04 '16 at 10:48

Scala server with Spray Akka overthreading

1 Answers1