3

I am working on an agent-based simulation with approx. 10 million agents.

The agents will be doing nothing for days/weeks at a time, and when they do have to do something, it is not cpu intensive (should take < 1ms).

Is spawning 10 million threads a reasonable thing to try to do? I have read up on Thread/sleep and future which seem promising.

Any advice or suggested reading on such a project would be greatly appreciated.

user2179977
  • 656
  • 6
  • 14
  • 1
    I would not recommend creating 10 million threads for the scenario you described. [This answer](http://stackoverflow.com/questions/5483047/why-is-creating-a-thread-said-to-be-expensive) talks about some of the overhead associated with creating a thread. For example, by default every thread is allocated 256Kb of stack space. You can calculate what that means for 10,000,000 threads. Of course you could reduce the per-thread stack size, but it would still amount to a lot of memory. – user100464 May 02 '14 at 20:40
  • 3
    `Thread/sleep` is **absolutely** the wrong thing to use -- it means that you're blocking a slot in your thread pool, so no other process can run with that thread, so you're increasing the number of real (OS-level) threads you need if you use it (or do any other blocking operations within your executors). If you use something like `core.async`, then a slot in the thread pool can be handed off to a different task when an individual goroutine doesn't need it. That said, we don't know _nearly_ enough about your problem space to recommend the right tool. – Charles Duffy May 02 '14 at 22:00

2 Answers2

7

If you use Clojure's built in agents, these run in a thread pool and will consume only a small amount of ram. events can be dispatched to these agents in various ways depending on your needs. It is fine to have 10M agents in a Clojure program. It may also be worth using one of the async programing libraries such as core.async and have 10M go blocks instead of agents.

Arthur Ulfeldt
  • 90,827
  • 27
  • 201
  • 284
1

Do those agents have significantly different behavior? Or can their behavior be derived from a few models and further configured with data? If you answer no to both questions you might want to consider using asynchronous queues as the main building block of your design.

You should definitely take Charles Duffy's advice seriously and find out more about how what Thread/sleep does and how threads work in general. But I think the issue with the design is not so much technical here as much as relating building blocks of a program with real world objects to a fault.

If you can provide some more details of the nature of what you're trying to accomplish, I might be able to edit this answer and provide a more concrete answer.

muhuk
  • 15,777
  • 9
  • 59
  • 98