61

In their arXiv paper, the original authors of Julia mention the following:

2.14 Parallelism. Parallel execution is provided by a message-based multi-processing system implemented in Julia in the standard library. The language design supports the implementation of such libraries by providing symmetric coroutines, which can also be thought of as cooperatively scheduled threads. This feature allows asynchronous communication to be hidden inside libraries, rather than requiring the user to set up callbacks. Julia does not currently support native threads, which is a limitation, but has the advantage of avoiding the complexities of synchronized use of shared memory.

What do they mean by saying that Julia does not support native threads? What is a native thread?

Do other interpreted languages such as Python or R support this type of parallelism? Is Julia alone in this?

StefanKarpinski
  • 32,404
  • 10
  • 86
  • 111
Amelio Vazquez-Reina
  • 91,494
  • 132
  • 359
  • 564

1 Answers1

76

Update

When this question was asked in 2013, Julia indeed had no support for multithreading. Today, however, Julia supports native threading with what has emerged as the best language model for composable thread programming. This model was pioneered by Cilk and Intel's Threading Building Blocks, pushed further at the language level by Go, and is now also used by Julia. Statements in the original answer below about other dynamic languages remain true: they still do not support native threading with support for parallel execution of user code. The history of adding threading capabilities to Julia progressed in the following high-level stages:

  • Julia 0.3: support for native threads with an OpenMP-like compute model (i.e. parallel for loops). This functionality was highly limited: no I/O or networking was allowed in parallel code.

  • Julia 1.3: fully composable high performance M:N threading. This threading model is the same as Go (and Cilk and Intel TBB), where tasks are used to express potential concurrency, and those tasks are run on threads from a pool of native threads by a scheduler.

  • Julia 1.7: support for migration of tasks between native threads. This allows a task to begin execution on one native thread, get suspended, and then resume on a different thread, allowing full utilization of available compute resources.

Original Answer

"Native threads" are separate contexts of execution, managed by the operating system kernel, accessing a shared memory space and potentially executing concurrently on separate cores. Compare this with separate processes, which may execute concurrently on multiple cores but have separate memory spaces. Making sure that processes interact nicely is easy since they can only communicate with each other via the kernel. Ensuring that threads don't interact in unpredictable, buggy ways is very hard since they can read and write to the same memory in an unrestricted manner.

The R situation is fairly straightforward: R is not multithreaded. Python is a little more complicated: Python does support threading, but due to the global interpreter lock (GIL), no actual concurrent execution of Python code is possible. Other popular open source dynamic languages are in various mixed states with respect to native threading (Ruby: no/kinda/yes?; Node.js: no), but in general, the answer is no, they do not support fully concurrent native threading, so Julia is not alone in this.

When we do add shared-memory parallelism to Julia, as we plan to – whether using native threads or multiple processes with shared memory – it will be true concurrency and there will be no GIL preventing simultaneous execution of Julia code. However, this is an incredibly tricky feature to add to a language, as attested by the non-existent or limited support in other very popular, mature dynamic languages. Adding a shared-memory concurrency model is technically difficult, but the real problem is designing a programming model that will allow programmers to make effective use of hardware concurrency in a productive and safe way. This problem is generally unsolved and is a very active area of research and experimentation – there is no "gold standard" to copy. We could just add POSIX threads support, but that programming model is general considered to be dangerous and incredibly difficult to use correctly and effectively. Go has an excellent concurrency story, but it is designed for writing highly concurrent servers, not for concurrently operating on large data, so it's not at all clear that simply copying Go's model is a good idea for Julia.

StefanKarpinski
  • 32,404
  • 10
  • 86
  • 111
  • 6
    Thanks. When you say that with Python `"no actual concurrent execution of Python code is possible"`, you mean within the same process, correct? From what I understand Python **already** supports shared-memory parallelism using subprocesses via the multiprocessing package. In other words, when you said `"When we add [...] it will be true concurrency and there will be no GIL preventing simultaneous execution of Julia code."`, you are referring to the scenario where Julia does this with **native threads**, otherwise Julia would implement shared-memory parallelism as Python does. Correct? – Amelio Vazquez-Reina May 07 '13 at 19:17
  • 6
    Yes, if it's in separate processes, then we're not talking about threading anymore. It's unclear at this point whether the nature of the shared-memory parallelism in Julia will be based on threads or processes with shared memory, and that distinction is something of an implementation detail in any case. Python's multiprocessing package does seem to do a form of multi-process shared-memory parallelism, but it's not as convenient or natural as I would want it to be for a language-level primitive. – StefanKarpinski May 07 '13 at 21:47
  • I should also point out that experimental work on something like Python's multiprocessing package has been done: https://groups.google.com/forum/?fromgroups=#!searchin/julia-users/shmem$20mmap/julia-users/Y_ME9uO5tVQ/HR0PPwtRIfIJ. – StefanKarpinski May 07 '13 at 21:50
  • Thanks. For some reason the last link doesn't seem to work. What is the title of the thread? – Amelio Vazquez-Reina May 07 '13 at 21:52
  • I'm guessing this one: [shmem and mmap](https://groups.google.com/forum/?fromgroups=#!searchin/julia-users/shmem/julia-users/Y_ME9uO5tVQ/HR0PPwtRIfIJ) – remram Jul 11 '14 at 03:25
  • 8
    JPython has no GIL and supports concurrency better than CPython. IronPython too. – Alexander Jul 14 '14 at 08:02
  • How does it compare to other specialized languages such as Chapel, Loci or Liszt? – skan Aug 09 '17 at 16:10
  • @StefanKarpinski should checkout Rust's concurrency model which very low level and allows the programmer to choose how to proceed and build higher level concurrent programs. – xiaodai Oct 03 '17 at 05:34