25

I think Erlang-style concurrency is the answer to exponential growth of core count. You can kind of fake it with other main stream languages. But the solutions always leave me wanting. I am not willing to give up multi-paradigm programming (C++/D) to switch to Erlang's draconian syntax.

What is Erlang-style concurrency:

From one of the language authors(What is Erlang's concurrency model actually ?):

  • Lightweight concurrency.
    Cheap to create threads and cheap to maintain insane numbers.
  • Asynchronous communication.
    Threads only communicate via messages.
  • Error handling.
  • Process isolation.

Or from an informed blogger (What is Erlang-Style Concurrency?):

  • Fast process creation/destruction
  • Ability to support >> 10 000 concurrent processes with largely unchanged characteristics.
  • Fast asynchronous message passing.
  • Copying message-passing semantics (share-nothing concurrency).
  • Process monitoring.
  • Selective message reception.

I think D's message passing can accomplish most of these features. The ones I wonder about are ">>10,000 concurrent processes(threads)" and "fast process creation/destruction".

How well does D handle these requirements?

I think that to support them correctly you'd have to use green threads. Can D's message passing features be used with green threads library?

Community
  • 1
  • 1
deft_code
  • 57,255
  • 29
  • 141
  • 224
  • And at Rust. Rust has message passing, pattern matching AND green threads, which allows for huge number of concurrent tasks / processes. It also has some type of channels (similar to Go). – Samuel Lampa Aug 06 '14 at 14:50

3 Answers3

19

Storage is thread-local by default in D, so nothing is shared between threads unless it is specifically marked as shared. If you mark a variable as shared, you can then use the traditional mutexes and conditions as well as synchronized objects and the like to deal with concurrency. However, the preferred means of communicating between threads is to use the message passing facilities in std.concurrency and let all data stay thread-local, only using shared when you must. All objects passed between threads using std.concurrency must either be passed by value or be immutable, so no sharing occurs there and it is completely thread-safe. However, it can currently be a bit of a pain to get an immutable reference type which isn't an array (idup generally makes it easy for arrays), so it can be a bit annoying to pass anything other than value types or arrays (though hopefully that situation improves soon as compiler and standard library bugs relating to const and immutable get fixed and more code is made const-correct).

Now, while message passing in D will definitely result in cleaner, safer code than what you'd get in languages like C++ or Java, it is built on top of normal, C threads (e.g. Linux uses pthreads), so it does not have the kind of light-weight threads that Erlang does, and so dealing with multiple threads is not going to be as efficient as Erlang.

Of course, I don't see any reason why a more efficient thread system could not be written using D, at which point you might be able to get thread efficiency similar to that of Erlang, and it could presumably use an API similar to that of std.concurrency, but all of D's standard threading stuff is built on top of normal, C threads, so you'd have to do all of that yourself, and depending on how you implemented it and depending on how exactly the thread-local/shared stuff is dealt with by the compiler and druntime, it could be difficult to get the type system to enforce that everything be thread-local with your "green" threads. I'm afraid that I don't know enough about exactly how shared is implemented or how "green" threads work to know for sure.

Regardless, D's message passing system will certainly result in dealing with threads being more pleasant than C++ or even Java, but it's not designed to be streamlined in the same way that Erlang is. D is a general purpose systems language, not a language specifically designed to use threads for everything and thus to use them absolutely as efficiently as possible. A large portion of D's standard facilities are built on top of C, so a lot of its efficiency characteristics will be similar to those of C.

Jonathan M Davis
  • 37,181
  • 17
  • 72
  • 102
  • 7
    Erlang threads are really just a stack frame and a bit of context - probably fiber is a more apt term. The Erlang VM schedules a few OS threads to service 1000's of user-mode fibers. This is neat and scales better than any other scheme, but renders useless D's notion of shared vs. thread local storage. It's not D's fault since at least the x86 processor offers no hardware enforcement to isolate one fibers data from another. For example, in the case of real OS threads, the x86 GS register keeps per-thread data isolated in 64-bit machines. – srking Nov 03 '10 at 05:15
  • 5
    Also, D does have a fiber library that works well if you aren't worried about the thread-local storage issue. Context switch times between D fibers are around 6x faster than between D threads. – srking Nov 03 '10 at 05:41
  • D Fibers are included into standart libruary. But they implement only non-preemtive multitasking. – uhbif19 May 12 '13 at 19:14
  • @srking the neat thing about Erlang's "fiber" like threads is they are effectively pre-emptively scheduled (unlike eqivalent green threads/co-routines in Go, Node etc) – corford Feb 05 '20 at 01:28
7

This functionality is frequently used in combination with async I/O to efficiently communicate with external sources of data as well. The vibe.d framework seems to offer both the many-fibers-on-a-few-OS-threads threading model and async I/O libraries (in addition to a whole bunch of web application libraries and project management tools).

As an unrelated side note, it's pretty freaking cool that D is both low-level enough that you could write this framework in it and high-level enough to be a compelling language to write your web applications in on top of the framework. Other popular languages with similar frameworks (node.js, Ruby's EventMachine, coroutines in Python and Go) are unable to compete with D on low-level systems coding. Other popular languages with similar systems programming facilities (C, C++) can't compete on high-level application coding.

I'm new to D, but I gotta say, I like what I see.

Dan
  • 7,155
  • 2
  • 29
  • 54
3

From whatever little I know about D: its message passing infrastructure is built on top its threading facilities. If the core threading library is a wrapper on OS threads, there is little chance that concurrency in D will reach the magnitude (>> 10000) of Erlang. Moreover D do not enforce immutability on objects, so it is easy to mess things up. So, Erlang is the best choice for heavy concurrency. Probably you can write the concurrency stuff in Erlang and the rest of the project in D. Still, it is possible to have efficient green threads in C like languages (C++, D etc) - have a look at Protothreads and ZeroMQ. You can implement very efficient messaging frameworks using these, and calling them via a C shim or directly from D.

BCS
  • 75,627
  • 68
  • 187
  • 294
Vijay Mathew
  • 26,737
  • 4
  • 62
  • 93
  • 7
    You *can* make an object (deep) immutable in D with the `immutable` modifier. It is just not the default. – kennytm Oct 25 '10 at 12:03
  • 1
    @KennyTM Yes, that's why I said "D do not *enforce* immutability on objects..." – Vijay Mathew Oct 25 '10 at 12:24
  • 6
    @Vijay: Yes, so? You can't send mutable data via D's message passing interface. The compiler will stop you if you send an object that is not `immutable`. – kennytm Oct 25 '10 at 12:55
  • @KennyTM Is it possible for two concurrent objects in D to *share* a mutable object, say a global variable? – Vijay Mathew Oct 25 '10 at 13:22
  • 4
    @Vijay: Mutable global variables are "thread local" by default. It is possible to make it shared by adding the `shared` modifier (ref: http://www.digitalmars.com/d/2.0/migrate-to-shared.html.) – kennytm Oct 25 '10 at 13:25
  • 2
    In short, D doesn't prevent the use of thread hostile features (it has inline ASM and direct access to C after all), but it makes you go a bit out of your way to use them. (And that's a big chunk of the design philosophy of D.) – BCS Oct 25 '10 at 14:06
  • As to doing light weight threads in C and calling that; don't, unless your using a library that's already written, there is very little that C can do that can't be done just a well in D. – BCS Oct 25 '10 at 14:12