How to code two-way duplex streams in NodeJS

Question

In the latest few versions of NodeJS (v0.10.X as of writing), the Streams API has undergone a welcome redesign and I would like to start using it now.

I want to wrap both the input and output of a socket with an object which implements a protocol.

The so-called Duplex interface, seems to just be any stream which is readable and writable (like a socket).

It is not clear whether Duplexes should be like A or B, or whether it doesn't matter.

   +---+        +---+
-->| A |-->     |   |-->
   +---+        | B |
                |   |<--
                +---+

What is the correct code-structure/interface for an object which has two writeables and two readables?

+--------+   +----------+   +----
|       r|-->|w        r|-->|w    
| socket |   | protocol |   | rest of app
|       w|<--|r        w|<--|r    
+--------+   +----------+   +----

The problem with the diagram above is that the protocol object needs two separate read methods and two write methods.

Off the top of my head, I could make the protocol produce 'left' and 'right' duplex objects, or 'in' and 'out' duplex objects (to slice it a different way).

Are either of these the preferred way, or is there a better solution?

fadedbee · Accepted Answer · 2013-10-30T12:19:42.693

          |      app      |
          +---------------+
              ^       |
              |       V      
           +-----+ +-----+
           |     | |     |
+----------|     |-|     |-+
| protocol | .up | |.down| |
+----------|     |-|     |-+
           |     | |     |
           +-----+ +-----+
              ^       |
              |       V
          +---------------+
          |     socket    |

My solution was to make a Protocol class, which created an Up Transform and a Down Transform.

The Protocol constructor passes a reference (to itself) when constructing the Up and Down transforms. The _transform method in each of the up and down transforms can then call push on itself, on the other Transform, or both as required. Common state can be kept in the Protocol object.

+1, I think "up" and "down" are good names, because it can stand for abstraction level. You can pipe all the up channels of every layers, and also the down. But, I think it could be usefull to have a basic pipe2 and unpipe2 methods for doing both at the same time. — ESL, May 31 '18 at 22:56

score 1 · Answer 2 · edited May 23 '17 at 12:08

1

A duplex stream is like your diagram B, at least for the user. A more complete view of a stream would be to include producer(source) with the consumer(user). See my previous answer. Try not to think both read/write from a consumer point of view.

What you are doing is building a thin layer over the socket for protocol, so your design is correct :

                         -------+     +----------+     +------
                               r|---->|         r|---->|      
                         socket |     | protocol |     | rest of app
                               w|<----|         w|<----|      
                         -------+     +----------+     +------

You can use duplex or transform for the protocol part.

                 +---------+--------+---------+       +------------------+ 
                 | _write->|        |         |r      |   Transform  ->  |r
                 |-----------Duplex-----------|       +------------------+    
                 |         |        | <-_read |w      |   <- Transform   |w
                 +---------+--------+---------+       +------------------+

process being your protocol related processing on incoming/outgoing data using internal _read, _write. Or you can transform streams. You would pipe protocol to socket and socket to protocol .

edited May 23 '17 at 12:08

Community

1
1

answered Oct 16 '13 at 18:53

user568109

47,225
17
99
123

The protocol needs to share state between the two transforms. How is it best to do this? – fadedbee Oct 16 '13 at 20:13
You can set flags to represent state or store data within a stream. See the [simpleprotocolv2](http://nodejs.org/api/stream.html#stream_example_simpleprotocol_parser_v2) example. You do it with these `this._sawFirstCr = false; this._rawHeader = [];` – user568109 Oct 17 '13 at 03:27
For your task it is best done with duplex as both streams can see the flags easily. For transforms you will have to set flags twice for both the streams. – user568109 Oct 17 '13 at 03:30
My point is that there are two intimately connected transforms in the protocol. A packet from the socket, may cause a packet to be sent to the socket and/or a packet to be sent to the app. Likewise a packet from the app may cause a packet to be sent to the socket and/or a packet to be sent to the app. – fadedbee Oct 17 '13 at 08:08

How to code two-way duplex streams in NodeJS

2 Answers2