Why is node.js breaking incoming data into chunks?

Question

The following code in node.js does not log all incoming data inside the brackets, rather, it breaks the data into chunks. So for example, if the incoming data is ABCDEF...XYZ it logs the data as [ABC][DEF]...[XYZ] rather than [ABCDEF...XYZ]. The data is much larger of course, the alphabet is just an example.

How should I write this so that all incoming data is logged once inside the brackets and not in parts?

chatServer.on('connection', function(client) 
{
    client.on('data', function(data) 
    {
        console.log('[' + data.toString() + ']');
    })    
})

score 8 · Accepted Answer · answered Jun 28 '12 at 01:47

8

Well your data is arriving in packets, so (in this case) you should be concatenating the packets into a variable that you define outside the function.

buffer = '';

chatServer.on('connection', function(client) 
{
    client.on('data', function(data) 
    {
        buffer += data.toString();
    })    
});

console.log('[' + buffer + ']');

answered Jun 28 '12 at 01:47

matchdav

715
1
7
16

1

So the chunks correspond to arriving TCP packets? – Zaz Jul 17 '16 at 14:56
@Zaz No. It all depends on the `highwaterMark` value, which you can pass to the streams constructor, read more here: https://nodejs.org/api/stream.html#stream_buffering – basickarl Sep 05 '17 at 11:30
@Zaz: Provided that `client` (which is a dublex stream) is in **flowing mode**, the data added with `stream.push(chunk)` will be delivered with a 'data' event. As a result, **the size of the chunk corresponds to the data of the `TCP segment` which arrived**. See here: https://nodejs.org/api/stream.html#stream_readable_push_chunk_encoding However, if the stream is paused and then resumed, it will be buffering up to `highwaterMark`, so the chunk released afterwards will not correspond to the data contained to a single `TCP segment`. – Charidimos Feb 06 '18 at 00:53

vossad01 · Answer 2 · 2017-12-13T15:25:49.217

4

Like matthewdavidson said, you are subscribing to every "chunk" of data that is sent rather than the whole message. It is more likely you want to capture the data in a closure within the function and still respond asynchronously. Try the following:

chatServer.on('connection', function(client) 
{
    var buffer = '';

    client.on('data', function(data) 
    {
        buffer += data;
    })

    client.on('end', function(){
        console.log('[' + buffer + ']');
    })
});

Checkout http://www.nodebeginner.org/#handling-post-requests for more information

edited Dec 13 '17 at 15:25

answered Jun 28 '12 at 01:57

vossad01

11,552
8
56
109

But doesn't client.on('end', function(){...}) get called only when a client disconnects? The reason why I am looking into determining the end of the buffer is so I can call JSON.parse with a complete string. – Hahnemann Jun 28 '12 at 02:30
Ahh now I have found the module you are using [http://nodejs.org/api/net.html] What is your client look like? Does it close its connection when it is finishing its queue? Does it just keep sending data sets and you are responsible for recognizing start/end of each set? – vossad01 Jun 28 '12 at 02:50
The second one. The client as an input and an output stream that sends and receives data to/from node. They are closed (terminated) by the user. – Hahnemann Jun 28 '12 at 04:32
How do you know you've got a "complete string"? Is there some character that marks the end of a "message"? Or is a message considered finished when nothing more comes in for a certain period of time (guessing from the name "chatServer" that that might be the case)? In any case, you need to buffer up the incoming data and then decide when you've got enough of it to parse it. – ebohlman Jun 28 '12 at 05:22
I don't. I checked the [net](http://nodejs.org/api/net.html) specs (thanks to @vossad01) but I could not find an event that triggers when the end of data occurs. – Hahnemann Jun 28 '12 at 14:06
What you get in net is simply a stream, a stream does not know about the protocol you use for messages, so it cannot notify you when a complete "message" has buffered. It recognizes FIN as the end of the stream and will notify you with 'end.' @ebohlman has some good suggestions on ways you might determine the end of a "message". If you use a control character/sequence you would need to do that from your 'data' handling function. For a timeout look at [setting a timeout](http://nodejs.org/api/net.html#net_socket_settimeout_timeout_callback) on you socket and using the the 'timeout' event. – vossad01 Jun 29 '12 at 00:15

Why is node.js breaking incoming data into chunks?

2 Answers2

Linked