Poorly Ordered Data Over Websockets

Question

I've been working on learning to deliver and display data in "real-time" over websockets. Real-time, in this context, just means that the data is being read from a sensor (camera with image processing) every .03 to 1 seconds. Each data point consists of a time and a value (t,v) which are encoded as doubles (although the time is always an integer in this case, I'm not assuming it will be).

The server side uses Alchemy Websockets implementation (C#) as I found it very easy to understand/modify for my purposes.

I'm leveraging the websockets examples found here and here as well as the examples included with Alchemy.

I'm using HighCharts to display the data in "real-time", but I also have it printing to a div for debug purposes (independent example so they don't interfere with each other).

So much of it already works pretty well, but there's an obvious problem that happens when I send data too quickly (to be clear, sending the data about a point every second or two results in a nice graph which doesn't appear to have any problems - the problems become more pronounced the faster I call the alchemy server "send" function).

The data appears to be coming in in the wrong order resulting in an interesting "mangled" effect.

I'm going to start delving into the packet order contained on the server side buffer (the server is instructed to send a certain number of "historical" points when a new user connects and it is already running - this results in a pronounced problem like the one shown above) as well as the client side receive order by looking at the timestamps.

The error is inconsistent in that each time I reload the page it results in a different "mangled" data set. This makes me suspect the communication over websockets to be responsible or something involving the alchemy server.

I will attach the full code if necessary, but right now it's rather messy so I am more looking for troubleshooting ideas.

I've gathered this is not expected behavior for a web socket as it is built on TCP.

Any suggestions/ideas for things to look at?

Thanks!

Edit: I ran another test to check how many data points were out of order each time I refreshed the page. The numbers are as follows:

1 2 3 25 6 5 10 11 96 2 8

Very inconsistent (never 0). Certainly interesting!

This result was taken by excluding the charting component and only using websockets and an array to store the data.

Update:

I decided I'd start analyzing the order things come in in and it does appear to be randomly receiving out of order points using an identical data set. I implemented an "insert" function which will take into account out of order packets. The result (plus a little theme change) looks pretty good!

Open question remains: Is it expected that a websocket can deliver information out of order? Or is there something wrong with my implementation on the server side (or Alchemy). I will investigate further when I have time.

SOLUTION!

I figured it out! After a lot of testing, I realized that my Connection object (which is responsible for watching a data set for new data and sending it as is appropriate given how the connection is configured) was implemented using a Timer object. This was something I took from an example (normally I just use the Thread object for most things asynchronous).

As the Timer object speeds up, it starts executing asynchronously with it's previous calls to it's Tick function. This means that very occasionally, one call to it's Tick function will happen a little faster than another (due to the latency in the Alchemy Send function). This causes minor out of order problems.

I switched the implementation of the communication loop from a Timer object to a Thread object, thus enforcing synchronization, and the out of order packets went away!

Answer 1

Websockets use TCP, and TCP guarantees that data is delivered in order.

I would hope that the browser fires websocket message events in order as well. In my tests, this seemed to be the case.

Using this simple Node app to test:

var sio = require('socket.io')
    , http = require('http')
    , express = require('express')
    , app = express()
    , srv = http.createServer(app)
    , io = sio.listen(srv)
    , fs = require('fs')
    , idx = fs.readFileSync('index.html').toString()
    ;

app.get('/', function(req, res) {
    res.send(idx);
});

var i = 0;
setInterval(function() {
    io.sockets.emit('count', i++);
}, 1);

srv.listen(888);

This simply sends websocket messages as fast as it can with a number that's incremented each message. The client:

<script src="/socket.io/socket.io.js"></script>
<script>
var last = 0;
var socket = io.connect('/');
socket.on('count', function(d) {
    if (d-1 != last) console.warn('out of order!', last, d);
    last = d;
});
</script>

Throws a console warning if it receives a message that contains a number that is not one more than the previous message.

In Chrome and Firefox, I saw zero out-of-order messages.

I also tried blocking for a while in the message received event (for (var i = 0; i < 1000000; i++) { }) to simulate work that would cause messages to queue up. Message events still fired in order.

In other words, it's something else in your setup. Most likely, the Alchemy server is actually sending the messages in a different order.

Answer 2

Do not use an object like Timer which has asynchronous callbacks when the task is synchronous. Use a Thread and run the communication loop that way.

Answer 3

I don't know when the issue was posted. I also have similar problem. I use Alchemy Client send small data, then there is no problem. There are a lot example for chat Service. But when I send some file more than 4 KB (not precisely), the problem take place. I try to find what happened. I wrote a program sent numbers from 0-7000 by Alchemy client, and received from UserContext.DataFrame(onreceive), there will happen that DataFrame.ToString will get extra "\0\0\0\0" on about position 508. Then, after this position, the data ordering will be wrong. I used the 2.2.1 from nuget. And I read version 2.0 on GiHub. The source code is not workable. So, to old and no reference value.