DDP between two servers doesn't reconnect

Question

I have two meteor applications connected via DDP on different servers and server A send data to server B. This is the way they work.

Server A

Items = new Meteor.Collection('items');
Items.insert({name: 'item 1'});
if (Meteor.isServer) {
  Meteor.publish('items', function() {
    return Items.find();
  });
}

Server B

var remote = DDP.connect('http://server-a/');
Items = new Meteor.Collection('items', remote); 

remote.subscribe('items');
Items.find().observe({
  added: function(item) {
    console.log(item);
  }
});

Every time I call Items.insert(something) on server A, on Server B I got a log on the console with the object I saved on Server A. But if Server B lost Internet connection, the data inserted on Server A doesn't appear anymore on Server B when it reconnect to Internet.

Server B is connected to Internet through a router. This problem only happen when I disconnect and reconnect the router, not when I disconnect and reconnect the server from the router. Both servers are on different networks and connect via Internet.

I created a timer on Server B that call remote.status() but always get { status: 'connected', connected: true, retryCount: 0 } when connected or disconnected from Internet.

Update: steps to reproduce

I created a project on github with the testing code https://github.com/camilosw/ddp-servers-test. Server A is installed on http://ddpserverstest-9592.onmodulus.net/

My computer is connected to Internet through a wireless cable modem.

1. Run mrt on server-b folder
2. Go to http://ddpserverstest-9592.onmodulus.net/ and click the link Insert (you can click delete to remove all previous inserts). You must see a message on your local console with the added item.
3. Turn off the wireless on the computer and click the insert link again. (You will need to click on another computer with Internet access, I used an smartphone to click the link)
4. Turn on the wireless on the computer. You must see a message on your local console with the second item.
5. Now, turn off the cable modem and click the insert link again.
6. Turn on the cable modem. This time, the new item doesn't appear on the console.

I also did it with an android smartphone using the option to share Internet to my computer via wireless. First I turned off and on the wireless on my computer and worked right. Then I turned off and on the Internet connection on the smartphone and I got the same problem.

Update 2

I have two wireless router on my office. I found that the same problem happen if I move between routers.

Answer 1

Emily Stark, from the Meteor Team, confirmed that this is due to a missing feature on the current implementation (version 0.7.0.1 at the moment I write this answer). Their answer is here https://github.com/meteor/meteor/issues/1543. Below is their answer and a workaround she suggest:

The server-to-server connection is not reconnecting because Meteor currently doesn't do any heartbeating on server-to-server DDP connections. Just as in any other TCP connection, once you switch to a different router, no data can be sent or received on the connection, but the client will not notice unless it attempts to send some data and times out. This differs from browser-to-server DDP connections, which run over SockJS. SockJS does its own heartbeating that we can use to detect dead connections.

To see this in action, here is some code that I added to server-b in your example:

var heartbeatOutstanding = false;
Meteor.setInterval(function () {
  if (! heartbeatOutstanding) {
   console.log("Sending heartbeat");
    remote.call("heartbeat", function () {
      console.log("Heartbeat returned");
      heartbeatOutstanding = false;
    });
    heartbeatOutstanding = true;
  }
}, 3000);

remote.onReconnect = function () {
  console.log("RECONNECTING REMOTE");
};

With this code added in there, server-b will reconnect after a long enough time goes by without an ACK from server-a for the TCP segments that are delivering the heartbeat method call. On my machine, this is just a couple minutes, and I get an ETIMEDOUT followed by a reconnect.

I've opened a separate task for us to think about implementing heartbeating on server-to-server DDP connections during our next bug week. In the meantime, you can always implement heartbeating in your application to ensure that a DDP reconnection happens if the client can no longer talk to the server.

Answer 2

I think you are not passing DDP connection object to the Collection correctly, try:

var remote = DDP.connect('http://server-a/');
Items = new Meteor.Collection('items', { connection: remote }); 

It might be useful for debugging to try all these connection games from the browser console first, since Meteor provides the same API of connection/collections on the client (except for the control flow). Just open any Meteor application and try this lines from the console.

Answer 3

I revised a sample of communication between two ddp server, based on camilosw's code.

Server A as Cloud Data Center. Server B as Data Source, if some data changed, should be send to Server A.

You can find the code from https://github.com/iascchen/ddp-servers-test