Meteor one time or "static" publish without collection tracking

Question

Suppose that one needs to send the same collection of 10,000 documents down to every client for a Meteor app.

At a high level, I'm aware that the server does some bookkeeping for every client subscription - namely, it tracks the state of the subscription so that it can send the appropriate changes for the client. However, this is horribly inefficient if each client has the same large data set where each document has many fields.

It seems that there used to be a way to send a "static" publish down the wire, where the initial query was published and never changed again. This seems like a much more efficient way to do this.

Is there a correct way to do this in the current version of Meteor (0.6.5.1)?

EDIT: As a clarification, this question isn't about client-side reactivity. It's about reducing the overhead of server-side tracking of client collections.

A related question: Is there a way to tell meteor a collection is static (will never change)?

Update: It turns out that doing this in Meteor 0.7 or earlier will incur some serious performance issues. See https://stackoverflow.com/a/21835534/586086 for how we got around this.

Answer 1

http://docs.meteor.com/#find:

Statics.find({}, {reactive: false} )

Edited to reflect comment:

Do you have some information that the reactive: false param is only client side? You may be right, it's a reasonable, maybe likely interpretation. I don't have time to check, but I thought this may also be a server side directive, saying not to poll the mongo result set. Willing to learn...

You say

However, this is horribly inefficient if each client has the same large data set where each document has many fields.

Now we are possibly discussing the efficiency of the server code, and its polling of the mongo source for updates that happen outside of from the server. Please make that another question, which is far above my ability to answer! I doubt that is happening once per connected client, more likely is a sync between app server info and mongo server.

The client requests you issue, including sorting, should all be labelled non-reactive. That is separate from whether you can issue them with sorting instructions, or whether they can be retriggered through other reactivity, but which need not include a trip to the server. Once each document reaches the client side, it is cached. You can still do whatever minimongo does, no loss in ability. There is no client asking server if there are updates, you don't need to shut that off. The server pushes only when needed.

Answer 2

I think using the manual publish ( this.added ) still works to get rid of overhead created by the server observing data for changes. The observers either need to be added manually or are created by returning a Collection.curser.

If the data set is big you might also be concerned about the overhead of a merge box holding a copy of the data for each client. To get rid of that you could copy the collection locally and stop the subscription.

var staticData = new Meteor.Collection( "staticData" );

if (Meteor.isServer ){
  var dataToPublish = staticData.find().fetch();  // query mongo when server starts

  Meteor.publish( "publishOnce" , function () {
    var self = this;
    dataToPublish.forEach(function (doc) {
      self.added("staticData", doc._id, doc);  //sends data to client and will not continue to observe collection
    });
  });
}

if ( Meteor.isClient ){
  var subHandle = Meteor.subscribe( "publishOnce" );  // fills client 'staticData' collection but also leave merge box copy of data on server

  var staticDataLocal = new Meteor.Collection( null );  // to store data after subscription stops

  Deps.autorun( function(){
    if ( subHandle.ready() ){
      staticData.find( {} ).forEach( function ( doc ){ 
        staticDataLocal.insert( doc );  // move all data to local copy
      });
      subHandle.stop();  // removes 'publishOnce' data from merge box on server but leaves 'staticData' collection empty on client
    }
  });
}

update: I added comments to the code to make my approach more clear. The meteor docs for stop() on the subscribe handle say "This will typically result in the server directing the client to remove the subscription's data from the client's cache" so maybe there is a way to stop the subscription ( remove from merge box ) that leaves the data on the client. That would be ideal and avoid the copying overhead on the client.

Anyway the original approach with set and flush would also have left the data in merge box so maybe that is alright.

Answer 3

As you've already pointed out yourself in googlegroups, you should use a Meteor Method for sending static data to the client.
And there is this neat package for working with Methods without async headaches.

Answer 4

Also, you could script out the data to a js file, as either an array or an object, minimize it, then link to it as a distinct resource. See http://developer.yahoo.com/performance/rules.html for Add an Expires or a Cache-Control Header. You probably don't want meteor to bundle it for you.

This would be the least traffic, and could make subsequent loads of your site much swifter.

Answer 5

as a response to a Meteor call, return an array of documents (use fetch()) No reactivity or logging. On client, create a dep when you do a query, or retrieve the key from the session, and it is reactive on the client.

Mini mongo just does js array/object manipulation with an syntax interpreting dsl between you and your data.

Answer 6

The new fast-render package makes one time publish to a client collection possible.

var staticData = new Meteor.Collection ('staticData');

if ( Meteor.isServer ){

  FastRender.onAllRoutes( function(){
    this.find( staticData, {} );
  });
}