Publishing/subscribing multiple subsets of the same server collection

Question

EDIT: this question, some of the answers, and some of the comments, contain a lot of misinformation. See how Meteor collections, publications and subscriptions work for an accurate understanding of publishing and subscribing to multiple subsets of the same server collection.

---

How does one go about publishing different subsets (or "views") of a single collection on the server as multiple collections on the client?

Here is some pseudo-code to help illustrate my question:

items collection on the server

Assume that I have an items collection on the server with millions of records. Let's also assume that:

1. 50 records have the enabled property set to true, and;
2. 100 records have the processed property set to true.

All others are set to false.

items:
{
    "_id": "uniqueid1",
    "title": "item #1",
    "enabled": false,
    "processed": false
},
{
    "_id": "uniqueid2",
    "title": "item #2",
    "enabled": false,
    "processed": true
},
...
{
    "_id": "uniqueid458734958",
    "title": "item #458734958",
    "enabled": true,
    "processed": true
}

Server code

Let's publish two "views" of the same server collection. One will send down a cursor with 50 records, and the other will send down a cursor with 100 records. There are over 458 million records in this fictitious server-side database, and the client does not need to know about all of those (in fact, sending them all down would probably take several hours in this example):

var Items = new Meteor.Collection("items");

Meteor.publish("enabled_items", function () {
    // Only 50 "Items" have enabled set to true
    return Items.find({enabled: true});
});

Meteor.publish("processed_items", function () {
    // Only 100 "Items" have processed set to true
    return Items.find({processed: true});
});

Client code

In order to support the latency compensation technique, we are forced to declare a single collection Items on the client. It should become apparent where the flaw is: how does one differentiate between Items for enabled_items and Items for processed_items?

var Items = new Meteor.Collection("items");

Meteor.subscribe("enabled_items", function () {
    // This will output 50, fine
    console.log(Items.find().count());
});

Meteor.subscribe("processed_items", function () {
    // This will also output 50, since we have no choice but to use
    // the same "Items" collection.
    console.log(Items.find().count());
});

My current solution involves monkey-patching _publishCursor to allow the subscription name to be used instead of the collection name. But that won't do any latency compensation. Every write has to round-trip to the server:

// On the client:
var EnabledItems = new Meteor.Collection("enabled_items");
var ProcessedItems = new Meteor.Collection("processed_items");

With the monkey-patch in place, this will work. But go into Offline mode and changes won't appear on the client right away -- we'll need to be connected to the server to see changes.

What's the correct approach?

---

EDIT: I just revisited this thread and I realize that, as it stands, my question and answers and plethora of comments carry a lot of misinformation.

What it comes down to is that I misunderstood the publish-subscribe relationship. I thought that when you published a cursor, it would land on the client as a separate collection from other published cursors that originated from the same server collection. This is simply not how it works. The idea is that both the client and server have the same collections, but it's what is in the collections that differs. The pub-sub contracts negotiate which documents end up on the client. Tom's answer is technically correct, but was missing a few details to turn my assumptions around. I answered a similar question to mine in another SO thread based on Tom's explanation, but keeping in mind my original misunderstanding of Meteor's pub-sub: Meteor publish/subscribe strategies for unique client-side collections

Hope this helps those who run across this thread and come away more confused than anything!

Answer 1

Could you not just use the same query client-side when you want to look at the items?

In a lib directory:

enabledItems = function() {
  return Items.find({enabled: true});
}
processedItems = function() {
  return Items.find({processed: true});
}

On the server:

Meteor.publish('enabled_items', function() {
  return enabledItems();
});
Meteor.publish('processed_items', function() {
  return processedItems();
});

On the client

Meteor.subscribe('enabled_items');
Meteor.subscribe('processed_items');

Template.enabledItems.items = function() {
  return enabledItems();
};
Template.processedItems.items = function() {
  return processedItems();
};

If you think about it, it is better this way as if you insert (locally) an item which is both enabled and processed, it can appear in both lists (a opposed to if you had two separate collections).

NOTE

I realised I was kind of unclear, so I've expanded this a little, hope it helps.

Answer 2

you could make two separate publications like this..

Server publications

Meteor.publish("enabled_items", function(){
    var self = this;

    var handle = Items.find({enabled: true}).observe({
        added: function(item){
            self.set("enabled_items", item._id, item);
            self.flush();
        },
        changed: function(item){
            self.set("enabled_items", item._id, item);
            self.flush();
        }
    });

    this.onStop(function() {
        handle.stop();
    });
});

Meteor.publish("disabled_items", function(){
    var self = this;

    var handle = Items.find({enabled: false}).observe({
        added: function(item){
            self.set("disabled_items", item._id, item);
            self.flush();
        },
        changed: function(item){
            self.set("disabled_items", item._id, item);
            self.flush();
        }
    });

    this.onStop(function() {
        handle.stop();
    });
});

Client Subscriptions

var EnabledItems = new Meteor.Collection("enabled_items"),
    DisabledItems = new Meteor.Collection("disabled_items");

Meteor.subscribe("enabled_items");
Meteor.subscribe("disabled_items");

Answer 3

I've managed to achieve some promising preliminary results by approaching the problem with a single publish/subscribe per collection, and leveraging $or in the find query.

The idea is to provide a wrapper around Meteor.Collection that allows you to add "views", which are basically named cursors. But what's really happening is that these cursors aren't run individually... their selectors are extracted, $or'd together and run as a single query and onto a single pub-sub.

It's not perfect, in that an offset/limit won't work with this technique, but at the moment minimongo doesn't support it anyway.

But ultimately what it allows you to do is to declare what looks like different subsets of the same collection, but under the hood they are the same subset. There's just a bit of abstraction in front to make them feel cleanly separated.

Example:

// Place this code in a file read by both client and server:
var Users = new Collection("users");
Users.view("enabledUsers", function (collection) {
    return collection.find({ enabled: true }, { sort: { name: 1 } });
});

Or if you want to pass parameters:

Users.view("filteredUsers", function (collection) {
    return collection.find({ enabled: true, name: this.search, { sort: { name: 1 } });
}, function () {
    return { search: Session.get("searchterms"); };
});

The parameters are given as objects, because it's a single publish/subscribe $or'd together, I needed a way to get the right parameters since they get mixed together.

And to actually use it in a template:

Template.main.enabledUsers = function () {
    return Users.get("enabledUsers");
};
Template.main.filteredUsers = function () {
    return Users.get("filteredUsers");
};

In short, I take advantage of having the same code running in both server and client, and if the server isn't doing something, the client will, or vice versa.

And most importantly, only the records you are interested in are getting sent down to the client. This is all achievable without an abstraction layer by simply doing the $or yourself, but that $or will get pretty ugly as more subsets get added. This just helps manage it with minimal code.

I wrote this quickly to test it out, apologies for the length and lack of documentation:

test.js

// Shared (client and server)
var Collection = function () {
    var SimulatedCollection = function () {
        var collections = {};

        return function (name) {
            var captured = {
                find: [],
                findOne: []
            };

            collections[name] = {
                find: function () {
                    captured.find.push(([]).slice.call(arguments));
                    return collections[name];
                },
                findOne: function () {
                    captured.findOne.push(([]).slice.call(arguments));
                    return collections[name];
                },
                captured: function () {
                    return captured;
                }
            };

            return collections[name];
        };
    }();

    return function (collectionName) {
        var collection = new Meteor.Collection(collectionName);
        var views = {};

        Meteor.startup(function () {
            var viewName, view, pubName, viewNames = [];

            for (viewName in views) {
                view = views[viewName];
                viewNames.push(viewName);
            }

            pubName = viewNames.join("__");

            if (Meteor.publish) {
                Meteor.publish(pubName, function (params) {
                    var viewName, view, selectors = [], simulated, captured;

                    for (viewName in views) {
                        view = views[viewName];

                        // Run the query callback but provide a SimulatedCollection
                        // to capture what is attempted on the collection. Also provide
                        // the parameters we would be passing as the context:
                        if (_.isFunction(view.query)) {
                            simulated = view.query.call(params, SimulatedCollection(collectionName));
                        }

                        if (simulated) {
                            captured = simulated.captured();
                            if (captured.find) {
                                selectors.push(captured.find[0][0]);
                            }
                        }
                    }

                    if (selectors.length > 0) {
                        return collection.find({ $or: selectors });
                    }
                });
            }

            if (Meteor.subscribe) {
                Meteor.autosubscribe(function () {
                    var viewName, view, params = {};

                    for (viewName in views) {
                        view = views[viewName];
                        params = _.extend(params, view.params.call(this, viewName));
                    }

                    Meteor.subscribe.call(this, pubName, params);
                });
            }
        });

        collection.view = function (viewName, query, params) {
            // Store in views object -- we will iterate over it on startup
            views[viewName] = {
                collectionName: collectionName,
                query: query,
                params: params
            };

            return views[viewName];
        };

        collection.get = function (viewName, optQuery) {
            var query = views[viewName].query;
            var params = views[viewName].params.call(this, viewName);

            if (_.isFunction(optQuery)) {
                // Optional alternate query provided, use it instead
                return optQuery.call(params, collection);
            } else {
                if (_.isFunction(query)) {
                    // In most cases, run default query
                    return query.call(params, collection);
                }
            }
        };

        return collection;
    };
}();

var Items = new Collection("items");

if (Meteor.isServer) {
    // Bootstrap data -- server only
    Meteor.startup(function () {
        if (Items.find().count() === 0) {
            Items.insert({title: "item #01", enabled: true, processed: true});
            Items.insert({title: "item #02", enabled: false, processed: false});
            Items.insert({title: "item #03", enabled: false, processed: false});
            Items.insert({title: "item #04", enabled: false, processed: false});
            Items.insert({title: "item #05", enabled: false, processed: true});
            Items.insert({title: "item #06", enabled: true, processed: true});
            Items.insert({title: "item #07", enabled: false, processed: true});
            Items.insert({title: "item #08", enabled: true, processed: false});
            Items.insert({title: "item #09", enabled: false, processed: true});
            Items.insert({title: "item #10", enabled: true, processed: true});
            Items.insert({title: "item #11", enabled: true, processed: true});
            Items.insert({title: "item #12", enabled: true, processed: false});
            Items.insert({title: "item #13", enabled: false, processed: true});
            Items.insert({title: "item #14", enabled: true, processed: true});
            Items.insert({title: "item #15", enabled: false, processed: false});
        }
    });
}

Items.view("enabledItems", function (collection) {
    return collection.find({
        enabled: true,
        title: new RegExp(RegExp.escape(this.search1 || ""), "i")
    }, {
        sort: { title: 1 }
    });
}, function () {
    return {
        search1: Session.get("search1")
    };
});

Items.view("processedItems", function (collection) {
    return collection.find({
        processed: true,
        title: new RegExp(RegExp.escape(this.search2 || ""), "i")
    }, {
        sort: { title: 1 }
    });
}, function () {
    return {
        search2: Session.get("search2")
    };
});

if (Meteor.isClient) {
    // Client-only templating code

    Template.main.enabledItems = function () {
        return Items.get("enabledItems");
    };
    Template.main.processedItems = function () {
        return Items.get("processedItems");
    };

    // Basic search filtering
    Session.get("search1", "");
    Session.get("search2", "");

    Template.main.search1 = function () {
        return Session.get("search1");
    };
    Template.main.search2 = function () {
        return Session.get("search2");
    };
    Template.main.events({
        "keyup [name='search1']": function (event, template) {
            Session.set("search1", $(template.find("[name='search1']")).val());
        },
        "keyup [name='search2']": function (event, template) {
            Session.set("search2", $(template.find("[name='search2']")).val());
        }
    });
    Template.main.preserve([
        "[name='search1']",
        "[name='search2']"
    ]);
}

// Utility, shared across client/server, used for search
if (!RegExp.escape) {
    RegExp.escape = function (text) {
        return text.replace(/[-[\]{}()*+?.,\\^$|#\s]/g, "\\$&");
    };
}

test.html

<head>
    <title>Collection View Test</title>
</head>

<body>
    {{> main}}
</body>

<template name="main">
    <h1>Collection View Test</h1>
    <div style="float: left; border-right: 3px double #000; margin-right: 10px; padding-right: 10px;">
        <h2>Enabled Items</h2>
        <input type="text" name="search1" value="{{search1}}" placeholder="search this column" />
        <ul>
            {{#each enabledItems}}
                <li>{{title}}</li>
            {{/each}}
        </ul>
    </div>
    <div style="float: left;">
        <h2>Processed Items</h2>
        <input type="text" name="search2" value="{{search2}}" placeholder="search this column" />
        <ul>
            {{#each processedItems}}
                <li>{{title}}</li>
            {{/each}}
        </ul>
    </div>
</template>