7

I have a node-express application that currently uses Mongoose to connect to MongoDB, and am attempting to migrate it to Azure Cosmos DB.

When I simply allow Mongoose to create the database, the application works fine, however the database is created with individual collection RU pricing.

If I create a new database with Shared throughput enabled and attempt to use that, I get the error Shared throughput collection should have a partition key

I have tried updating the collection schema to include a shard key like this:

const mongoose = require('mongoose');

module.exports = function() {
  const types = mongoose.Schema.Types;
  const messages = new mongoose.Schema({
    order: { type: types.ObjectId, required: true, ref: 'orders' },
    createdAt: { type: Date, default: Date.now },
    sender: { type: types.ObjectId, required: true, ref: 'users' },
    recipient: { type: types.ObjectId, ref: 'users' },
    text: { type: String, required: true },
    seen: { type: Boolean, default: false },
  }, { shardKey: { order: 1 } });

  return mongoose.model('messages', messages);
};

However this does not work.

Any ideas on how to create/use a partition key? Alternatively the database is small, so if its possible to remove the requirement for the partition key that would also be fine.

alt
  • 2,356
  • 2
  • 21
  • 28

4 Answers4

4

Now I don't have an exact answer for this question so no need to accept this unless you feel it's correct.

The best solution I've found so far is that this is due to "Provision Throughput" being checked when the database is created in the Azure console. If you delete and recreate the database with this box not checked (it's right below the input for the database name) then you should no longer encounter this error.

Nolski
  • 443
  • 4
  • 8
  • 24
  • 6
    I believe that creates the database with dedicated throughput, so you have a minimum cost per collection, not per database. – alt Jan 23 '19 at 18:17
1

You specify it when you're creating a collection in the DB that you've opted in for Shared Throughput.

Collection vs Database

If you're using individual collection pricing, you can set the throughput on the individual collections. If you're using the lesser pricing option, you'd get the shared throughput (at the database level) which is less granular but less expensive.

Details here: https://azure.microsoft.com/en-us/blog/sharing-provisioned-throughput-across-multiple-containers-in-azure-cosmosdb/

Partition keys

If you're using the Shared throughput, you'll need a partition Id for the collection that you're adding.

So - create a DB with Shared throughput (check the checkbox below)

After that when you're attempting to add a new document you should be able to create a partition key.

Mavi Domates
  • 4,262
  • 2
  • 26
  • 45
  • The trouble is as far as I understand adding a partition key requires rewriting the application to support partitions, rather than the advertised "just change the mongo connection string" – alt Jan 23 '19 at 18:16
  • @alt - Partitions work transparently once created, so the only logic that would need changed in the app is logic which creates collections, where you are now required to specify a partition key. Unfortunately, *if* you have app logic which creates collections (like in your original question), there doesn't seem to be a good workaround aside from using the official Cosmos SDKs to create the collection. – Dusty Feb 04 '19 at 17:55
  • @Dusty Partitions are not transparent as far as I could tell. You need to specify the partition key that you're wanting to search when you do queries, etc – alt Feb 07 '19 at 22:20
  • @alt - Amending my previous comment, it seems that specifying the partition key in a *find* query is not required, but when updating a document by query, it does seem that you have to specify it, and of course it has to _exist_ on the document, so specifying it during creation is obviously required... so you're right that it's not transparent at least for most write operations. – Dusty Feb 20 '19 at 21:33
  • @Dusty is right, you'll have to go through and be sure all updates, creates, or deletes target a specific shard by including the shard key in the query. – Dan May 03 '19 at 18:51
1

I have yet another not-quite-complete answer for you. It seems like, yes, it is required to use partitioned collections if you are using the shared/db-level throughput model in Cosmos. But it turns out it is possible to create a CosmosDb collection with a partition key using only the MongoDb wire protocol (meaning no dependency on an Azure SDK, and no need to pre-create every collection via the Azure Portal).

The only remaining catch is, I don't think it's possible to run this command via Mongoose, it will probably have to be run directly via the MongoDb Node.js Driver, but at least it can still be run from code.

From a MongoDb Shell:

db.runCommand({shardCollection: "myDbName.nameOfCollectionToCreate", 
               key: {nameOfDesiredPartitionKey: "hashed"}})

This command is meant to set the sharding key for a collection and start sharding the collection, but in CosmosDb it works to create the collection with the desired partitionKey already set.

Dusty
  • 3,946
  • 2
  • 27
  • 41
1

I have an even little more complete answer. You actually can do it with mongoose. I usually do it like this in an Azure Function:

mongoose.connect(process.env.COSMOSDB_CONNSTR, {
useUnifiedTopology: true,
useNewUrlParser: true,
auth: {
    user: process.env.COSMODDB_USER,
    password: process.env.COSMOSDB_PASSWORD,
},
})
.then(() => {
    mongoose.connection.db.admin().command({
        shardCollection: "mydb.mycollection",
        key: { _id: "hashed" }
    })
    console.log('Connection to CosmosDB successful ')
})
.catch((err) => console.error(err))
ltwlf
  • 219
  • 3
  • 5