3

Tl;dr
What happens when one shard in a MongoDB cluster goes out of space? Will all new documents be written to the remaining shards, and the cluster as a whole will continue to function normally? Or is it that such a scenario can never happen as the balancer will make sure that the shards are equally distributed?

Context:
I was having a standalone mongod instance (A) running on a machine with 500GB disk space and was running out of disk space. I've since then sharded the database so that we do not run out of disk space - the cluster has 2 shards as of now (A & B).

After successful sharding, I can see that mongodb has taken up some ~100GB of space on B as chunks are being migrated to the new shard. However, I can see that, it has taken up an additional 30GB of additional space on shard A (the original standalone mongod instance). After some research, it seems ~28GB of that is taken up by the MoveChunk directory, which I guess is safe to remove?

MongoDB version - 2.6.5

Ashwin Sadeep
  • 145
  • 1
  • 8

1 Answers1

1

First off, read through what I wrote here about the balancing of chunks to get an idea of how that actually works. TL;DR version is that no, the balancer will not care how full a shard is, it only cares about making the chunk counts on each shard the same.

To answer the headline question: when one shard runs out of space, inserts to the chunk ranges that live on that shard will fail. It will probably get into a bad state too, and might eventually crash (though MongoDB has gotten better at handling this type of problem).

Basically my advice is: don't let this happen - you can't truly predict what state the database might end up in when you run out of space, so it is better to just not get to that point than try to deal with it afterward (more on this later).

Regarding the moveChunk files: yes, it is safe to remove them once the shards are balanced.

Next, you should read up on how space is used/reused in MongoDB - when you sharded and documents started moving to the new shard, they were deleted on the old one. But that does not mean that space on disk was reclaimed, and that shard will continue to see new documents added usually (depending on your chosen shard key). Hence, even after you clean up you may still see data growth.

What I am getting at is that you may eventually run out of space, and you want to avoid that. You mention that you are running a single node which means that you will essentially need to take down time to reclaim your disk space. Therefore I see two possible paths:

  1. Convert the standalone to a replica set (see guide) and then rotate through the nodes reclaiming space (minimal down time for the conversion)
  2. Take the shard down and run a repair (lengthy down time, usually bounded by your disk speed)

I would take the first path. Once you have done the necessary work, you can even go back to a single node if you wish (single node replica set = no down time, standalone mongod = some down time to convert back from a replica set)

Community
  • 1
  • 1
Adam Comerford
  • 21,336
  • 4
  • 65
  • 85
  • Uptime is not that big of a deal for me. As long as the db is available for a 4 - 5 hour window a day, it's good enough. My concern was primarily about all that disk space MongoDB is not releasing to the OS. Your third link answers that. Any new documents will use the existing disk space which was allocated earlier, right? (I've already enabled the usePowerOf2Sizes flag) – Ashwin Sadeep Oct 25 '14 at 12:26
  • Assuming that the free space is suitable, it will be re-used (adding new docs that are significantly bigger would be an example of potential unsuitability). With a big enough free list you can hit timeouts too, with the database defaulting to new space, though that is far less likely with PowerOf2 sizing. I'd recommend a repair/resync at some point to clean things up, then this likely won't be a concern on that host again in the near future – Adam Comerford Oct 25 '14 at 15:27