I have a CouchDB cluster with 9 shards running on 3 nodes (and n=3). I brought down the three instances the nodes were running on and created 3 new instances with new internal IP addresses. After this I was getting "Database failed to load" errors for the databases.
I started to read about moving shards and I am aware of the change when going from 2.1.0 to 2.1.1 (as answered in this question).
While trying to solve this problem I modified a copy of the data from one of the nodes to try and adjust the _dbs metadata. However, I think I only made a mess of the metadata.
I am thinking that in order to fix this problem I can:
Change the IP address of one of the nodes to match an old IP address.
Start the cluster on just this instance with an original copy of the data.
Tell CouchDB to delete the _membership info about the two non-functioning nodes from the cluster.
Add two new nodes to the cluster with new IP addresses.
... and then maybe the cluster will sync the shards from the one node to all three nodes and everything will work?
Still trying to figure out how CouchDB 2.1.1 clustering works so any information beyond that CouchDB docs is appreciated.