1

I accidentally deleted a volume of docker mongo-data:/data/db , i have a copy of that folder , now the problem is when i run docker-compose up mongodb container doesn't start and gives an error of mongo_1 exited with code 14 below more details of the error and the mongo-data folder , can you someone help me please

in docker-compose.yml

volumes: - ./mongo-data:/data/db

enter image description here

enter image description here

enter image description here

ABDELJALIL AIT ETALEB
  • 146
  • 2
  • 9
  • check file permission, file owner and check if the files are corrupted. – jordanvrtanoski Feb 25 '21 at 12:29
  • @jordanvrtanoski i have checked the permission and owner all fine , how can i fix corrupted files? – ABDELJALIL AIT ETALEB Feb 25 '21 at 14:27
  • What you can try to run a container with `mongod --repair` on top of the data folder and if this works, run the server container over the data. !Keep a copy of the data befre you do anything! Read [this](http://man.hubwiz.com/docset/MongoDB.docset/Contents/Resources/Documents/docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/index.html) before proceeding for more details on the options you have for the recovery. – jordanvrtanoski Feb 25 '21 at 14:31

1 Answers1

2

Restore from backup files

A step-by-step process to repair the corrupted files from a failed mongodb in a docker container:

  1. ! Before you start, make copy of the files. !

  2. Make sure you know which version of the image was running in the container

  3. Spawn new container with to run the repair process as follows

    docker run -it -v <data folder>:/data/db <image-name>:<image-version> mongod --repair

  4. Once the files are repaired, you can start the containers from the docker-compose

If the repair fails, it usually means that the files are corrupted beyond repair. There is still a chance to repair it with exporting the data as described here.

How to secure proper backup files

The database is constantly working with the files, so the files are constantly changed on the disks. In addition, the database will keep some of the changes in the internal memory buffers before they are flushed to the filesystem. Although the database engines are doing very good job to assure the the database can recover from abrupt failure by using the 2-stage commit process (first update the transaction-log than the datafile), when the files are copied there could be a corruption that will prevent the database from recovery.

Reason for such corruption is that the copy process is not aware of the database written process progress, and this creates a racing condition. With very simple words, while the database is in middle of writing, the copy process will create a copy of the file(s) that is half-updated, hence it will be corrupted.

When the database writer is in middle of writing to the files, we call them hot files. hot files are term from the OS perspective, and MongoDB also uses a term hot backup which is a term from MongoDB perspective. Hot backup means that the backup was taken when the database was running.

To take a proper snapshot (assuring the files are cold) you need to follow the procedure explained here. In short, the command db.fsyncLock() that is issued during this process will inform the database engine to flush all buffers and stop writing to the files. This will make the files cold, however the database remains hot, hence the difference between the terms hot files and hot backup. Once the copy is done, the database is informed to start writing to the filesystem by issuing db.fsyncUnlock()

Note the process is more complex and can change with different version of the databse. Here I give a simplification of it, in order to illustrate the point about the problems with the file snapshot. To secure proper and consistent backup, always follow the documented procedure for the database version that you use.

Suggested backup method

Preferred backup should always be the data dump method, since this assures that you can restore even in case of upgraded/downgraded database engines. MongoDB provides very useful tool called mongodump that can be used to create database backups by dumping the data, instead by copy of the files.

For more details on how to use the backup tools, as well as for the other methods of backup read the MongoDB Backup Methods chapter of the MondoDB documentation.

jordanvrtanoski
  • 5,104
  • 1
  • 20
  • 29
  • hello thanks for help it did not work for me , i managed to reciver old folder , when i start the container with docker-compose it works for some time and again give a message exited with code 14 , no idea what's the issue – ABDELJALIL AIT ETALEB Feb 26 '21 at 21:41
  • and the command above give this message "msg":"Validation complete for collection. No corruption found" – ABDELJALIL AIT ETALEB Feb 26 '21 at 21:54
  • Error 14 is related to the data storage library "WiredTiger" and is mainly associated with the state of the database files which can be: 1) permissions, 2) corruption, 3) time syncronization, 4) in case of macOS, the VM mechanism to share fils from macOS. Troubleshooting can be time consuming, so I suggest, if you manage to start the database with a stand-alone container after a repair, export the data and import it in another clean instance. – jordanvrtanoski Feb 27 '21 at 07:35
  • 1
    am running it in linux CentOS 8 , made sure permissions are the same , no sign for corruption , but time syncronization no clue how to check that , i ended up reinserting the data allover again hopping that never happens again but , i need to know what's the issue , for the future issues like this , i have to fix the existing database folder (volume) – ABDELJALIL AIT ETALEB Mar 04 '21 at 21:18
  • The issue is that you had copy of the data while the datafiles ware `hot`, meaning there ware still buffers in memory that ware not flushed to the datafiles. To use the copying of files as backup strategy, you need to make sure that the files are `cold`. – jordanvrtanoski Mar 05 '21 at 05:10
  • 1
    what you said is exectly what i did and caused this issue, (copying hot files) , now i created a script that uses mongodump command to create the backup , and will read more about fsyncLock and fsyncUnLock thank you – ABDELJALIL AIT ETALEB Mar 05 '21 at 15:15