I had a node application running on EC2 managed by Elastic Beanstalk. The elastic beanstalk removed an instance and recreated new one due to some issue.

I had mongo store its db in a separate Elastic Block Store volume, and did re-attach the volume, mounted etc..

However when I tried to start mongodb using systemctl, I got various errors. I tried --repair, chown the data directory to mongod and it finally worked, but now the user db was gone and the application re-created it and all collections are empty, but I do see large collection-x-xxxxxxx.wt and index-x-xxxxx.wt files in the data directory.

What am I doing wrong ? Is there any way to recover the data.

PS: I did try the --repair before I saw the warning about how it would remove all corrupted data

CodePudding user response：

I was able to restore from the collection-x-xxxxx.wt files....

In my 'corrupted' mongo data directory, there was a WiredTiger.wt.orig file and WiredTiger.wt file.

If I try to start mongod by removing the 'orig' extension, mongod will not start and start showing errors like WiredTiger.wt read error: failed to read 4096 bytes at offset 73728: WT_ERROR: non-specific WiredTiger error.

Searching for restoring a corrupted 'WiredTiger' file, I came across this medium article about repairing MongoDB after a corrupted WiredTiger file.

Steps from article as I followed them :

1. Stop mongod. (the one with empty collections)

2. Point mongod to a new data directory

3. Start mongod and create new db, and new collections with the same names as the ones from corrupted mongo db.

4. Insert atleast one dummy record into each of these collection.

5. Find the names of the collection*.wt files in this new location using db.<insert-collectionName>.stats(), look in the uri property of the output.

6. Stop mongod.

7. Copy over collection-x-xxxxx.wt from corrupted directory to the new directory and rename them to the corresponding ones from step 5.

   7.1. i.e. Say, if your collection named 'testCollection' had the wt collection file name as collection-1-1111111.wt in corrupted directory and the name as collection-6.6666666.wt in new directory, you will have to copy the 'collection-1-1111111.wt' into the new directory and rename it to collection-6.6666666.wt

   7.2. To find the collection wt file name of say 'testCollection', you can open the collection-x.xxxx.wt files in a text editor and scroll past the 'gibberish' to see your actual data matching the ones from 'testCollection'. (mine is not encrypted at rest).

8. Repeat the copy - rename step for all collections you have.

9. Run repair in new db path with --repair switch, you can see mongo fixing stuff in logs.

10. Start db.

11. Once done, verify the collections, mongodump from new db and mongorestore to fresh db and recreate indexes.

That article was a god sent, I cant believe it worked. Thank you Ido Ozeri from Medium