How to avoid "duplicate id" error during restoring collections using mongorestore

[venkata_reddy]

When restoring collections on destination database mongorestore is giving “duplicate id” error.
Should we treat this as serious Error or just ignore it as Warning or Informational message??
Is it due to the fact that the collections already exist.
In what other cases this error will creep up.

[Prasad_Saya]

Hello @venkata_reddy, welcome to the MongoDB Community. I will explain the “error” with an example.

Lets take an example collection movie with two documents:

{ _id: 1, title: "star wars" }
{ _id: 2, title: "return of jedi" }

Do a mongodump of this collection and restore it to a new collection movie_new.

os > mongodump  --db=test --collection=movie
os > mongorestore --db=test --collection=movie_new dump/test/movie.bson

Now, the movie_new has two documents with _id values 1 and 2.

Insert one more document into the movie collection.

{ _id: 9, title: "the empire strikes back" }

Again, do a mongodump and mongorestore from movie to movie_new. You will see the mongorestore completes with inserting only the new document with _id: 9 into the movie_new collection.

During the restore process you will see messages on the console like this:

2020-07-03T07:38:08.519+0530    checking for collection data in dump\test\movie.bson
2020-07-03T07:38:08.527+0530    restoring to existing collection test.movie_new without dropping
2020-07-03T07:38:08.532+0530    reading metadata for test.movie_new from dump\test\movie.metadata.json
2020-07-03T07:38:08.537+0530    restoring test.movie_new from dump\test\movie.bson
2020-07-03T07:38:08.627+0530    continuing through error: E11000 duplicate key error collection: test.movie_new index: _id_
 dup key: { _id: 1 }
2020-07-03T07:38:08.631+0530    continuing through error: E11000 duplicate key error collection: test.movie_new index: _id_
 dup key: { _id: 2 }
2020-07-03T07:38:08.638+0530    restoring indexes for collection test.movie_new from metadata
2020-07-03T07:38:08.646+0530    finished restoring test.movie_new (1 document, 2 failures)
2020-07-03T07:38:08.648+0530    1 document(s) restored successfully. 2 document(s) failed to restore.

In the above output, the error 2020-07-03T07:38:08.627+0530 continuing through error: E11000 duplicate key error collection: test.movie_new index: _id_ dup key: { _id: 1 } is expected, as there is already a document in the target collection with the same _id that is being restored; and the document doesn’t get inserted (see below note on inserts). The restore process continues to process remaining document(s) after logging this “error” message - it does not abort the process.

The process completes inserting only one document into movie_new collection.

From the MongoDB documentation: mongorestore Inserts Only

mongorestore can create a new database or add data to an existing database. However, mongorestore performs inserts only and does not perform updates. That is, if restoring documents to an existing database and collection and existing documents have the same value _id field as the to-be-restored documents, mongorestore will not overwrite those documents.

[venkata_reddy]

Hi @Prasad_Saya, Thank you for the useful information. This has clarified my confusion.