Bulk insert failing with large number of duplicate values

Working with Data
1

!!Need a help.I have added 1000000 numbers and make ordered:false if it have duplicate value in every 1000 items less than 200 it work fine.But if more than 500 duplicates in 1000 it gave a error and stop the insert other values.

2

You can use Bulk.insert() with db.collection.initializeUnorderedBulkOp().

3

Yes i have done with this but its insert as 1000 bulk and if in 1000 ,800 duplicates then all process failed

4

@Buddika_Bandara Can you provide some more details to help investigate this:

  • The specific error message received.
  • The driver/client version you are using.
  • Your MongoDB server version.
  • Your deployment type (standalone, replica set, or sharded cluster).

A snippet of code showing how you are performing the bulk insert operation would also be useful.

Thanks,
Stennie

5

@Stennie this what i found on the error

error occur:- when addding 1000000 number to database with bulk insert
error- process stop with giving too man duplication value
finding:-as i think bulk insert happen with 1000 chunk.so if in 1000 chunk 800 duplicate all process stop without trying other 900000.But if duplicate value less than 100 the insert process happen smoothly

mongo ---- MongoDB shell version: 3.2.10
deployment — standalone i m using molecular js

6

this was the error

{ MongoError: E11000 duplicate key error collection: campaignDB.campaign_number index: _id_ dup key: { : "5e673dc3a53cf3730568a19b_94715987119" }
    at Function.create (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/core/error.js:44:12)
    at toError (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/utils.js:150:22)
    at UnorderedBulkOperation.handleWriteError (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/bulk/common.js:1125:11)
    at resultHandler (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/bulk/common.js:501:23)
    at handler (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/core/sdam/topology.js:973:24)
    at wireProtocol.(anonymous function) (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/core/sdam/server.js:437:5)
    at /Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/core/connection/pool.js:420:18
    at process._tickCallback (internal/process/next_tick.js:61:11)
  name: 'BulkWriteError',
  driver: true,
  code: 11000,
  writeErrors:
   [ WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     WriteError { err: [Object] },
     ... 899 more items ],
  result:
   BulkWriteResult {
     result:
      { ok: 1,
        writeErrors: [Array],
        writeConcernErrors: [],
        insertedIds: [Array],
        nInserted: 0,
        nUpserted: 0,
        nMatched: 0,
        nModified: 0,
        nRemoved: 0,
        upserted: [] } },
  [Symbol(mongoErrorContextSymbol)]: {} }

** :roll_eyes: :roll_eyes: :roll_eyes:**