Mongo console freezes in attempt to create a new index on a populated collection within a replica set

I want to add an index on an existing collection in a MongoDB Comunity Edition (mongo version v4.4.3) replica set. This replica set has only a Primary, at the moment, and is configured in this way to be able to use MongoDB change streams.
I log into the Mongo shell using:
mongo -u <Account>
where <Account> is an account with role ‘userAdminAnyDatabase’ and ‘ClusterManager’.
Then I issue the commands:

use dbOfInterest 
db.collectionOfInterest.createIndex({fieldOne:-1})

But after issuing the command above the console seems not responding anymore (after several minutes I had to stop it using CTRL+C). I also tried to create the index using Compass but the result is the same, GUI not responding. After killing Compass GUI and restarting it, I found the new index but is it created correctly?
The considered collection is a test one with very few documents.
What am I doing wrong?

I would expect this from the cli as is waiting for the index build to complete before returning the prompt to you, I cannot comment on the GUI.

The index build will have continued. The specific conditions of when the build fails are outlines in index
builds on populated collections
.

If the collection is busy the index build will yield for operations, but usually a small collection will index quickly. The index build is logged by mongod so you can see when it started, it’s progress and completion.

1 Like

Thanks Chris.

So, If I understand well:

  • there is no problem regarding a replica set with only one Primary
  • it is not necessary to stop replica/restart as stand-alone/build index/ restart as a replica (I highly hope this is not the case)
  • no issue related to user’s role

Nevertheless, I am very hesitating to start a new index building on a production server, as, as I said, the console does not respond and I have to terminate it. In this case, the index building goes on the same?

To monitor the index building process I have to use:

    db.adminCommand(
        {
          currentOp: true,
          $or: [
            { op: "command", "command.createIndexes": { $exists: true }  },
            { op: "none", "msg" : /^Index Build/ }
          ]
        }
    )

I am pretty sure it keeps building once you terminate the issuing client. Easy to check in the logs or your current_op query. You can use a terminal multiplexor like screen or tmux (assuming linux) on the server and leave it with confidence.

Prior to 4.2 an index build would lock the collection of the indexing unless run with the background option. 4.2+ an exclusive lock is only required at the beginning and end of the index build.

Running an Index Build on an active database/collection is going to prolong an index build and may have an impact on running operations.
From the previous link:

Index Build Impact on Database Performance

Index Builds During Write-Heavy Workloads

Building indexes during time periods where the target collection is under heavy write load can result in reduced write performance and longer index builds.

Consider designating a maintenance window during which applications stop or reduce write operations against the collection. Start the index build during this maintenance window to mitigate the potential negative impact of the build process.

I think there is some issue in creating indexes in my configuration.
I found this and following the provided instructions I added a ‘keyfile’ for replica set members internal authentication, but still, I’m not able to resolve the issue. Moreover, the command to monitor index creation can be executed only with a “root” account, ClusterAdmin role was not sufficient. I wasn’t able to create new indexes so far (tried on other collections).

Finally solved!!.
The issue is indeed related to “replica set node with auth can connect to itself” (more here).
I had to modify my mongod (file /etc/mongod.conf) configuration as follow:

# mongod.conf
# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

security:
  keyFile: /home/user1/Desktop/mongo.keyfile
  authorization: enabled
#operationProfiling:

replication:
  replSetName:  AirHeritage

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

Note keyFile section into YAML mongod.conf file (pay attention to white spaces).
To correctly generate this mongo.keyfile I used:

openssl rand -base64 756 > <path-to-keyfile>
chmod 400 <path-to-keyfile>

Then do:

sudo chown mongodb:mongodb /home/user1/Desktop/mongo.keyfile

Check that results are something as:

ls -al /home/user1/Desktop/mongo.keyfile                                                           
-r-------- 1 mongodb mongodb 1024 gen 13 09:19 /home/user1/Desktop/mongo.keyfile

Then stop and restart mongod using:

sudo systemctl stop mongod 
sudo systemctl start mongod 

Check status:

 sudo systemctl status mongod
● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-01-13 09:25:59 CET; 44min ago
       Docs: https://docs.mongodb.org/manual
   Main PID: 10156 (mongod)
     Memory: 348.6M
     CGroup: /system.slice/mongod.service
             └─10156 /usr/bin/mongod --config /etc/mongod.conf

gen 13 09:25:59 eanx-XPS-13-9350 systemd[1]: Started MongoDB Database Server.

Then log into mongo console as root (not sure it is necessary root, should suffice ClusterAdmin role):

mongo -u root
use dbOfInterest
db.collectionOfInterest.createIndex({field:1})

Having done all that before, index creation should have worked without hanging the console with a result as:

 db.collectionOfInterest.createIndex({'field':1})
{
        "createdCollectionAutomatically" : false,
        "numIndexesBefore" : 1,
        "numIndexesAfter" : 2,
        "commitQuorum" : "votingMembers",
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1610526522, 5),
                "signature" : {
                        "hash" : BinData(0,"qSqkVswHQA/IzWGYCd8HwNhXoQk="),
                        "keyId" : NumberLong("6877458498892857349")
                }
        },
        "operationTime" : Timestamp(1610526522, 5)
}

To check MongoDB logs use:

sudo tail /var/log/mongodb/mongod.log | jq

(if not installed in your system use sudo apt install jq, jq is very useful to pretty print json files)

Finally check indexes on collection with:

 db.collectionOfInterest.getIndexes()
[
        {
                "v" : 2,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_"
        },
        {
                "v" : 2,
                "key" : {
                        "field" : 1
                },
                "name" : "field.name_1"
        }

Note that two key fields are reported: “_id” (by default on collection creation) and “field”!!
Hope this can help someone else having a similar issue.
Only move the mongo.keyfile to a more suitable location (someone can suggest where?)

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.