HW-2.4, step by step checking. rs.add() keeps failing

Here is a detailed report of all my actions when trying to do the HW-2.4.
If someone can follow and tell me where I am doing something wrong that will be very helpful. I am not using any config file, doing it all with command line options.

I have also added some ls and ps commands on the way so one can be aware of the state of things at various moments.

vagrant@database:~$ 
vagrant@database:~$ ps -ex
  PID TTY      STAT   TIME COMMAND
 7809 ?        S      0:00 sshd: vagrant@pts/0 
 7810 pts/0    Ss+    0:00 -bash LC_CTYPE=UTF-8 USER=vagrant LOGNAME=vagrant HOME=/home/vagrant PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
 8302 ?        S      0:00 sshd: vagrant@pts/1 
 8303 pts/1    Ss     0:00 -bash LC_CTYPE=UTF-8 USER=vagrant LOGNAME=vagrant HOME=/home/vagrant PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
 8539 pts/1    R+     0:00 ps -ex XDG_SESSION_ID=5 TERM=xterm-256color SHELL=/bin/bash SSH_CLIENT=10.0.2.2 50077 22 SSH_TTY=/dev/pts/1 LC_ALL=C USER=vag
vagrant@database:~$ ls M310-HW-2.4/*
M310-HW-2.4/r0:

M310-HW-2.4/r1:

M310-HW-2.4/r2:
vagrant@database:~$ 
vagrant@database:~$ ls -l smds 
-rwxrw-r-- 1 vagrant vagrant 218 Dec  2 03:26 smds
vagrant@database:~$ 
vagrant@database:~$ cat smds 
#!/bin/bash

mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r0 --logpath M310-HW-2.4/r0/mongodb.log --port 31240 --auth --fork --sslMode requireSSL --sslPEMKeyFile shared/certs/server.pem --sslCAFile shared/certs/ca.pem
vagrant@database:~$ ./smds
about to fork child process, waiting until server is ready for connections.
forked process: 8553
child process started successfully, parent exiting
vagrant@database:~$ 
vagrant@database:~$ ps -ex
  PID TTY      STAT   TIME COMMAND
 7809 ?        S      0:00 sshd: vagrant@pts/0 
 7810 pts/0    Ss+    0:00 -bash LC_CTYPE=UTF-8 USER=vagrant LOGNAME=vagrant HOME=/home/vagrant PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
 8302 ?        S      0:00 sshd: vagrant@pts/1 
 8303 pts/1    Ss     0:00 -bash LC_CTYPE=UTF-8 USER=vagrant LOGNAME=vagrant HOME=/home/vagrant PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
 8553 ?        Rl     0:00 mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r0 --logpath M310-HW-2.4/r0/mongodb.log --port 31240 --auth --fork --sslMode req
 8578 pts/1    R+     0:00 ps -ex XDG_SESSION_ID=5 TERM=xterm-256color SHELL=/bin/bash SSH_CLIENT=10.0.2.2 50077 22 SSH_TTY=/dev/pts/1 LC_ALL=C USER=vag
vagrant@database:~$ ls M310-HW-2.4/*
M310-HW-2.4/r0:
WiredTiger         WiredTiger.wt     collection-0--2494074014422010889.wt  index-1--2494074014422010889.wt  mongod.lock    storage.bson
WiredTiger.lock    WiredTigerLAS.wt  collection-2--2494074014422010889.wt  index-3--2494074014422010889.wt  mongodb.log
WiredTiger.turtle  _mdb_catalog.wt   diagnostic.data                       journal                          sizeStorer.wt

M310-HW-2.4/r1:

M310-HW-2.4/r2:
vagrant@database:~$ 
vagrant@database:~$ sed "s/r0/r1/g;s/40/41/" smds | sh
about to fork child process, waiting until server is ready for connections.
forked process: 8584
child process started successfully, parent exiting
vagrant@database:~$ 
vagrant@database:~$ ps -ex
  PID TTY      STAT   TIME COMMAND
 7809 ?        S      0:00 sshd: vagrant@pts/0 
 7810 pts/0    Ss+    0:00 -bash LC_CTYPE=UTF-8 USER=vagrant LOGNAME=vagrant HOME=/home/vagrant PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
 8302 ?        S      0:00 sshd: vagrant@pts/1 
 8303 pts/1    Ss     0:00 -bash LC_CTYPE=UTF-8 USER=vagrant LOGNAME=vagrant HOME=/home/vagrant PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
 8553 ?        Rl     0:01 mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r0 --logpath M310-HW-2.4/r0/mongodb.log --port 31240 --auth --fork --sslMode req
 8584 ?        Rl     0:00 mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r1 --logpath M310-HW-2.4/r1/mongodb.log --port 31241 --auth --fork --sslMode req
 8610 pts/1    R+     0:00 ps -ex XDG_SESSION_ID=5 TERM=xterm-256color SHELL=/bin/bash SSH_CLIENT=10.0.2.2 50077 22 SSH_TTY=/dev/pts/1 LC_ALL=C USER=vag
vagrant@database:~$ ls M310-HW-2.4/*
M310-HW-2.4/r0:
WiredTiger         WiredTiger.wt     collection-0--2494074014422010889.wt  index-1--2494074014422010889.wt  mongod.lock    storage.bson
WiredTiger.lock    WiredTigerLAS.wt  collection-2--2494074014422010889.wt  index-3--2494074014422010889.wt  mongodb.log
WiredTiger.turtle  _mdb_catalog.wt   diagnostic.data                       journal                          sizeStorer.wt

M310-HW-2.4/r1:
WiredTiger         WiredTiger.wt     collection-0--9213632957865206374.wt  index-1--9213632957865206374.wt  mongod.lock    storage.bson
WiredTiger.lock    WiredTigerLAS.wt  collection-2--9213632957865206374.wt  index-3--9213632957865206374.wt  mongodb.log
WiredTiger.turtle  _mdb_catalog.wt   diagnostic.data                       journal                          sizeStorer.wt

M310-HW-2.4/r2:
vagrant@database:~$ 
vagrant@database:~$ sed "s/r0/r2/g;s/40/42/" smds | sh
about to fork child process, waiting until server is ready for connections.
forked process: 8616
child process started successfully, parent exiting
vagrant@database:~$ 
vagrant@database:~$ ps -ex
  PID TTY      STAT   TIME COMMAND
 7809 ?        S      0:00 sshd: vagrant@pts/0 
 7810 pts/0    Ss+    0:00 -bash LC_CTYPE=UTF-8 USER=vagrant LOGNAME=vagrant HOME=/home/vagrant PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
 8302 ?        S      0:00 sshd: vagrant@pts/1 
 8303 pts/1    Ss     0:00 -bash LC_CTYPE=UTF-8 USER=vagrant LOGNAME=vagrant HOME=/home/vagrant PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
 8553 ?        Sl     0:02 mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r0 --logpath M310-HW-2.4/r0/mongodb.log --port 31240 --auth --fork --sslMode req
 8584 ?        Sl     0:00 mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r1 --logpath M310-HW-2.4/r1/mongodb.log --port 31241 --auth --fork --sslMode req
 8616 ?        Sl     0:00 mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r2 --logpath M310-HW-2.4/r2/mongodb.log --port 31242 --auth --fork --sslMode req
 8641 pts/1    R+     0:00 ps -ex XDG_SESSION_ID=5 TERM=xterm-256color SHELL=/bin/bash SSH_CLIENT=10.0.2.2 50077 22 SSH_TTY=/dev/pts/1 LC_ALL=C USER=vag
vagrant@database:~$ ls M310-HW-2.4/*
M310-HW-2.4/r0:
WiredTiger         WiredTiger.wt     collection-0--2494074014422010889.wt  index-1--2494074014422010889.wt  mongod.lock    storage.bson
WiredTiger.lock    WiredTigerLAS.wt  collection-2--2494074014422010889.wt  index-3--2494074014422010889.wt  mongodb.log
WiredTiger.turtle  _mdb_catalog.wt   diagnostic.data                       journal                          sizeStorer.wt

M310-HW-2.4/r1:
WiredTiger         WiredTiger.wt     collection-0--9213632957865206374.wt  index-1--9213632957865206374.wt  mongod.lock    storage.bson
WiredTiger.lock    WiredTigerLAS.wt  collection-2--9213632957865206374.wt  index-3--9213632957865206374.wt  mongodb.log
WiredTiger.turtle  _mdb_catalog.wt   diagnostic.data                       journal                          sizeStorer.wt

M310-HW-2.4/r2:
WiredTiger         WiredTiger.wt     collection-0--4494793726918118218.wt  index-1--4494793726918118218.wt  mongod.lock    storage.bson
WiredTiger.lock    WiredTigerLAS.wt  collection-2--4494793726918118218.wt  index-3--4494793726918118218.wt  mongodb.log
WiredTiger.turtle  _mdb_catalog.wt   diagnostic.data                       journal                          sizeStorer.wt
vagrant@database:~$ 
vagrant@database:~$ mongo --host database.m310.mongodb.university --ssl --sslPEMKeyFile shared/certs/client.pem --sslCAFile shared/certs/ca.pem --port 31240
MongoDB shell version: 3.2.22
connecting to: database.m310.mongodb.university:31240/test
MongoDB Enterprise > 
MongoDB Enterprise > use admin
switched to db admin
MongoDB Enterprise > rs.initiate({_id: 'HW-2.4',
...              members: [
...               {_id:1, host: 'database.m310.mongodb.university:31240'}
...              ]
...             })
{ "ok" : 1 }
MongoDB Enterprise HW-2.4:OTHER> 
MongoDB Enterprise HW-2.4:PRIMARY> db.createUser({
...   user: "m103-admin",
...   pwd: "m103-pass",
...   roles: [
...     {role: "root", db: "admin"}
...   ]
... })
Successfully added user: {
	"user" : "m103-admin",
	"roles" : [
		{
			"role" : "root",
			"db" : "admin"
		}
	]
}
MongoDB Enterprise HW-2.4:PRIMARY> db.auth({user:"m103-admin",pwd:"m103-pass"})
1
MongoDB Enterprise HW-2.4:PRIMARY> rs.isMaster()
{
	"hosts" : [
		"database.m310.mongodb.university:31240"
	],
	"setName" : "HW-2.4",
	"setVersion" : 1,
	"ismaster" : true,
	"secondary" : false,
	"primary" : "database.m310.mongodb.university:31240",
	"me" : "database.m310.mongodb.university:31240",
	"electionId" : ObjectId("7fffffff0000000000000001"),
	"maxBsonObjectSize" : 16777216,
	"maxMessageSizeBytes" : 48000000,
	"maxWriteBatchSize" : 1000,
	"localTime" : ISODate("2020-12-02T04:42:21.031Z"),
	"maxWireVersion" : 4,
	"minWireVersion" : 0,
	"ok" : 1
}
MongoDB Enterprise HW-2.4:PRIMARY> 
MongoDB Enterprise HW-2.4:PRIMARY> rs.add('database.m310.mongodb.university:31241')
{
	"ok" : 0,
	"errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: database.m310.mongodb.university:31240; the following nodes did not respond affirmatively: database.m310.mongodb.university:31241 failed with not authorized on admin to execute command { replSetHeartbeat: \"HW-2.4\", pv: 1, v: 2, from: \"database.m310.mongodb.university:31240\", fromId: 1, checkEmpty: false }",
	"code" : 74
}
MongoDB Enterprise HW-2.4:PRIMARY> exit
bye
vagrant@database:~$ 
vagrant@database:~$ tail M310-HW-2.4/r0/mongodb.log 
2020-12-02T04:35:55.476+0000 I REPL     [ReplicationExecutor] transition to PRIMARY
2020-12-02T04:35:56.470+0000 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2020-12-02T04:42:10.041+0000 I ACCESS   [conn1] Successfully authenticated as principal m103-admin on admin
2020-12-02T04:42:43.292+0000 I REPL     [conn1] replSetReconfig admin command received from client
2020-12-02T04:42:43.298+0000 I REPL     [conn1] replSetReconfig config object with 2 members parses ok
2020-12-02T04:42:43.298+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to database.m310.mongodb.university:31241
2020-12-02T04:42:43.303+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to database.m310.mongodb.university:31241, took 5ms (1 connections now open to database.m310.mongodb.university:31241)
2020-12-02T04:42:43.304+0000 W REPL     [ReplicationExecutor] Got error (Unauthorized: not authorized on admin to execute command { replSetHeartbeat: "HW-2.4", pv: 1, v: 2, from: "database.m310.mongodb.university:31240", fromId: 1, checkEmpty: false }) response on heartbeat request to database.m310.mongodb.university:31241; { ok: 1.0, hbmsg: "" }
2020-12-02T04:42:43.304+0000 E REPL     [conn1] replSetReconfig failed; NodeNotFound: Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: database.m310.mongodb.university:31240; the following nodes did not respond affirmatively: database.m310.mongodb.university:31241 failed with not authorized on admin to execute command { replSetHeartbeat: "HW-2.4", pv: 1, v: 2, from: "database.m310.mongodb.university:31240", fromId: 1, checkEmpty: false }
2020-12-02T04:43:13.079+0000 I NETWORK  [conn1] end connection 127.0.0.1:49828 (0 connections now open)
vagrant@database:~$

A few recommendations:

  1. Do not use tilde in your configuration files. The issue is if your switch users the location will not be the same. In some cases that is what you want. But it is easier to diagnose issues with full path name.

  2. Bravo for your use of sed to parameterize your script. However real shell parameters are there for this. And it is easier to follow and less error prone with parameters.

  3. You can limit the output of ps to your mongod using ps -aef | grep [m]ongod. This reduce the amount of text we have to scan.

I am mixing your many different threads as point 1 above is related to your other thread.

In you ps output, I do not see the certificates. I see that you specify it in your script but since it is no on the ps output I am not too sure if it is there or not. That is why I recommend to use configuration files. You shown the log lines of the :31240, but the one from :31241 are also of interest. So far your problem points to a certificate issue.

Yes, well, as you know with my other thread; when I try to use a configuration file, I can’t even start a mongod.
At least here I start something, even if there is a certificate issue (that I will look into now) as you say.

If you cannot start with configuration files then may be that’s the real issue. Looks like a file permission issue that might be better to fix first.

OK. What kind of permission do you think I might be missing?
I have put a detailed report of what is happening there too.

I replied to your other thread.

OK, thanks. I will look.

In the meanwhile, concerning this thread; I did the experiment again and this is the ps output you want. It seems to me that the certificates are there, but please have a look:

vagrant@database:~$ ps -aef | grep [m]ongod
vagrant   7865     1  1 14:50 ?        00:00:02 mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r0 --logpath M310-HW-2.4/r0/mongodb.log --port 31240 --auth --fork --sslMode requireSSL --sslPEMKeyFile shared/certs/server.pem --sslCAFile shared/certs/ca.pem
vagrant   7892     1  1 14:50 ?        00:00:02 mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r1 --logpath M310-HW-2.4/r1/mongodb.log --port 31241 --auth --fork --sslMode requireSSL --sslPEMKeyFile shared/certs/server.pem --sslCAFile shared/certs/ca.pem
vagrant   7919     1  1 14:50 ?        00:00:02 mongod --replSet HW-2.4 --dbpath M310-HW-2.4/r2 --logpath M310-HW-2.4/r2/mongodb.log --port 31242 --auth --fork --sslMode requireSSL --sslPEMKeyFile shared/certs/server.pem --sslCAFile shared/certs/ca.pem
vagrant@database:~$

I think it is the same issue where you do not bind to the fully qualified domain name of the certificates.

The new option –tlsAllowInvalidHostnames might be useful. But using the same host names is much better.

This is the retrying of rs.add(...), but this time with the logs on both port (31240[r0] & 31241[r1]) as you wanted.

MongoDB Enterprise HW-2.4:PRIMARY> rs.add('database.m310.mongodb.university:31241')
{
	"ok" : 0,
	"errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: database.m310.mongodb.university:31240; the following nodes did not respond affirmatively: database.m310.mongodb.university:31241 failed with not authorized on admin to execute command { replSetHeartbeat: \"HW-2.4\", pv: 1, v: 2, from: \"database.m310.mongodb.university:31240\", fromId: 1, checkEmpty: false }",
	"code" : 74
}
MongoDB Enterprise HW-2.4:PRIMARY> exit
bye
vagrant@database:~$ 
vagrant@database:~$ tail M310-HW-2.4/r0/mongodb.log
2020-12-02T15:01:42.811+0000 I REPL     [ReplicationExecutor] transition to PRIMARY
2020-12-02T15:01:43.804+0000 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2020-12-02T15:03:03.043+0000 I ACCESS   [conn1] Successfully authenticated as principal m103-admin on admin
2020-12-02T15:04:17.278+0000 I REPL     [conn1] replSetReconfig admin command received from client
2020-12-02T15:04:17.285+0000 I REPL     [conn1] replSetReconfig config object with 2 members parses ok
2020-12-02T15:04:17.285+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to database.m310.mongodb.university:31241
2020-12-02T15:04:17.290+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to database.m310.mongodb.university:31241, took 5ms (1 connections now open to database.m310.mongodb.university:31241)
2020-12-02T15:04:17.290+0000 W REPL     [ReplicationExecutor] Got error (Unauthorized: not authorized on admin to execute command { replSetHeartbeat: "HW-2.4", pv: 1, v: 2, from: "database.m310.mongodb.university:31240", fromId: 1, checkEmpty: false }) response on heartbeat request to database.m310.mongodb.university:31241; { ok: 1.0, hbmsg: "" }
2020-12-02T15:04:17.290+0000 E REPL     [conn1] replSetReconfig failed; NodeNotFound: Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: database.m310.mongodb.university:31240; the following nodes did not respond affirmatively: database.m310.mongodb.university:31241 failed with not authorized on admin to execute command { replSetHeartbeat: "HW-2.4", pv: 1, v: 2, from: "database.m310.mongodb.university:31240", fromId: 1, checkEmpty: false }
2020-12-02T15:05:02.496+0000 I NETWORK  [conn1] end connection 127.0.0.1:46636 (0 connections now open)
vagrant@database:~$ 
vagrant@database:~$ tail M310-HW-2.4/r1/mongodb.log
2020-12-02T14:50:58.080+0000 I REPL     [initandlisten] Did not find local voted for document at startup.
2020-12-02T14:50:58.080+0000 I REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2020-12-02T14:50:58.080+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/home/vagrant/M310-HW-2.4/r1/diagnostic.data'
2020-12-02T14:50:58.080+0000 I NETWORK  [initandlisten] waiting for connections on port 31241 ssl
2020-12-02T14:50:58.080+0000 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2020-12-02T15:04:17.279+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:49384 #1 (1 connection now open)
2020-12-02T15:04:17.285+0000 I ACCESS   [conn1] note: no users configured in admin.system.users, allowing localhost access
2020-12-02T15:04:17.285+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:49385 #2 (2 connections now open)
2020-12-02T15:04:17.285+0000 I NETWORK  [conn1] end connection 127.0.0.1:49384 (1 connection now open)
2020-12-02T15:04:17.290+0000 I ACCESS   [conn2] Unauthorized: not authorized on admin to execute command { replSetHeartbeat: "HW-2.4", pv: 1, v: 2, from: "database.m310.mongodb.university:31240", fromId: 1, checkEmpty: false }
vagrant@database:~$

The above seems to indicate that you are still not binding to the fully qualified domain name as specified in the certificate file.

the same issue where you do not bind to the fully qualified domain name of the certificates

What do you mean by that?

But using the same host names is much better.

I am sorry, but I don’t quite see what this means either. When you write “same”, you mean same as what?

You do rs.add with the above. That is the name specified in the certificates. It is the name you have to bind with and connect with.

OK, I think I may need to review some lectures, because I am not following here.
This “fully qualified domain” reminds me of LDAP, but this is probably different.

OK. When I run this to launch mongo shell:

mongo --host database.m310.mongodb.university --ssl --sslPEMKeyFile shared/certs/client.pem --sslCAFile shared/certs/ca.pem --port 31240

Isn’t this what I am doing ?
Or is something missing ? Or are you talking about something else ?

In fact at some point I also tried this:

mongo --host ‘database.m310.mongodb.university:31240’ --ssl --sslPEMKeyFile shared/certs/client.pem --sslCAFile shared/certs/ca.pem

The command mongo starts the client. That’s the connecting part. And this one looks ok.

The command mongod starts the server. That’s the listening part. You have to listen using database.m310.mongodb.university.

Another thing I have just notice.

Since you are connecting to a replica set, you should be using ReplicaSetName/Hostname as the host part when connecting.

OK. That’s clear.

So this means ‘database.m310.mongodb.university’ should appear somewhere as an option when I launch mongod. Or in the config file if I am using one.

I did my best to use the form I saw in the lecture for the mongod command, but I may have missed that.

Yes.

I know. It shows. Don’t worry. Security is tricky. Every details must be nailed in correctly.

So that would make it:

mongo --host HW-2.4/database.m310.mongodb.university --ssl --sslPEMKeyFile shared/certs/client.pem --sslCAFile shared/certs/ca.pem --port 31240

or:

mongo --host 'HW-2.4/database.m310.mongodb.university:31240' --ssl --sslPEMKeyFile shared/certs/client.pem --sslCAFile shared/certs/ca.pem

I will try that.

Yes it should. On the client side. The mongod must use the same host name for bindIp.