`clusterAuthMode: x509` fails when `authorization: enabled`

Hi,
When I enable clusterAuthMode: x509 for replica set authentication, with the proper certs in place per this documentation, the replica members can authenticate with each other just fine. However, when I also enable authorization: enabled, the members are unable to communicate with each other. In the logs I see entries like this:

authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=mongod1", $db: "$external" }
2020-09-11T19:15:50.644+0000 I  ACCESS   [conn10] Failed to authenticate CN=mongod1@$external from client 172.19.62.122:37102 with mechanism MONGODB-X509: UserNotFound: Could not find user "CN=mongod1" for db "$external"

This would suggest that a corresponding user needs to be in place for each member of the replica set. In the documentation, it’s clear that a user is needed for clients to use when connecting to the replica set and I do see log entries indicating that my client is successfully authenticating using the user I’ve setup, but I can’t find anything in the docs showing that such a user is needed for internal/membership authorization.

Is such a user needed for each member of the replica set for internal/membership authorization, or am I missing something else here? If a user for each member is needed, what db should the user be created in, and with what permissions/roles; where are the docs I’ve overlooked that specify this?

Thanks for any help.

Hi @DanielP and welcome in the MongoDB Community :muscle: !

Are the cluster IP addresses correctly setup in the bind_ip list?
There are definitely no “internal users” for internal membership authorization, etc. Users are just for real users. Not for internal purposes.

Can you please share your config files & command lines and walk us through the steps you have been through so we can maybe spot the issue?

Cheers,
Maxime.

Appreciate your response!

Here is my conf for the 2 data bearing members:

net:
port: 17017
bindIpAll: true
tls:
mode: requireTLS
CAFile: /certs/myCA.pem
certificateKeyFile: /certs/myCert.pem
replication:
oplogSizeMB: 4096
replSetName: mySet
enableMajorityReadConcern: false
setParameter:
enableLocalhostAuthBypass: true
processManagement:
fork: “false”
storage:
dbPath: /data/db
engine: wiredTiger
journal:
enabled: true
wiredTiger:
collectionConfig:
blockCompressor: snappy
engineConfig:
directoryForIndexes: true
cacheSizeGB: 20
security:
enableEncryption: true
encryptionKeyFile: /etc/mongodb.keyfile
authorization: enabled
clusterAuthMode: x509

And here is my arb conf:

net:
port: 17017
bindIpAll: true
tls:
mode: requireTLS
CAFile: /certs/myCA.pem
certificateKeyFile: /certs/myCert.pem
replication:
oplogSizeMB: 1
replSetName: mySet
enableMajorityReadConcern: false
setParameter:
enableLocalhostAuthBypass: true
processManagement:
fork: “false”
storage:
dbPath: /data/db
engine: wiredTiger
wiredTiger:
collectionConfig:
blockCompressor: snappy
engineConfig:
directoryForIndexes: true
security:
enableEncryption: true
encryptionKeyFile: /etc/mongodb.keyfile
authorization: enabled
clusterAuthMode: x509

When I comment the two lines in the security section, the replica set members can communicate just fine, so it seems to be an issue relating to membership authentication.

Thank you for confirming there are no users needed for internal membership authentication; that was my initial assumption but I’ve been grinding on this problem for several days now so I didn’t want to rule anything out, particularly since the error messages I’m seeing in the logs seem to indicate it’s in fact expecting a particular user named after the member that’s trying to connect.

When you comment security section members are able to communicate means some issue with certificates

From mongo documentation

“…Certificat Requirements: The Distinguished Name (DN), found in the member certificate’s subject, must specify a non-empty value for at least one of the following attributes: Organization (O), the Organizational Unit (OU) or the Domain Component (DC).”

1 Like

I’ve double-checked the certs of the three members; all three have the exact same subject, except for the value of CN; e.g.

$ openssl x509 -in myCert.pem -inform PEM -subject -nameopt RFC2253
subject= emailAddress=dev-team@mycompany.org,CN=mongod1,OU=ops,O=myCompany,L=myCity,ST=myState,C=US

I’ve gone over the documentation you’re referencing several times and I’m meeting all the requirements, namely:

  • All certificates were created from the same CA
  • All certificates contain a non-empty value for at least one of the following: O, OU, or DC
  • All certificates have the exact same DN (excepting the CN value)
  • CN value on each certificate matches the hostname used by the other members
  • extendedKeyUsage is present on all certs and has value clientAuth (TLS Web Client Authentication)

Also, can someone shed light on why I’m seeing errors about:

MONGODB-X509: UserNotFound: Could not find user “CN=mongod1”

when membership authentication is not using users?

some more info:
When I start up the replica set with the following enabled in my conf:

security:
authorization: enabled
clusterAuthMode: x509

And then access the shell on mongod1, authorize w/ the appropriate user, then run rs.status(), I see entries like this for the other members:

{
“_id” : 2,
“name” : “mongod2:17017”,
“health” : 0,
“state” : 8,
“stateStr” : “(not reachable/healthy)”,
“uptime” : 0,
“lastHeartbeat” : ISODate(“2020-09-18T15:37:18.349Z”),
“lastHeartbeatRecv” : ISODate(“1970-01-01T00:00:00Z”),
“pingMs” : NumberLong(0),
“lastHeartbeatMessage” : “Could not find user “CN=mongod1” for db “$external””,
“syncingTo” : “”,
“syncSourceHost” : “”,
“syncSourceId” : -1,
“infoMessage” : “”,
“configVersion” : -1
}

It looks like mongod2 is refusing to communicate with mongod1 because it can’t find the user CN=mongod1. Based on what @MaBeuLux88 stated earlier about users not being needed for membership authentication, should I assume this error message is a red herring? I have no idea why this isn’t working.

per @Ramachandra_Tummala’s comment, I’m further scrutinizing the certs. I don’t know if this is relevant, but I’m creating the certs from an easyrsa CA with the following X509 extensions:

basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
extendedKeyUsage = clientAuth,serverAuth
keyUsage = digitalSignature,keyEncipherment

Anything here that might cause mongod to fail when attempting to use the certificates?

Also, can anyone confirm that error messages like with mechanism MONGODB-X509: UserNotFound: Could not find user "CN=mongod1" for db "$external" can indeed be indicative of problems with the certificates?