MongoDB Replica Set on Kubernetes

I am looking for help with connecting to a mongo db replica-set , on K8s.
Greatful for any help , pointers for the same…

I have created a mongodb replicaset using the following method.
- Created mongo instance in 3 pods.
- Attached the 3 pods at startup in a replicaset
- Exposed the services using K8s Services.
The Replicas are all up and running and can access each other [tested using ‘mongo --hostname ’]
However, I am not able to access the services using the connection string
“mongodb://mongodb-0-service:27017,mongodb-1-service:27017,mongodb-2-service:27017/admin?replicaSet=rs0”

Attached is one set of K8s manifests use to set this env up.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongo-0-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: "2Gi"
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/db/mongo-0"
---
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: default
  name: mongo-0-pv-claim
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 1Gi
----
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-0-service
  labels:
    run: mongodb-0-service
spec:
  ports:
  - port: 27017
    targetPort: 27017
    protocol: TCP
  selector:
    defacementComponent: mongodb-0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: mongodb-0
  labels:
    env: test
    defacementComponent: mongodb-0
spec:
  replicas: 1
  selector:
    matchLabels:
      defacementComponent: mongodb-0
  template:
    metadata:
      labels:
        defacementComponent: mongodb-0
    spec:
      terminationGracePeriodSeconds: 10
      affinity:
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
           - labelSelector:
               matchExpressions:
               - key: "app"
                 operator: In
                 values:
                 - mongo
             topologyKey: "kubernetes.io/hostname"
      containers:
        - image: mtlbillyfong/mongodb-replica-set:20200330-stable-1
          name: mongodb-0
          resources:
            requests:
              ephemeral-storage: "1Gi"
              cpu: "500m"
              memory: "1Gi"
            limits:
              ephemeral-storage: "2Gi"
              cpu: "700m"
              memory: "2Gi"
          env:
            - name: "MONGO_INITDB_ROOT_USERNAME"

Error that I am getting is

# mongo mongodb://mongodb-0-service:27017,mongodb-1-service:27017,mongodb-2-service:27017
MongoDB shell version v4.4.4
connecting to: mongodb://mongodb-0-service:27017,mongodb-1-service:27017,mongodb-2-service:27017/?compressors=disabled&gssapiServiceName=mongodb
Error: couldn't connect to server mongodb-2-service:27017, connection attempt failed: SocketException: Error connecting to mongodb-2-service:27017 (10.106.185.83:27017) :: caused by :: Connection timed out :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6
exception: connect failed
exiting with code 1

PS : I have also tried the usual method of creating a headless service and Statefulset , there too, I was not able to access the replicaset using the connection string .

I am following the repo :

First my version of mongo does not recognize the –hostname option. Which version are you using? I guess you mean that you tried all 3 of them with:

mongo --host mongodb-0-service
mongo --host mongodb-1-service
mongo --host mongodb-2-service

and all were successful. The output of the command rs.status() when executed from the PRIMARY instance.

rs0:PRIMARYrs.status()
{
        "set" : "rs0",
        "date" : ISODate("2021-03-07T17:33:49.772Z"),
        "myState" : 1,
        "term" : NumberLong(2),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "majorityVoteCount" : 2,
        "writeMajorityCount" : 2,
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1615138429, 1),
                        "t" : NumberLong(2)
                },
                "lastCommittedWallTime" : ISODate("2021-03-07T17:33:49.596Z"),
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1615138429, 1),
                        "t" : NumberLong(2)
                },
                "readConcernMajorityWallTime" : ISODate("2021-03-07T17:33:49.596Z"),
                "appliedOpTime" : {
                        "ts" : Timestamp(1615138429, 1),
                        "t" : NumberLong(2)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1615138429, 1),
                        "t" : NumberLong(2)
                },
                "lastAppliedWallTime" : ISODate("2021-03-07T17:33:49.596Z"),
                "lastDurableWallTime" : ISODate("2021-03-07T17:33:49.596Z")
        },
        "lastStableRecoveryTimestamp" : Timestamp(1615138389, 1),
        "lastStableCheckpointTimestamp" : Timestamp(1615138389, 1),
        "electionCandidateMetrics" : {
                "lastElectionReason" : "stepUpRequestSkipDryRun",
                "lastElectionDate" : ISODate("2021-03-07T14:57:48.407Z"),
                "electionTerm" : NumberLong(2),
                "lastCommittedOpTimeAtElection" : {
                        "ts" : Timestamp(1615129063, 1),
                        "t" : NumberLong(1)
                },
                "lastSeenOpTimeAtElection" : {
                        "ts" : Timestamp(1615129063, 1),
                        "t" : NumberLong(1)
                },
                "numVotesNeeded" : 2,
                "priorityAtElection" : 1,
                "electionTimeoutMillis" : NumberLong(10000),
                "priorPrimaryMemberId" : 0,
                "numCatchUpOps" : NumberLong(0),
                "newTermStartDate" : ISODate("2021-03-07T14:57:49.345Z"),
                "wMajorityWriteAvailabilityDate" : ISODate("2021-03-07T14:57:49.564Z")
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "10.110.249.195:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 9325,
                        "optime" : {
                                "ts" : Timestamp(1615138419, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1615138419, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2021-03-07T17:33:39Z"),
                        "optimeDurableDate" : ISODate("2021-03-07T17:33:39Z"),
                        "lastHeartbeat" : ISODate("2021-03-07T17:33:47.938Z"),
                        "lastHeartbeatRecv" : ISODate("2021-03-07T17:33:47.934Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "10.96.135.26:27017",
                        "syncSourceHost" : "10.96.135.26:27017",
                        "syncSourceId" : 1,
                        "infoMessage" : "",
                        "configVersion" : 194403
                },
                {
                        "_id" : 1,
                        "name" : "10.96.135.26:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 9518,
                        "optime" : {
                                "ts" : Timestamp(1615138429, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2021-03-07T17:33:49Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1615129068, 1),
                        "electionDate" : ISODate("2021-03-07T14:57:48Z"),
                        "configVersion" : 194403,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 2,
                        "name" : "10.106.185.83:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 9427,
                        "optime" : {
                                "ts" : Timestamp(1615138419, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1615138419, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2021-03-07T17:33:39Z"),
                        "optimeDurableDate" : ISODate("2021-03-07T17:33:39Z"),
                        "lastHeartbeat" : ISODate("2021-03-07T17:33:47.934Z"),
                        "lastHeartbeatRecv" : ISODate("2021-03-07T17:33:47.937Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "10.96.135.26:27017",
                        "syncSourceHost" : "10.96.135.26:27017",
                        "syncSourceId" : 1,
                        "infoMessage" : "",
                        "configVersion" : 194403
                }
        ],
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1615138429, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1615138429, 1)
}

On my installation , I have tried using the ‘mongo --host seconday-IP’ seconday-IP is as listed in the rs.status() output.

mongo --version

MongoDB shell version v4.2.3
git version: 6874650b362138df74be53d366bbefc321ea32d4
OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1804
distarch: x86_64
target_arch: x86_64

My guess is that you must use IP addresses rather than host name like you have in the following:

That is what I thought too… I tried the following

  • Created a temperory mongo client using ‘kubectl exec -it mongo --/bin/sh’
  • Connected to my replica-set from the temp mongo pod (using the IP address )., which was successful.
  • Now tried to use the same connection string , from my application Pod … which didnot connect …
    Looks like some DNS issue…

Same as what? See

So try

mongodb://10.110.249.195:27017,10.96.135.26:27017,10.106.185.83:27017/admin?replicaSet=rs0