Trying to bring up Ops manager in OpenShift CRC

I’m trying to bring up Ops Manager on OpenShift Code Ready Containers. I’m following instructions here: https://docs.mongodb.com/kubernetes-operator/v1.7/tutorial/deploy-om-container/#deploy-om-container

What I consistently see is, three pods come up successfully: ops-manager-db-0,1,2. Then ops-manager-0,1,2. Then the ops-manager-db pods restart in order: 3,2,1. 3 and 2 succeed, but 1 never succeeds. If shows READY=0/1. Ops manager is also stuck in a state of “Reconciling”

$ oc get opsmanager
NAME          REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE       WARNINGS
ops-manager   3          4.4.1     Running              Reconciling                      21m

ops-manager-db-0 pod repeats the same log messages indefinitely:

> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:52.410] [.info] [cm/director/director.go:planAndExecute:544] <ops-manager-db-0> [16:01:52.410] Step=WaitFeatureCompatibilityVersionCorrect as part of Move=WaitFeatureCompatibilityVersionCorrect in plan failed : <ops-manager-db-0> [16:01:52.410] Postcondition not yet met for step WaitFeatureCompatibilityVersionCorrect because "}
> {"logType":"automation-agent-verbose","contents":"[<Current FeatureCompatibilityVersion =  is not equal to desired = 4.2 >]."}
> {"logType":"automation-agent-verbose","contents":" Recomputing a plan..."}
> {"logType":"mongodb","contents":"2020-09-24T16:01:52.197+0000 I  NETWORK  [listener] connection accepted from 10.116.0.110:49708 #85 (25 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:52.198+0000 I  NETWORK  [conn85] received client metadata from 10.116.0.110:49708 conn85: { driver: { name: \"mongo-java-driver|legacy\", version: \"3.10.2\" }, os: { type: \"Linux\", name: \"Linux\", architecture: \"amd64\", version: \"4.18.0-193.12.1.el8_2.x86_64\" }, platform: \"Java/AdoptOpenJDK/11.0.8+10\" }"}
> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:53.110] [.info] [main/components/agent.go:LoadClusterConfig:197] [16:01:53.110] clusterConfig unchanged"}
> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:53.515] [.info] [cm/mongoclientservice/mongoclientservice.go:func1:1364] [16:01:53.515] Testing auth with username __system db=local to ops-manager-db-0.ops-manager-db-svc.backing.svc.cluster.local:27017 (local=false) connectMode=SingleConnect ipversion=0 tls=false"}
> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:53.602] [.info] [cm/mongoctl/processctl.go:GetKeyHashes:1742] <ops-manager-db-0> [16:01:53.602] Able to successfully auth to ops-manager-db-0.ops-manager-db-svc.backing.svc.cluster.local:27017 (local=false) using desired auth key"}
> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:53.716] [.info] [cm/mongoctl/replsetctl.go:RsMemberCanPerformGlobalAction:468] <ops-manager-db-0> [16:01:53.716] Mongod (ops-manager-db-0.ops-manager-db-svc.backing.svc.cluster.local:27017) cannot perform Global Update Action because there is no healthy primary in the replica set."}
> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:53.717] [.info] [cm/mongoctl/globalaction.go:MongodCanPerformGlobalAction:296] <ops-manager-db-0> [16:01:53.716] Mongod cannot perform Global Update Action because other members of rs set are not done"}
> {"logType":"automation-agent-stdout","contents":"<ops-manager-db-0> [16:01:54.020] ... process has a plan : WaitFeatureCompatibilityVersionCorrect"}
> {"logType":"automation-agent-stdout","contents":"<ops-manager-db-0> [16:01:54.020] Running step 'WaitFeatureCompatibilityVersionCorrect' as part of move 'WaitFeatureCompatibilityVersionCorrect'"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.424+0000 I  NETWORK  [conn84] end connection 10.116.0.114:52980 (24 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.425+0000 I  NETWORK  [listener] connection accepted from 10.116.0.114:53046 #86 (25 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.425+0000 I  NETWORK  [listener] connection accepted from 10.116.0.114:53048 #87 (26 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.425+0000 I  NETWORK  [conn87] end connection 10.116.0.114:53048 (25 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.425+0000 I  NETWORK  [listener] connection accepted from 10.116.0.114:53050 #88 (26 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.425+0000 I  NETWORK  [conn88] received client metadata from 10.116.0.114:53050 conn88: { driver: { name: \"mongo-go-driver\", version: \"v1.0.1\" }, os: { type: \"linux\", architecture: \"amd64\" }, platform: \"go1.13.4\", application: { name: \"MongoDB Automation Agent v10.2.15.5958 (git: 0e81a83f7adc69fb862335072e7c36ceb868b8dd)\" } }"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.425+0000 I  NETWORK  [conn83] end connection 10.116.0.114:52978 (25 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.507+0000 I  ACCESS   [conn88] Successfully authenticated as principal __system on local from client 10.116.0.114:53050"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.516+0000 I  NETWORK  [listener] connection accepted from 10.116.0.114:53052 #89 (26 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.518+0000 I  NETWORK  [listener] connection accepted from 10.116.0.114:53054 #90 (27 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.518+0000 I  NETWORK  [conn90] received client metadata from 10.116.0.114:53054 conn90: { driver: { name: \"mongo-go-driver\", version: \"v1.0.1\" }, os: { type: \"linux\", architecture: \"amd64\" }, platform: \"go1.13.4\", application: { name: \"MongoDB Automation Agent v10.2.15.5958 (git: 0e81a83f7adc69fb862335072e7c36ceb868b8dd)\" } }"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.599+0000 I  ACCESS   [conn90] Successfully authenticated as principal __system on local from client 10.116.0.114:53054"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.602+0000 I  NETWORK  [conn89] end connection 10.116.0.114:53052 (26 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.603+0000 I  NETWORK  [listener] connection accepted from 10.116.0.114:53056 #91 (27 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.603+0000 I  NETWORK  [conn86] end connection 10.116.0.114:53046 (26 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.603+0000 I  NETWORK  [listener] connection accepted from 10.116.0.114:53058 #92 (27 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.603+0000 I  NETWORK  [conn88] end connection 10.116.0.114:53050 (26 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.603+0000 I  NETWORK  [conn92] end connection 10.116.0.114:53058 (25 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.603+0000 I  NETWORK  [conn90] end connection 10.116.0.114:53054 (24 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.603+0000 I  NETWORK  [listener] connection accepted from 10.116.0.114:53060 #93 (25 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.604+0000 I  NETWORK  [conn93] received client metadata from 10.116.0.114:53060 conn93: { driver: { name: \"mongo-go-driver\", version: \"v1.0.1\" }, os: { type: \"linux\", architecture: \"amd64\" }, platform: \"go1.13.4\", application: { name: \"MongoDB Automation Agent v10.2.15.5958 (git: 0e81a83f7adc69fb862335072e7c36ceb868b8dd)\" } }"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:53.616+0000 I  ACCESS   [conn93] Successfully authenticated as principal __system on local from client 10.116.0.114:53060"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:54.094+0000 I  NETWORK  [listener] connection accepted from 10.116.0.109:47232 #94 (26 connections now open)"}
> {"logType":"mongodb","contents":"2020-09-24T16:01:54.095+0000 I  NETWORK  [conn94] received client metadata from 10.116.0.109:47232 conn94: { driver: { name: \"mongo-java-driver|legacy\", version: \"3.10.2\" }, os: { type: \"Linux\", name: \"Linux\", architecture: \"amd64\", version: \"4.18.0-193.12.1.el8_2.x86_64\" }, platform: \"Java/AdoptOpenJDK/11.0.8+10\" }"}
> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:54.020] [.info] [cm/director/director.go:computePlan:269] <ops-manager-db-0> [16:01:54.020] ... process has a plan : WaitFeatureCompatibilityVersionCorrect"}
> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:54.020] [.info] [cm/director/director.go:executePlan:876] <ops-manager-db-0> [16:01:54.020] Running step 'WaitFeatureCompatibilityVersionCorrect' as part of move 'WaitFeatureCompatibilityVersionCorrect'"}
> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:54.020] [.info] [cm/director/director.go:tracef:781] <ops-manager-db-0> [16:01:54.020] Precondition of 'WaitFeatureCompatibilityVersionCorrect' applies because "}
> {"logType":"automation-agent-verbose","contents":"[All the following are true: "}
> {"logType":"automation-agent-verbose","contents":"    ['currentState.Up' = true]"}
> {"logType":"automation-agent-verbose","contents":"]"}
> {"logType":"automation-agent-verbose","contents":"[2020/09/24 16:01:54.021] [.info] [cm/director/director.go:planAndExecute:544] <ops-manager-db-0> [16:01:54.020] Step=WaitFeatureCompatibilityVersionCorrect as part of Move=WaitFeatureCompatibilityVersionCorrect in plan failed : <ops-manager-db-0> [16:01:54.020] Postcondition not yet met for step WaitFeatureCompatibilityVersionCorrect because "}

Here is the manifest for the Ops Manager. (I changed the failure threshhold because Ops Manager was triming out starting up.)

apiVersion: mongodb.com/v1
kind: MongoDBOpsManager
metadata:
  name: ops-manager
  namespace: backing
spec:
  adminCredentials: ops-manager-admin-secret
  applicationDatabase:
    additionalMongodConfig:
      operationProfiling:
        mode: slowOp
    members: 3
    persistent: false
    podSpec:
      cpu: "0.25"
    version: 4.2.6-ent
  backup:
    enabled: false
  configuration:
    automation.versions.source: mongodb
    mms.adminEmailAddr: foo@bar.com
    mms.fromEmailAddr: foo@bar.com
    mms.ignoreInitialUiSetup: "true"
    mms.mail.hostname: foo@bar.com
    mms.mail.port: "465"
    mms.mail.ssl: "true"
    mms.mail.transport: smtp
    mms.minimumTLSVersion: TLSv1.2
    mms.replyToEmailAddr: foo@bar.com
  replicas: 3
  statefulSet:
    spec:
      template:
        spec:
          containers:
          - name: mongodb-ops-manager
            readinessProbe:
              failureThreshold: 100
  version: 4.4.1

Any help is appreciated. Thanks.

In case it helps anyone, I was able to resolve this by adding:

spec:
  applicationDatabase:
    featureCompatibilityVersion: "4.2"

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.