AtlasError 8000 when connecting from Kubernetes

Hey community,

… realy tried hard to get it working but I need your help:
I want to deploy following tech-stack on Kubernetes (Django, Celery, Redis, Flower, …) Problem: I can’t connect to my MongoDB-Atlas Cluster as this is the only one not deployed directly in Kubernetes.
It throws AtlasError 8000 when I want to open the connection:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 1278, in _get_socket
    sock_info = self.sockets.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/pymongo/cursor.py", line 647, in __getitem__
    for doc in clone:
  File "/usr/local/lib/python3.9/site-packages/pymongo/cursor.py", line 1207, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python3.9/site-packages/pymongo/cursor.py", line 1124, in _refresh
    self.__send_message(q)
  File "/usr/local/lib/python3.9/site-packages/pymongo/cursor.py", line 999, in __send_message
    response = client._run_operation_with_response(
  File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1368, in _run_operation_with_response
    return self._retryable_read(
  File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1464, in _retryable_read
    with self._slaveok_for_server(read_pref, server, session,
  File "/usr/local/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1309, in _slaveok_for_server
    with self._get_socket(server, session, exhaust=exhaust) as sock_info:
  File "/usr/local/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1246, in _get_socket
    with server.get_socket(
  File "/usr/local/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 1231, in get_socket
    sock_info = self._get_socket(all_credentials)
  File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 1281, in _get_socket
    sock_info = self.connect(all_credentials)
  File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 1197, in connect
    sock_info.check_auth(all_credentials)
  File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 793, in check_auth
    self.authenticate(credentials)
  File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 810, in authenticate
    auth.authenticate(credentials, self)
  File "/usr/local/lib/python3.9/site-packages/pymongo/auth.py", line 673, in authenticate
    auth_func(credentials, sock_info)
  File "/usr/local/lib/python3.9/site-packages/pymongo/auth.py", line 591, in _authenticate_default
    return _authenticate_scram(credentials, sock_info, 'SCRAM-SHA-1')
  File "/usr/local/lib/python3.9/site-packages/pymongo/auth.py", line 295, in _authenticate_scram
    res = sock_info.command(source, cmd)
  File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 683, in command
    return command(self, dbname, spec, slave_ok,
  File "/usr/local/lib/python3.9/site-packages/pymongo/network.py", line 159, in command
    helpers._check_command_response(
  File "/usr/local/lib/python3.9/site-packages/pymongo/helpers.py", line 164, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Authentication failed., full error: {'ok': 0, 'errmsg': 'Authentication failed.', 'code': 8000, 'codeName': 'AtlasError'}

I found a solution that fixes the connection problem but opens new issues that I can’t resolve.
Connection works when I change dnsPolicy of my django deployment from ClusterFirst to Default but this crashes the Kubernetes DNS which I need to connect to different Kubernetes Services (e.g. with SQL_HOST = postgres or CELERY_BACKEND = redis://redis-svc:6379/0). It seems like, that there are connection problems between Kubernetes-Cluster and Atlas. I had the feeling, that I need to open/allow internet traffic for my Deployment, but this should be alright as I am able to run requests to public webpages with Reponse 200.

Does anyone of you have an idea how to fix it?

If you’re having DNS problems I recommend looking into whether the mongodb cluster connection string DNS SRV record might be the issue. You can get the legacy longer connection string that doesn’t use SRV in the connection modal under the legacy driver versions: this can be used with newer drivers

1 Like

If I ran it from Docker or tried a simple connection in a Python Script it worked.

I manged to get it working in the Kubernetes Cluster by adding authSource=admin to the connection string.
DB_URI=“mongodb+srv://username:password@examplecluster.v10yc.mongodb.net/mydatabase?retryWrites=true&w=majority&authSource=admin”

The user itself does not have an admin role.
Is this correct, that authSource=admin only refers to the authentication database which is by default “admin” on a MongoDB Cluster?

If I got it right, following situation is the problem:

  • Kubernetes Pods/Deployments change their IP, e.g. on recreation.

  • My Pods run behind a service. The service acts as an access point to link traffic to the correct Pod. The service keeps track of the IP’s of the underlying Pods.

  • My theory: The connection from PyMongo to Atlas worked but the response from Atlas back to PyMongo didn’t. Maybe because it takes an internal IP of the underlying Pod which is not accessible from outside.
    Adding “authSource=admin” to connection string seems to change the communication in a way that the response from Atlas can make it’s way back to PyMongo driver.

Does someone have an idea why it works now?

I ran into the same situation, but with a StatefulSet that doesn’t have any service backing it. It is strange because I had another StatefulSet application that had the exact same configuration for the mongo URI that worked fine without authSource and another that needed the authSource=admin param to make work. They were even running on the same K8s Node which rules out anything specific to the node proxy I think.

Like in your case, when I ran it with the same config in a plain Docker container outside of K8s it worked fine.

I’m using k3s so it might be something specific to that.