Joe Drumgoole

Sr. Director, Developer Relations

How to perform multi-document transactions with Python.

Introduction to Multi-Document ACID Transactions in Python



<div class="introduction">

One of the most important things to think about in the cloud is what is burning dollars while you sleep. In the case of MongoDB Atlas, that is your live clusters. The minute you start a cluster (with the exception of our free tier), we start accumulating cost.

If you're using a dedicated cluster—not one of the cheaper, shared cluster types, such as M0, M2 or M5—then it's easy enough to pause a cluster using the Atlas UI, but logging in over 2FA can be a drag. Wouldn't it be great if we could just jump on a local command line to look at our live clusters?

This you can do with a command line tool like `curl`, some programming savvy, and knowledge of the [MongoDB Atlas Admin API](https://docs.atlas.mongodb.com/api/). But who has time for that? Not me, for sure.

</div>

<div class="content">

That is why I wrote a simple script to automate those steps. It's now a Python package up on PyPi called [mongodbatlas](https://pypi.org/project/mongodbatlas/).

You will need Python 3.6 or better installed to run the script. (This is your chance to escape the clutches of 2.x.)

Just run:

``` bash
$ pip install mongodbatlas
  Collecting mongodbatlas
   Using cached mongodbatlas-0.2.6.tar.gz (17 kB)
  ...
  ...
  Building wheels for collected packages: mongodbatlas
   Building wheel for mongodbatlas (setup.py) ... done
   Created wheel for mongodbatlas: filename=mongodbatlas-0.2.6-py3-none-any.whl size=23583       sha256=d178ab386a8104f4f5100a6ccbe61670f9a1dd3501edb5dcfb585fb759cb749c
   Stored in directory: /Users/jdrumgoole/Library/Caches/pip/wheels/d1/84/74/3da8d3462b713bfa67edd02234c968cb4b1367d8bc0af16325
  Successfully built mongodbatlas
  Installing collected packages: certifi, chardet, idna, urllib3, requests, six, python-dateutil, mongodbatlas
  Successfully installed certifi-2020.11.8 chardet-3.0.4 idna-2.10 mongodbatlas-0.2.6 python-dateutil-2.8.1 requests-2.25.0 six-1.15.0 urllib3-1.26.1
```

Now you will have a script installed called `atlascli`. To test the install worked, run `atlascli -h`.

``` bash
$ atlascli -h
  usage: atlascli [-h] [--publickey PUBLICKEY] [--privatekey PRIVATEKEY]
                 [-p PAUSE_CLUSTER] [-r RESUME_CLUSTER] [-l] [-lp] [-lc]
                 [-pid PROJECT_ID_LIST] [-d]

  A command line program to list organizations,projects and clusters on a
  MongoDB Atlas organization.You need to enable programmatic keys for this
  program to work. See https://docs.atlas.mongodb.com/reference/api/apiKeys/

  optional arguments:
   -h, --help            show this help message and exit
   --publickey PUBLICKEY
                         MongoDB Atlas public API key.Can be read from the
                         environment variable ATLAS_PUBLIC_KEY
   --privatekey PRIVATEKEY
                         MongoDB Atlas private API key.Can be read from the
                         environment variable ATLAS_PRIVATE_KEY
   -p PAUSE_CLUSTER, --pause PAUSE_CLUSTER
                         pause named cluster in project specified by project_id
                         Note that clusters that have been resumed cannot be
                         paused for the next 60 minutes
   -r RESUME_CLUSTER, --resume RESUME_CLUSTER
                         resume named cluster in project specified by
                         project_id
   -l, --list            List everything in the organization
   -lp, --listproj       List all projects
   -lc, --listcluster    List all clusters
   -pid PROJECT_ID_LIST, --project_id PROJECT_ID_LIST
                         specify the project ID for cluster that is to be
                         paused
   -d, --debug           Turn on logging at debug level

  Version: 0.2.6
```

To make this script work, you will need to do a little one-time setup on your cluster. You will need a [programmatic key](https://docs.atlas.mongodb.com/configure-api-access/#atlas-prog-api-key) for your cluster. You will also need to [enable the IP address](https://docs.atlas.mongodb.com/configure-api-access/#edit-the-api-whitelist) that the client is making requests from.

There are two ways to create an API key:

-   If you have a single project, it's probably easiest to create a [single project API key](#single-project-api-key)
-   If you have multiple projects, you should probably create an [organization API key](#organization-api-key) and add it to each of your projects.

## <a name="single-project-api-key">Single Project API Key

Going to your "Project Settings" page by clicking on the "three dot" button next your project name at the top-left of the screen and selecting "Project Settings". Then click on "Access Manager" on the left side of the screen and click on "Create API Key". Take a note of the public *and* private parts of the key, and ensure that the key has the "Project Cluster Manager" permission. More detailed steps can be found in the [documentation](https://docs.atlas.mongodb.com/configure-api-access/#create-one-api-key-for-one-project).

![A screenshot of the project's Access Manager page.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/how-to/pause-resume-atlas-clusters/annotated-project-key.png)

## <a name="organization-api-key">Organization API Key

Click on the cog icon next to your organization name at the top-left of the screen. Click on "Access Manager" on the left-side of the screen and click on "Create API Key". Take a note of the public *and* private parts of the key. Don't worry about selecting any specific organization permissions.

![A screenshot showing the buttons described above this image](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/how-to/pause-resume-atlas-clusters/annotated-org-key.png)

Now you'll need to invite the API key to each of the projects containing clusters you wish to control. Click on "Projects' on the left-side of the screen. For each of the projects, click on the "three dots" icon on the same row in the project table and select "Visit Project Settings" Click on "Access Manager", and click on "Invite to Project" on the top-right. Paste your public key into the search box and select it in the menu that appears. Ensure that the key has the "Project Cluster Manager" permission that it will need to pause and resume clusters in that project.

More detailed steps can be found in the [documentation](https://docs.atlas.mongodb.com/configure-api-access/#create-one-api-key-in-one-organization).

## Configuring `atlascli`

The programmatic key has two parts: a public key and a private key. Both of these are used by the `atlascli` program to query the projects and clusters associated with the organization.

You can pass the keys in on the command line, but this is not recommended because they will be stored in the command line history. It's better to store them in environment variables, and the `atlascli` program will look for these two:

-   `ATLAS_PUBLIC_KEY`: stores the public key part of the programmatic key
-   `ATLAS_PRIVATE_KEY`: stores the private part of the programmatic key

Once you have created these environment variables, you can run `atlascli -l` to list the organization and its associated projects and clusters. I've blocked out part of the actual IDs with `xxxx` characters for security purposes:

``` bash
$ atlascli -l
  {'id': 'xxxxxxxxxxxxxxxx464d175c',
  'isDeleted': False,
  'links': [{'href': 'https://cloud.mongodb.com/api/atlas/v1.0/orgs/599eeced9f78f769464d175c',
             'rel': 'self'}],
  'name': 'Open Data at MongoDB'}
  Organization ID:xxxxxxxxxxxxf769464d175c Name:'Open Data at MongoDB'
      project ID:xxxxxxxxxxxxd6522bc457f1 Name:'DevHub'
        Cluster ID:'xxxxxxxxxxxx769c2577a54' name:'DRA-Data'                state=running
      project ID:xxxxxxxxx2a0421d9bab Name:'MUGAlyser Project'
        Cluster ID:'xxxxxxxxxxxb21250823bfba' name:'MUGAlyser'              state=paused
      project ID:xxxxxxxxxxxxxxxx736dfdcddf Name:'MongoDBLive'
      project ID:xxxxxxxxxxxxxxxa9a5a04e7 Name:'Open Data Covid-19'
        Cluster ID:'xxxxxxxxxxxxxx17cec56acf' name:'pre-prod'               state=running
        Cluster ID:'xxxxxxxxxxxxxx5fbfe04313' name:'dev'                    state=running
        Cluster ID:'xxxxxxxxxxxxxx779f979879' name:'covid-19'               state=running
      project ID xxxxxxxxxxxxxxxxa132a8010 Name:'Open Data Project'
        Cluster ID:'xxxxxxxxxxxxxx5ce1ef94dd' name:'MOT'                    state=paused
        Cluster ID:'xxxxxxxxxxxxxx22bf6c226f' name:'GDELT'                  state=paused
        Cluster ID:'xxxxxxxxxxxxxx5647797ac5' name:'UKPropertyPrices'       state=paused
        Cluster ID:'xxxxxxxxxxxxxx0f270da18a' name:'New-York-Taxi'          state=paused
        Cluster ID:'xxxxxxxxxxxxxx11eab32cf8' name:'demodata'               state=running
        Cluster ID:'xxxxxxxxxxxxxxxdcaef39c8' name:'stackoverflow'          state=paused
      project ID:xxxxxxxxxxc9503a77fcce0c Name:'Realm'
```

To pause a cluster, you will need to specify the `project ID` and the `cluster name`. Here is an example:

``` bash
$ atlascli --project_id xxxxxxxxxxxxxxxxa132a8010 --pause demodata
  Pausing 'demodata'
  Paused cluster 'demodata'
```

To resume the same cluster, do the converse:

``` bash
$ atlascli --project_id xxxxxxxxxxxxxxxxa132a8010 --resume demodata
  Resuming cluster 'demodata'
  Resumed cluster 'demodata'
```

Note that once a cluster has been resumed, it cannot be paused again for a while.

This delay allows the Atlas service to apply any pending changes or patches to the cluster that may have accumulated while it was paused.

</div>

<div class="summary">

Now go save yourself some money. This script can easily be run from a `crontab` entry or the Windows Task Scheduler.

Want to see the code? It's in this [repo](https://github.com/jdrumgoole/MongoDB-Atlas-API) on GitHub.

For a much more full-featured Atlas Admin API in Python, please check out my colleague [Matthew Monteleone's](https://www.linkedin.com/in/monteleone/) PyPI package [AtlasAPI](https://pypi.org/project/atlasapi/).

> If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

</div>


Learn how to easily pause and resume MongoDB Atlas clusters.

How to Easily Pause and Resume MongoDB Atlas Clusters



<div class="content">

If you have logged into [MongoDB Atlas](https://www.mongodb.com/atlas/database)
recently - and you should, the entry-level tier is free! - you may have
noticed a strange new syntax on 3.6 connection strings.

![Atlas SRV Address screenshot](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/srv-address.png)

## MongoDB Seed Lists

What is this `mongodb+srv` syntax?

Well, in MongoDB 3.6 we introduced the concept of a [seed
list](https://docs.mongodb.com/manual/reference/connection-string/#dns-seedlist-connection-format)
that is specified using DNS records, specifically
[SRV](https://en.wikipedia.org/wiki/SRV_record) and
[TXT](https://en.wikipedia.org/wiki/TXT_record) records. You will recall
from using replica sets with MongoDB that the client must specify at
least one replica set member (and may specify several of them) when
connecting. This allows a client to connect to a replica set even if one
of the nodes that the client specifies is unavailable.

You can see an example of this URL on a 3.4 cluster connection string:

![Atlas screenshot: using driver 3.4 or earlier](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/connection-string.png)

Note that without the SRV record configuration we must list several
nodes (in the case of Atlas we always include all the cluster members,
though this is not required). We also have to specify the `ssl` and
`replicaSet` options.

With the 3.4 or earlier driver, we have to specify all the options on
the command line using the [MongoDB URI
syntax](https://docs.mongodb.com/manual/reference/connection-string/).

The use of SRV records eliminates the requirement for every client to
pass in a complete set of state information for the cluster. Instead, a
single SRV record identifies all the nodes associated with the cluster
(and their port numbers) and an associated TXT record defines the
options for the URI.

## Reading SRV and TXT Records

We can see how this works in practice on a MongoDB Atlas cluster with a
simple Python script.

``` python
import srvlookup #pip install srvlookup
import sys 
import dns.resolver #pip install dnspython 

host = None 

if len(sys.argv) > 1 : 
   host = sys.argv[1] 

if host : 
   services = srvlookup.lookup("mongodb", domain=host) 
   for i in services:
      print("%s:%i" % (i.hostname, i.port)) 
   for txtrecord in dns.resolver.query(host, 'TXT'): 
      print("%s: %s" % ( host, txtrecord))

else: 
  print("No host specified") 
```

We can run this script using the node specified in the 3.6 connection
string as a parameter.

![The node is specified in the connection string](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/connection-string3.png)

``` sh
$ python mongodb_srv_records.py
freeclusterjd-ffp4c.mongodb.net
freeclusterjd-shard-00-00-ffp4c.mongodb.net:27017
freeclusterjd-shard-00-01-ffp4c.mongodb.net:27017
freeclusterjd-shard-00-02-ffp4c.mongodb.net:27017
freeclusterjd-ffp4c.mongodb.net: "authSource=admin&replicaSet=FreeClusterJD-shard-0" 
$ 
```

You can also do this lookup with nslookup:

``` sh
JD10Gen-old:~ jdrumgoole$ nslookup
> set type=SRV > \_mongodb._tcp.rs.joedrumgoole.com
Server: 10.65.141.1
Address: 10.65.141.1#53

Non-authoritative answer:
\_mongodb._tcp.rs.joedrumgoole.com  service = 0 0 27022 rs1.joedrumgoole.com.
\_mongodb._tcp.rs.joedrumgoole.com  service = 0 0 27022 rs2.joedrumgoole.com.
\_mongodb._tcp.rs.joedrumgoole.com  service = 0 0 27022 rs3.joedrumgoole.com.

Authoritative answers can be found from:
> set type=TXT
> rs.joedrumgoole.com
Server: 10.65.141.1
Address: 10.65.141.1#53

Non-authoritative answer:
rs.joedrumgoole.com     text = "authSource=admin&replicaSet=srvdemo"
```

You can see how this could be used to construct a 3.4 style connection
string by comparing it with the 3.4 connection string above.

As you can see, the complexity of the cluster and its configuration
parameters are stored in the DNS server and hidden from the end user. If
a node's IP address or name changes or we want to change the replica set
name, this can all now be done completely transparently from the
client's perspective. We can also add and remove nodes from a cluster
without impacting clients.

So now whenever you see `mongodb+srv` you know you are expecting a SRV
and TXT record to deliver the client connection string.

## Creating SRV and TXT records

Of course, SRV and TXT records are not just for Atlas. You can also
create your own SRV and TXT records for your self-hosted MongoDB
clusters. All you need for this is edit access to your DNS server so you
can add SRV and TXT records. In the examples that follow we are using
the [AWS Route 53 DNS](https://aws.amazon.com/route53/) service.

I have set up a demo replica set on AWS with a three-node setup. They
are

``` sh
rs1.joedrumgoole.com 
rs2.joedrumgoole.com 
rs3.joedrumgoole.com
```

Each has a mongod process running on port 27022. I have set up a
security group that allows access to my local laptop and the nodes
themselves so they can see each other.

I also set up the DNS names for the above nodes in AWS Route 53.

![Setting DNS names in AWS Route 53](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/dnsnames.png)

We can start the mongod processes by running the following command on
each node.

``` sh
$ sudo /usr/local/m/versions/3.6.3/bin/mongod --auth --port 27022 --replSet srvdemo --bind_ip 0.0.0.0 --keyFile mdb_keyfile"
```

Now we need to set up the SRV and TXT records for this cluster.

The SRV record points to the server or servers that will comprise the
members of the replica set. The TXT record defines the options for the
replica set, specifically the database that will be used for
authorization and the name of the replica set. It is important to note
that the **mongodb+srv** format URI implicitly adds "ssl=true". In our
case SSL is not used for the demo so we have to append "&ssl=false" to
the client connector. Note that the SRV record is specifically designed
to look up the **mongodb** service referenced at the start of the URL.

The settings in AWS Route 53 are:

![Settings page from AWS Route 53 - Name, Type, Alias: TTL and Value](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/awsroutes.png)

Which leads to the following entry in the zone file for Route 53.

![Generated entry in Route 53](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/awsroutes2.png)

Now we can add the TXT record. By convention, we use the same name as
the SRV record (`rs.joedrumgoole.com`) so that MongoDB knows where to
find the TXT record.

We can do this on AWS Route 53 as follows:

![Editing the record configuration to set 'rs.joedrumgoole.com' as name](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/awsroutes3.png)

This will create the following TXT record.

![Updated Route 53 entry](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/awsroutes4.png)

Now we can access this service as :

``` sh
mongodb+srv://rs.joedrumgoole.com/test
```

This will retrieve a complete URL and connection string which can then
be used to contact the service.

The whole process is outlined below:

![Setting SRV connection string process](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/srv-process.png)

Once your records are set up, you can easily change port numbers without
impacting clients and also add and remove cluster members.

SRV records are another way in which MongoDB is making life easier for
database developers everywhere.

You should also check out [full documentation on SRV and TXT records in
MongoDB
3.6](https://docs.mongodb.com/manual/reference/connection-string/#dns-seedlist-connection-format).

</div>

<div class="summary">

You can [sign up for a free MongoDB Atlas tier](https://www.mongodb.com/atlas/database)
which is suitable for single user use.

Find out how to use your favorite programming language with MongoDB via
our [MongoDB drivers](https://docs.mongodb.com/drivers/).

Please visit [MongoDB University](https://university.mongodb.com/) for
free online training in all aspects of MongoDB.

Follow [Joe Drumgoole on twitter](https://twitter.com/jdrumgoole) for
more news about MongoDB.

</div>


SRV records are another way in which MongoDB is making life easier for database developers everywhere.

Joe Drumgoole

Content by Joe Drumgoole

Introduction to Multi-Document ACID Transactions in Python

How to Easily Pause and Resume MongoDB Atlas Clusters

MongoDB 3.6: Here to SRV you with easier replica set connections