Getting Started with Aggregation Pipelines in Rust
Mark Smith15 min read • Published Feb 05, 2022 • Updated Sep 23, 2022
Rate this quickstart
MongoDB's aggregation pipelines are one of its most powerful features. They allow you to write expressions, broken down into a series of stages, which perform operations including aggregation, transformations, and joins on the data in your MongoDB databases. This allows you to do calculations and analytics across documents and collections within your MongoDB database.
This quick start is the second in a series of Rust posts. I highly recommend you start with my first post, Basic MongoDB Operations in Rust, which will show you how to get set up correctly with a free MongoDB Atlas database cluster containing the sample data you'll be working with here. Go read it and come back. I'll wait. Without it, you won't have the database set up correctly to run the code in this quick start guide.
In summary, you'll need:
- An up-to-date version of Rust. I used 1.49, but any recent version should work well.
- A MongoDB cluster containing the
sample_mflixdataset. You can find instructions to set that up in the first blog post in this series.
MongoDB's aggregation pipelines are very powerful and so they can seem a little overwhelming at first. For this reason, I'll start off slowly. First, I'll show you how to build up a pipeline that duplicates behaviour that you can already achieve with MongoDB's
find() method, but instead using an aggregation pipeline with $match, $sort, and $limit stages. Then, I'll show how to make queries that go beyond what can be done with find, demonstrating using $lookup to include related documents from another collection. Finally, I'll put the "aggregation" into "aggregation pipeline" by showing you how to use $group to group together documents to form new document summaries.All of the sample code for this quick start series can be found on GitHub. I recommend you check it out if you get stuck, but otherwise, it's worth following the tutorial and writing the code yourself!
All of the pipelines in this post will be executed against the sample_mflix database's
movies collection. It contains documents that look like this (I'm showing you what they look like in Python, because it's a little more readable than the equivalent Rust struct):
There's a lot of data there, but I'll be focusing mainly on the
_id, title, year, and cast fields.The first argument to
aggregate() is a sequence of pipeline stages to be executed. Much like a query, each stage of an aggregation pipeline is a BSON document. You'll often create these using the doc! macro that was introduced in the previous post.An aggregation pipeline operates on all of the data in a collection. Each stage in the pipeline is applied to the documents passing through, and whatever documents are emitted from one stage are passed as input to the next stage, until there are no more stages left. At this point, the documents emitted from the last stage in the pipeline are returned to the client program, as a cursor, in a similar way to a call to
find().Individual stages, such as
$match, can act as a filter, to only pass through documents matching certain criteria. Other stage types, such as $project, $addFields, and $lookup, will modify the content of individual documents as they pass through the pipeline. Finally, certain stage types, such as $group, will create an entirely new set of documents based on the documents passed into it taken as a whole. None of these stages change the data that is stored in MongoDB itself. They just change the data before returning it to your program! There are stages, like $out, which can save the results of a pipeline back into MongoDB, but I won't be covering it in this quick start.I'm going to assume that you're working in the same environment that you used for the last post, so you should already have the mongodb crate configured as a dependency in your
Cargo.toml file, and you should have a .env file containing your MONGODB_URI environment variable.First, paste the following into your Rust code:
The above code will provide a
Collection instance called movie_collection, which points to the movies collection in your database.Here is some code which creates a pipeline, executes it with
aggregate, and then loops through and prints the detail of each movie in the results. Paste it into your program.This pipeline has two stages. The first is a $match stage, which is similar to querying a collection with
find(). It filters the documents passing through the stage based on a read operation query. Because it's the first stage in the pipeline, its input is all of the documents in the movie collection. The query for the $match stage filters on the title field of the input documents, so the only documents that will be output from this stage will have a title of "A Star Is Born."The second stage is a $sort stage. Only the documents for the movie "A Star Is Born" are passed to this stage, so the result will be all of the movies called "A Star Is Born," now sorted by their year field, with the oldest movie first.
Calls to aggregate() return a cursor pointing to the resulting documents. Cursor implements the Stream trait. The cursor can be looped through like any other stream, as long as you've imported StreamExt, which provides the
next() method. The code above loops through all of the returned documents and prints a short summary, consisting of the title, the first actor in the cast array, and the year the movie was produced.Executing the code above results in:
It is possible to build up whole aggregation pipelines as a single data structure, as in the example above, but it's not necessarily a good idea. Pipelines can get long and complex. For this reason, I recommend you build up each stage of your pipeline as a separate variable, and then combine the stages into a pipeline at the end, like this:
Imagine I wanted to obtain the most recent production of "A Star Is Born" from the movies collection.
This can be thought of as three stages, executed in order:
- Obtain the movie documents for "A Star Is Born."
- Sort by year, descending.
- Discard all but the first document.
The first stage is already the same as
stage_match_title above. The second stage is the same as stage_sort_year_ascending, but with the value 1 changed to -1. The third stage is a $limit stage.The modified and new code looks like this:
If you make the changes above and execute your code, then you should see just the following line:
Wait a minute! Why isn't there a document for the amazing production with Lady Gaga and Bradley Cooper?
Why don't you use the skills you've just learned to find the latest date in the collection? That will give you your answer!
Okay, so now you know how to filter, sort, and limit the contents of a collection using an aggregation pipeline. But these are just operations you can already do with
find()! Why would you want to use these complex, new-fangled aggregation pipelines?Read on, my friend, and I will show you the true power of MongoDB aggregation pipelines.
There's a dirty secret, hiding in the
sample_mflix database. As well as the movies collection, there's also a collection called comments. Documents in the comments collection look like this:It's a comment for a movie. I'm not sure why people are writing Latin comments for these movies, but let's go with it. The second field,
movie_id, corresponds to the _id value of a document in the movies collection.So, it's a comment related to a movie!
Does MongoDB enable you to query movies and embed the related comments, like a JOIN in a relational database? Yes it does—with the $lookup stage.
I'll show you how to obtain related documents from another collection, and embed them in the documents from your primary collection.
First, modify the definition of the
MovieSummary struct so that it has a comments field, loaded from a related_comments BSON field. Define a Comment struct that contains a subset of the data contained in a comments document.Next, create a new pipeline from scratch, and start with the following:
The stage I've called
stage_lookup_comments is a $lookup stage. This $lookup stage will look up documents from the comments collection that have the same movie id. The matching comments will be listed as an array in a BSON field named related_comments, with an array value containing all of the comments that have this movie's _id value as movie_id.I've added a
$limit stage just to ensure that there's a reasonable amount of output without being overwhelming.Now, execute the code.
You may notice that the pipeline above runs pretty slowly! There are two reasons for this:
- There are 23.5k movie documents and 50k comments.
- There's a missing index on the
commentscollection. It's missing on purpose, to teach you about indexes!
I'm not going to show you how to fix the index problem right now. I'll write about that in a later post in this series, focusing on indexes. Instead, I'll show you a trick for working with slow aggregation pipelines while you're developing.
Working with slow pipelines is a pain while you're writing and testing the pipeline. But, if you put a temporary
$limit stage at the start of your pipeline, it will make the query faster (although the results may be different because you're not running on the whole dataset).When I was writing this pipeline, I had a first stage of
{ "$limit": 1000 }.When you have finished crafting the pipeline, you can comment out the first stage so that the pipeline will now run on the whole collection. Don't forget to remove the first stage, or you're going to get the wrong results!
The aggregation pipeline above will print out summaries of five movie documents. I expect that most or all of your movie summaries will end with this:
comments=[].If you're lucky, you may have some documents in the array, but it's unlikely, as most of the movies have no comments. Now, I'll show you how to add some stages to match only movies which have more than two comments.
Ideally, you'd be able to add a single
$match stage which obtained the length of the related_comments field and matched it against the expression { "$gt": 2 }. In this case, it's actually two steps:- Add a field (I'll call it
comment_count) containing the length of therelated_commentsfield. - Match where the value of
comment_countis greater than two.
Here is the code for the two stages:
The two stages go after the
$lookup stage, and before the $limit 5 stage:While I'm here, I'm going to clean up the output of this code to format the comments slightly better:
Now when you run this code, you should see something more like this:
It's good to see Sansa Stark from Game of Thrones really knows her Latin, isn't it?
Now I've shown you how to work with lookups in your pipelines, I'll show you how to use the
$group stage to do actual aggregation.I'll start with a new pipeline again.
The
$group stage is one of the more difficult stages to understand, so I'll break this down slowly.Start with the following code:
In the
movies collection, some of the years contain the "è" character. This database has some messy values in it. In this case, there's only a small handful of documents, and I think we should just remove them, so I've added a $match stage that filters out any documents with a year that's not numeric.Execute this code, and you should see something like this:
Each line is a document emitted from the aggregation pipeline. But you're not looking at movie documents anymore. The
$group stage groups input documents by the specified _id expression and outputs one document for each unique _id value. In this case, the expression is $year, which means one document will be emitted for each unique value of the year field. Each document emitted can (and usually will) also contain values generated from aggregating data from the grouped documents. Currently, the YearSummary documents are using the default values for movie_count and movie_titles. Let's fix that.Change the stage definition to the following:
This will add a
movie_count field, containing the result of adding 1 for every document in the group. In other words, it counts the number of movie documents in the group. If you execute the code now, you should see something like the following:There are a number of accumulator operators, like
$sum, that allow you to summarize data from the group. If you wanted to build an array of all the movie titles in the emitted document, you could add "movie_titles": { "$push": "$title" }, to the $group stage. In that case, you would get YearSummary instances that look like this:Add the following stage to sort the results:
Note that the
$match stage is added to the start of the pipeline, and the $sort is added to the end. A general rule is that you should filter documents out early in your pipeline, so that later stages have fewer documents to deal with. It also ensures that the pipeline is more likely to be able to take advantages of any appropriate indexes assigned to the collection.Aggregations using
$group are a great way to discover interesting things about your data. In this example, I'm illustrating the number of movies made each year, but it would also be interesting to see information about movies for each country, or even look at the movies made by different actors.You've learned how to construct aggregation pipelines to filter, group, and join documents with other collections. You've hopefully learned that putting a
$limit stage at the start of your pipeline can be useful to speed up development (but should be removed before going to production). You've also learned some basic optimization tips, like putting filtering expressions towards the start of your pipeline instead of towards the end.As you've gone through, you'll probably have noticed that there's a ton of different stage types, operators, and accumulator operators. Learning how to use the different components of aggregation pipelines is a big part of learning to use MongoDB effectively as a developer.
I love working with aggregation pipelines, and I'm always surprised at what you can do with them!
Aggregation pipelines are super powerful, and because of this, they're a big topic to cover. Check out the full documentation to get a better idea of their full scope.
Note that aggregation pipelines can also be used to generate new data and write it back into a collection, with the $out stage.
MongoDB provides a free GUI tool called Compass. It allows you to connect to your MongoDB cluster, so you can browse through databases and analyze the structure and contents of your collections. It includes an aggregation pipeline builder which makes it easier to build aggregation pipelines. I highly recommend you install it, or if you're using MongoDB Atlas, use its similar aggregation pipeline builder in your browser. I often use them to build aggregation pipelines, and they include export buttons which will export your pipeline as Python code (which isn't too hard to transform into Rust).
I don't know about you, but when I was looking at some of the results above, I thought to myself, "It would be fun to visualise this with a chart." MongoDB provides a hosted service called Charts which just happens to take aggregation pipelines as input. So, now's a good time to give it a try!