Video Transcript (English)

If you work with large quantities of data, you understand how vital it is to be able to transform, combine, and filter that data. This helps you get real-time insights and analytics, generate reports, and run calculations at scale.

MongoDB’s aggregation framework makes it easy and efficient to handle these workloads. This allows you to query your data in a highly expressive way and reduces application complexity.

In this video, we’ll introduce you to MongoDB’s aggregation framework. We’ll show you how it’s used to query, process, and transform data. We’ll cover some commonly used aggregation stages, discuss the importance of correctly ordering your stages and when and how to use the aggregation framework.

MongoDB's aggregation framework is a powerful tool for querying data and for performing data processing and analysis operations on collections.

An aggregation pipeline consists of one or more stages. These stages query and process documents.

Let’s look at how this works.

After an aggregation pipeline is initiated and optimized by MongoDB’s query optimizer, an initial cursor is automatically created to handle the flow of documents through the pipeline.

The execution engine begins retrieving documents from the collection and adding them to the cursor. If an index is available, the execution engine will use it to increase efficiency.

Then, the cursor sends the documents to the pipeline.

Each stage performs an operation on the input documents such as calculating values or traversing an array.

Then the documents that are output from a stage are passed to the next stage. This continues until the end of the pipeline where the final output is delivered.

MongoDB offers many different aggregation stages.

Let’s take a quick look at some of the most common stages to give you a sense for what’s possible.

Placing it at the beginning of your pipeline can optimize performance by reducing the amount of data that is passed to subsequent stages. When placed at the beginning of the pipeline, $match can also leverage existing indexes to improve performance

The $sort stage orders the documents in the pipeline based on a specified field or fields. Documents can be sorted in ascending or descending order.

A $sort stage can also leverage an index when it is placed towards the beginning of a pipeline!
The $limit stage restricts the number of documents passed to the next stage or final output to a number we specify.

This stage is especially helpful when it appears after a $sort stage. Whenever possible, the query optimizer will combine the two stages. This allows the sort operation to maintain fewer documents in memory as it works, which can improve performance.

The $group stage combines documents into groups based on a group key.

This stage is useful when you need to perform operations or calculations, such as sum, average, or count, on the grouped data.

The $lookup stage performs a left outer join to incorporate related data from another collection into the documents.

This operation is similar to a left outer SQL join.

The $unwind stage deconstructs an array field from the input documents. $unwind outputs a document for each element of the array. In other words, it flattens the array.

Finally, the $project stage reshapes each document so specific fields are included in, excluded from, or added to the output.

This isn’t an exhaustive list of stages, it’s just the beginning! Visit the MongoDB docs to learn about more stages.

When we build an aggregation pipeline, the sequence of the stages matters.

This is because the order in which the operations executed by each stage are performed on the documents in your pipeline will impact the accuracy of the results.

Additionally, only certain stages can leverage indexes to make execution more efficient.

Here are some best practices that you can follow to sequence stages.

Filter documents early with stages like $match to decrease the size of the data. This way, we are only retrieving and processing what we need.

Next, try to place stages that can leverage indexes, like $match and $sort, closer to the beginning of a pipeline. This helps the execution engine use relevant indexes to retrieve documents more efficiently.

Finally, use $project towards the end of a pipeline. This helps us avoid losing fields that are necessary for optimization and to calculate accurate results.

Luckily, the query optimization phase happens automatically before pipeline execution. This will reshape a pipeline to improve performance.

For example, the query optimizer might combine or move stages to leverage an index as long as the final output isn’t impacted.

This optimization is built into the framework, we don’t have to do anything to enable it. It’s just helpful to know that it’s there!

So now you know a little bit more about how MongoDB’s aggregation framework processes data. How do we actually build an aggregation pipeline?

MongoDB gives us a number of options.

We can choose to build an aggregation pipeline programmatically by using syntax from a MongoDB driver and include it in our application code.

We can also build, run, and export pipelines using MongoDB Compass. Compass helps us visualize documents as they are processed by each stage, which is really useful.

Any of these tools make it easy to get started using the aggregation framework to process and transform data.

Your next question may be: when is it a good idea to use aggregation?

Well, just like anything else, it depends on your use-case. However, the simplest solution is often the best.

For simple data retrieval tasks, it’s usually best to use find operations.

When you need to perform operations like complex data processing that involves multiple transformations or data restructuring steps, the aggregation framework is a great tool.

An aggregation pipeline consists of one or more stages that process documents.

There are a number of stages that you can use, but common stages include: $match, $group, $sort, $project, $limit, $lookup, and $unwind.

The sequencing of these stages matters and will impact results and performance.

Great work, see you in the next video!