Video Transcript (English)

Now that we've indexed our embeddings, let’s see the real power of Atlas Vector Search.

We'll query our data and find results based on semantic meaning.

In this lesson, we’ll create vectors for a query.

We'll use those vectors in the vectorSearch aggregation stage to perform a search.

After that, we’ll add a pre-filter to return results efficiently and increase performance.

Get ready because this is the fun part!

Now you may be wondering, how can we compare the text of a query with long lists of numbers in order to find similarity between our query and the vector embeddings?

Good question. Recall we had to generate embeddings for our plot field when we indexed our data.

We'll need to do the same thing with our query.

Good thing we already have a function to do this.

For the query to be successful we must use the same model we used when indexing the data. If we tried to use a different model, we wouldn't be able to compare the query with the vector embeddings and find similarity.

Now that we know how to generate an embedding for the query, let's talk about the $vectorSearch aggregation stage.

The vectorSearch stage performs a nearest neighbor search on our embeddings.

Similar to the search stage, the vectorSearch stage must be the first stage in our aggregation pipeline.

For this demonstration, I'll use the Pymongo driver, but vectorSearch also has driver support for node.js, Java, and CSharp.

Additional driver support is coming in the future so keep an eye on the MongoDB Documentation.

We're almost ready to build our pipeline but first, we need a query!

If you remember, vector search excels at extracting the semantic meaning from queries.

So what does that look like?

Lately, I've been really into movies about people who are escaping maximum security facilities, so let’s use that as our query.

If you noticed, this query isn’t focused on a particular keyword but rather a description of what I want. This is an example of a semantic query.

We'll plug this query into our embedding function, along with the model and api_key. We’ll then use the embedding in the vectorSearch stage.

To start, we’ll store our pipeline in a variable labeled pipeline.

Then, we’ll add the dollar vectorSearch stage which has a few options we need to define.

We'll use the index we created in a prior lesson.

Next, we specify the path to our embeddings, which is the plot underscore embeddings field.

After that, we provide the vector embeddings for the query.

Next, we have numCandidates, but let's skip it for a moment, and look at limit. This is the number of documents we want to return. Let's set it to 10, so we receive the 10 documents most similar to our query.

Ok, let's go back to numCandidates, which stands for number of candidates. What does that mean?

If you remember, to search our vectors with the approximate nearest neighbor algorithm, we use an HNSW graph. This is a multi-layered structure that maps our vectors.

We then use a nearest neighbors algorithm to identify vectors that are similar to our query.

The process begins at a random point on the top layer, which contains fewer vectors that are separated by longer distances.

Then, we keep moving down layers, which contain more and more vectors linked closer together, and we search until we reach the bottom layer and get as close as possible to the query point.

The number of times we repeat that process using a different random entry point at the top layer is known as the number of candidates.

The higher this number is relative to the limit, the more likely we are to find the closest vectors to our query.

With that in mind I'll set numCandidates to 100 and my limit is ten.

We recommend a ten to one ratio between the number of candidates and the limit. You can experiment with this of course.

A high setting for numCandidates will increase accuracy at the expense of latency.

Now we've completed the vector search stage, let’s add a projection stage to make our results easier to read.

Finally, let's run the pipeline.

Now remember, our original query was "A movie about people who are trying to escape from a maximum security facility."

When we view our list of results, we see that each plot field describes a movie that involves a prison escape without using those exact words.

We see phrases like "inmate leading the rebellion" and "escape with help from the inside."

This is awesome because Atlas took our query and returned results based on the meaning of the query.

All we had to do was describe what we wanted to watch.

Now suppose I'm in the mood for a more recent movie.

Remember when we added a filter to the index we created?

Well, we can use it in our query!

The filter will pre-filter results before performing the vector search.

This improves performance by removing irrelevant data before we start and therefore narrowing down the search space.

As you can imagine, searching an entire HNSW graph with thousands of dimensions is resource intensive.

Let's add the filter option to our vectorSearch stage.

We'll filter on the year field.

And specify that I want to find movies released after 2010.

We’ll also update the projection stage.

Now, when we run the pipeline, we receive exactly what we're looking for!

Prefiltering is a powerful tool and we recommend using it when possible.

Great work! Let’s recap what we learned.

First, we learned that we must vectorize our query in order to perform a search. And we need to use the same embedding model that we used for our index data.

We learned about the powerful $vectorSearch aggregation stage to perform a search.

Finally, we added a filter to the $vectorSearch aggregation stage, which pre-filtered our data. It returned specific results and improved performance.

Next, we’ll learn how to combine keyword search and vector search to get the best of both worlds!