Video Transcript (English)

Welcome back!

Now that we have a sense for how things work under the hood let's configure an Atlas Vector Index.

If you're familiar with Atlas Search, you'll find that configuring an index for Vector Search is similar to configuring an index for Atlas Search.

The main difference is that we use a vector field type mapping in the search index definition.

But before we configure our vector search index, let’s discuss two limitations.

The first one may seem obvious, but it's worth stating. We can only use the vector field type on fields that contain vector embeddings.

Second, we can't index fields in subdocuments that live inside an array field. So you should not store your embeddings field in a document that's part of an array of documents.

With that in mind, let’s get to the fun part and configure our vector search index.

To start, let's use the create search index method to create this index on the movies collection.

Next, we'll name the index vectorPlotIndex since we're indexing the vector embeddings.

After that, we specify that this is a vector search index. If we leave this blank, it'll default to a regular search index.

Now let’s configure the index definition.

We use the fields option to specify which fields to filter and index, so we'll define the field type as vector.

In the path, we specify the field that contains our embeddings,

so: the plot underscore embeddings field.

Then, we define the number of dimensions. We'll use a model with 1536 dimensions.

Now, you may be wondering, what you should do if you don’t know or can’t remember the number of dimensions from your model?

You can always find this information in the documentation for your embedding model. Or, if you've already generated the embeddings, you can count the number of elements in the embeddings array field, because each element represents one dimension.

Once we've defined the number of dimensions we need to choose a similarity function.

We have three choices: euclidean, cosine, or dotProduct.

Each of these measure similarity between vectors across multiple dimensions in a different way so they can lead to vastly different results.

Let's briefly go over each one of them.

Euclidean similarity uses the distance between vectors in a multidimensional space. The underlying formula is derived from the pythagorean theorem and generalized for any number of dimensions.

Cosine similarity uses the angle between vectors.

Note that cosine does not take magnitude into account and you can't use zero magnitude vectors with cosine.

To measure cosine similarity, we recommend that you normalize your vectors and use dotProduct instead.

Finally, we have dotProduct.

Similar to cosine, it uses the angle between the vectors, however, it also takes magnitude into consideration.

To use dotProduct, the vector must be normalized to unit length at index- and query-time.

So given all this, how do we choose the correct similarity function?

The first thing you should do is check your embedding model’s documentation to see what the model was trained with. This will ensure optimal results

If that doesn’t work, you’ll have to experiment with each one to see what returns the best results.

Now, back to our vector search index. The embedding model we chose uses cosine.

Alright, now that we've selected a similarity function, we could stop here and create this index but let's add a pre-filter.

Pre-filtering makes search operations more efficient by filtering out irrelevant data and narrowing down the search space.

We’ll use a filter when we write a query, so let’s configure our index for it.

We define our filter below the vector field type definition.

Then, in the path, we specify the field to filter on. Let's filter on the year field, which will allow us to focus on movies from specific years when we search.

Currently, we can only filter on fields with numbers, strings, and booleans as values.

Filtering for more data types will be released in the future so be sure to check the documentation for any updates.

Now that we've added a filter, let's create the vector search index. We’ll receive a confirmation message signifying that our index is being built.

Behind the scenes, the index is building an HNSW graph using the embeddings we generated for the movie data and any other metadata we’d like to filter our queries on.

Because HNSW builds and indexes a graph that contains thousands of dimensions, it's important to keep in mind that HNSW is memory constrained. To account for this, we recommend using dedicated search nodes for your vector search workloads.

This will ensure that you have resource isolation from your main cluster operations, so you can meet your search workload’s indexing and query needs in a cost-effective way.

Awesome job, let’s recap what we learned.

First, we learned that we need to use the vector field type when configuring a search index for vector search.

Next, we learned about the euclidean, cosine, and dotProduct similarity functions.

Finally, we learned how to add a filter to our index.

That was a lot of really neat information but don't go away because now we get to run some searches! See you soon.