Video Transcript (English)

A user files a ticket: "Search is slow." No error code. No reproduction steps. Just three words and a frustrated user on the other end.

This is one of the most common and most frustrating performance complaints you'll encounter with vector search. Is the index too big to fit in RAM? Is the hardware undersized? Did a batch ingest job kick off and starve your queries of CPU?

In this video, we'll help you diagnose a slow query and see what's happening behind the scenes.

We'll start by introducing you to relevant Atlas metrics which will help you identify potential bottlenecks in your system.

From there, you'll be able to dig deeper by reviewing metrics specific to your search indexes

You'll correlate these metrics with search page faults, which are another indicator of memory pressure.

And finally, you'll learn how to assess whether CPU contention between your database and search workloads may be compounding your issues.

After you have a clear diagnosis of the problem, we'll introduce the next steps you can take to mitigate it.

Before diagnosing a problem, it helps to understand how your search workload behaves when it’s healthy. While a performance baseline is not strictly required for troubleshooting, it is a powerful tool for correlating current symptoms with past behavior.

If you don’t have one when an incident occurs, you can often reconstruct a baseline later using historical metrics, logs, and representative test workloads once the system is stable again.
With your baseline in hand, head to Atlas cluster metrics dashboard.

Because search processes are memory intensive it is a good start to evaluate memory usage.

Depending on your deployment, you'll find the metrics we care about either under the Search Nodes tab for dedicated deployments, or the Search Metrics tab for co-located ones. Either way, the metrics we'll be working through are the same.

To get an overview of the memory usage in the cluster the first metric you will check is System Memory

Which shows how much RAM is used versus available on the node overall. If the Memory available is very low, we may be under memory pressure so we need to dig deeper.

So what's next? Search workloads are memory intense. And search indexes tend to take up a significant amount of memory.

So if we're observing potential memory pressure a useful metric to evaluate is search process memory.

Search Process Memory shows both the total space reserved and the approximate size currently occupied by mongot.

What you're looking for is the relationship between System Memory & Search Process Memory. If system memory used is consistently close to the total available, and Search Process Memory is also high, this means that search process is the main driver for RAM usage.

Okay, our search process memory indicates that our search processes are competing for available RAM on our node. Where should we look next?

One thing to keep in mind: infrequently queried indexes may not be fully resident in memory at any given moment, so Search Process Memory alone can understate the actual footprint of your indexes.

Read it alongside System Memory and the Search Index size, the next metric, for the full picture.

The Search Index Size shows the total on-disk size of all your Search indexes, including both full-text and vector, which is what you would need fully loaded into RAM for peak performance.

If your app only uses vector indexes, this number reflects your full vector indexes footprint. But if you have full-text indexes alongside vector indexes, you'll have to do a deeper investigation.

Search Index Size tells us if vector or full text search indexes are creating memory pressure but we cannot tell which vector or full text index is actually the problem.

Looking at Search Required Memory For Vector Indexes will help to disambiguate. Search Required Memory for Vector Indexes shows the total on disk size of only vector search indexes. If this value is not high, then you know that the issue is related to large full text indexes.

Okay, so we have a picture of our available system memory and search process memory. We dug deeper into our search index size and memory required for vector indexes.

The ultimate question we're trying to answer is does the index fit within available memory? This is important because if it doesn't, it is likely that it would be necessary to read data from disk.

If you recall, a vector index's size is a product of how many documents you've indexed, how many dimensions your embedding model produces, and the number of bytes used to store each dimension value. Those numbers multiply fast. Collections that seem modest in document count can produce indexes that easily outgrow the memory available to `mongot`.

This leads us to Search Page Faults which when correlated with the memory metrics we just reviewed, is a strong indicator that your index may be too large for the available memory.

A page fault in the context of a search process happens when mongot requests data that isn't in RAM and has to go to disk to get it. This usually occurs if your index does not fit within memory.

Given how HNSW traversal works, reading data from disk matters a lot.

Every query hops between dozens or hundreds of nodes in the graph, and each hop needs to read a vector value. If even a fraction of those reads miss RAM and hit disk, the latency compounds across every hop. What should take microseconds starts taking milliseconds, and you feel it immediately in query response times.

Occasional spikes during cache warm-up are normal and expected. What you're watching for is a sustained non-zero rate during normal query traffic. If you see that, you have your answer: the index is larger than available RAM, and queries are paying the cost of disk reads on every HNSW hop.

There's one more metric worth checking. Search Normalized Process CPU which tracks whether CPU usage for `mongot` is elevated.

CPU contention can degrade query performance independently of memory pressure, and it can come from two directions.

The first is a large batch ingest of documents: indexing new documents is CPU-intensive work, and after a significant write, mongot may still be merging and flushing index segments in the background, competing with incoming queries for CPU time.

The second is your core database workload itself. On a co-located deployment, `mongod` and `mongot` share the same hardware, so a heavy operational workload on your database nodes can starve `mongot` of the CPU it needs to serve search queries quickly.

In both cases, the pattern to watch for is CPU staying elevated during or well after the workload that triggered it. Once the pressure clears, CPU should return to near zero. If you're troubleshooting slow queries and CPU is still elevated, give it time to settle before drawing conclusions about whether your fix worked.

All of the metrics we've covered are reactive — you check them after a user files a ticket. But you can get ahead of these problems by setting up Atlas Alerts to notify you before users notice anything.

The two most useful alerts for this scenario are Atlas Search: Index Size on Disk and Search Process: Ran out of memory.

Nice work. Let's recap what we covered.

Using your baseline numbers performance, we walked through an investigation.

First, we used System Memory and Search Process Memory together to assess whether mongot is under RAM pressure.

Then, we looked at Search Index Size to understand whether your indexes can realistically fit in available RAM.

After that, we used Search Page Faults to confirm whether slow queries are the result of disk reads on every HNSW hop.

Finally, we checked Search Normalized Process CPU to catch a completely separate root cause: CPU contention from indexing or your database workload.