Retrieval Evaluation
In this skill badge, you'll learn to evaluate retrieval systems objectively using a three-part framework (corpus, queries, and relevance judgments) that underlies every benchmark and metric in the field. You'll build intuition for core retrieval metrics (Precision, Recall@k, NDCG@k, MRR), understanding what each reveals and when to use it based on your application's failure mode. You'll also examine two real benchmarks, MS MARCO and MIRACL, to see how their design choices shape which metrics apply and what their scores can and can't tell you. The skill ends with a hands-on lab: running a full evaluation pipeline with Voyage AI and MongoDB Atlas, comparing lexical, vector, and hybrid retrieval, and interpreting results across all three. By the end, you'll be able to choose the right metric, contextualize benchmark scores for your use case, and run structured evaluations on your own data.
|
Upon completion of the Retrieval Evaluation skill and assessment, you will earn a Credly Badge that you are able to share with your network. |
Learning Objectives
Benchmark embedding models using a golden dataset
Identify the three components of a test collection (corpus, queries, and relevance judgments).
Interpret core retrieval evaluation metrics
Explain what Recall@k, MRR, and NDCG@k each measure and what each metric reveals about an embedding model's retrieval quality.
Evaluate and compare retrieval strategies using real benchmark data
Read MS MARCO and MIRACL benchmark results in context, run a structured evaluation across lexical, vector, and hybrid search, and use metric results to make a justified model selection decision.
Who is this Course Good for?
If you are a backend developer, data engineer, or AI practitioner building search features with MongoDB or Voyage AI, the Retrieval Evaluation Skill Badge is designed for you. You may already have a working lexical search, vector search, or hybrid implementation but aren't sure whether it's actually returning the right results, or you've seen benchmark leaderboards and aren't sure how to interpret them for your own use case. Whether you're selecting an embedding model, tuning Atlas Search, or auditing an existing retrieval pipeline, this skill gives you the evaluation framework and metric vocabulary to measure quality objectively and make confident, data-backed decisions.
What to Expect in this Course
The skill begins with the fundamentals of retrieval evaluation. You will learn why retrieval systems don't always reveal their own weaknesses: results can look plausible and still be wrong, whether the system uses lexical search, vector search, or a combination of both. Only structured evaluation can surface that difference. You will establish the three-part framework at the center of every evaluation: a corpus of documents, queries representing real user information needs, and relevance judgments that record the known-correct answers. You will also understand why relevance must be judged against the underlying information need rather than the query text, and how this same framework applies across all three layers of a retrieval system.
From there, you will build intuition for the core retrieval metrics. Starting from the fundamental Recall/Precision trade-off and the two error types (false positives and false negatives) you will work through Recall@k, which measures coverage over a truncated ranked list, and NDCG@k, which goes further by rewarding systems that place the most relevant documents highest. You will also learn MRR, which measures how quickly the first correct result appears, and work through a practical decision framework for choosing the right metric based on the failure mode your application can least afford.
The next lessons take you inside real evaluation datasets. You will learn that golden dataset, ground truth, test collection, and judgment list are all names for the same thing (the known-correct reference that makes retrieval evaluation possible) and understand how binary versus graded relevance labels determine which metrics a dataset can support. You will then go inside MS MARCO and MIRACL, two of the most widely used retrieval benchmarks, to understand why their design decisions determine their primary metrics, where their scores apply, and how to determine whether a published benchmark result is actually predictive for your use case.
The skill closes with a hands-on lab. Using the SciFact BEIR benchmark dataset, Voyage AI embeddings, and MongoDB Atlas, you will compute Precision@k, Recall@k, NDCG@k, and MRR across lexical (BM25), vector, and hybrid retrieval strategies. You will compare results across configurations, sweep hybrid alpha values to find the optimal blend, and finish by curating your own custom evaluation set using an LLM-as-judge workflow. Throughout the skill, concepts are reinforced through detailed video lessons and this hands-on activity. By the end, you will have the vocabulary, the framework, and the practical experience to evaluate any retrieval system with confidence.
Summary of the Course
- Define information retrieval in the context of document search systems.
- Explain why retrieval systems require structured evaluation.
- Identify the three components of a test collection.
- Explain why evaluation of a vector or hybrid retrieval system should begin with the embedding model.
- Define false positives and false negatives in the context of retrieval systems.
- Explain what Precision and Recall each measure in retrieval.
- Explain the trade-off between optimizing for Precision versus Recall.
- Explain how Recall@k differs from Recall across a full corpus.
- Explain what NDCG@k measures in a ranked result set.
- Explain what MRR measures in a ranked result set.
- Identify the appropriate retrieval metric for a given search system goal.
- Distinguish between an information need and a query.
- Define corpus, document, and relevance in the context of information retrieval.
- Recognize the different terms used to refer to an evaluation dataset.
- Distinguish between binary and graded relevance labels.
- Explain how the label type determines which retrieval metrics can be computed.
- Explain how LLM judges are used to scale relevance labeling.
- Identify the design characteristics of the MS MARCO benchmark dataset.
- Explain why MRR is the primary metric used with MS MARCO.
- Contrast the design of MIRACL with MS MARCO.
- Explain how to determine whether a benchmark score applies to a given production use case.
- Compute Precision@k, Recall@k, NDCG@k, and MRR for lexical (BM25), vector, and hybrid retrieval strategies and compare results across configurations.
- Build a custom evaluation set using an LLM-as-judge workflow and run standard retrieval metrics against it.
Emily Pope | Senior Manager, Curriculum
Emily Pope is a Senior Manager, Technical Curriculum at MongoDB. She loves learning and loves making it easy for others to learn how and when to use deeply technical products. Her team is responsible for creating skill badges focused on search, vector search, and AI in addition to the core database. Beyond skill badges, they develop AI tooling to support the creation of technically sound learning content used by teams across MongoDB. Before joining MongoDB, she's created learning experiences on databases, computer science, full stack development, and even clinical trial design and analysis. Emily holds an Ed.M. in International Education Policy from Harvard Graduate School of Education and began her career as an English teacher in Turkiye with the Fulbright program.
Manuel Fontan Garcia | Staff Technologist
Manuel is a Staff Technologist on MongoDB’s Curriculum team, where he focuses on AI-assisted curriculum development and GenAI education. He has contributed to MongoDB University’s AI and Search training portfolio across topics including vector search, RAG, AI agents, memory, and Voyage AI integrations. He has also led the development of Curriculum AI tooling to improve the speed, consistency, and quality of content creation and maintenance. Previously, Manuel was a Senior Technical Services Engineer on MongoDB’s Core team. Between his two tenures at MongoDB, he worked for a little over two years as a Database Reliability Engineer at Slack and later at Cognite before rejoining the company. With more than 15 years of experience in software development and distributed systems, Manuel is naturally curious and holds an MSc in Telecommunications Engineering from the University of Vigo, Spain, and an MSc in Free and Open Source Software from Rey Juan Carlos University, Spain.
Parker Faucher | University Curriculum Engineer
Parker Faucher is a Senior Curriculum Engineer on the MongoDB University team, where he designs and develops technical learning content for developers and data professionals. Based in Phoenix, Arizona, Parker specializes in building skills-based learning experiences across topics like vector search, AI applications, and MongoDB performance, helping learners go from concept to hands-on practice.
With a strong focus on quality and efficiency, Parker also leads efforts to integrate AI tooling into the curriculum development workflow, streamlining reviews, accelerating iteration cycles, and freeing up more time for the work that matters most.
Sequoyha Pelletier | Senior Technologist
Sequoyha Pelletier is a Senior Technologist at MongoDB, bringing over 15 years of experience in technical curriculum development and delivery. Before joining MongoDB, he worked in the Worldwide Support team for DataStax, eventually leading the curriculum team for new hire onboarding.
Sequoyha is extremely passionate about providing quality education for free to those in need and enjoys pushing the boundaries of what is considered "normal" practices with delivering educational content.
Hey there! My name is Sequoyha and I'm a Senior Technologist at MongoDB. Welcome to retrieval evaluation skill badge.
Your search system returns results. Maybe it uses keyword matching, maybe it uses vector search powered by an embedding model, or maybe it combines both. Either way, the results look reasonable.
But reasonable-looking results and accurate results are not the same thing, and a poor result doesn't always announce itself: you get plausible documents, confident rankings, and no indication that anything was missed.
In this skill, we'll teach you how to measure retrieval quality across lexical search, vector search, and the embedding models that power semantic retrieval.
Every retrieval evaluation, regardless of the type of search system, is built from three things: a document corpus, a set of queries representing real user needs, and relevance judgments that record the correct answers. This skill covers how to use that structure from the ground up, across four lessons and a hands-on lab.
We start with the evaluation framework itself: what a test collection contains, why relevance is judged against an information need rather than query text, and which layer of a retrieval system to evaluate first.
From there, we cover the metrics you'll use to measure retrieval quality: Precision, Recall@k, NDCG@k, and MRR. Each one answers a different question about your results, and knowing which to reach for depends on what kind of failure your application can least afford.
We then go inside real evaluation datasets. You'll see how MS MARCO and MIRACL are structured, why their design decisions determine which metrics they support, and what those benchmarks can and can't tell you about your own system.
The course closes with a hands-on lab where you'll run a retrieval evaluation system yourself. You'll work with a real corpus, issue queries, compute metrics, and compare models, putting everything from the lessons into practice before committing to a production decision.
By the end of this skill badge, you'll be able to choose the right metric for your retrieval goal, read a benchmark score with enough context to know whether it applies to your use case, and run a structured evaluation on your own data before committing to a model in production.
When you're done, take a short skill check to demonstrate your knowledge. Pass it, and you'll earn an official Credly badge to share on LinkedIn. Let's get started!
