Code Summary: Inside Real Benchmark Datasets

The following summarizes the code used to view various retrieval evaluation datasets.

Prerequisites

• Python

• datasets

• beir

Usage

Print a Document from MS MARCO Dataset:

The following loads the MS MARCO v2.1 validation set in streaming mode, pulls the first sample, and prints the query ID, query text, total number of candidate passages, and each passage's binary relevance label (is_selected: 0 or 1) alongside its first 80 characters of text.

from datasets import load_dataset

dataset = load_dataset("microsoft/ms_marco", "v2.1", split="validation", streaming=True)

sample = next(iter(dataset))

print("MS MARCO — binary labels:")
print(f"  query_id : {sample['query_id']}")
print(f"  query    : {sample['query']!r}")
print(f"  passages : {len(sample['passages']['passage_text'])} candidates")
for i, (text, selected) in enumerate(zip(
    sample["passages"]["passage_text"],
    sample["passages"]["is_selected"],
)):
    print(f"    [{i}] is_selected={selected}  {text[:80]!r}…")

Load and Inspect an NFCorpus Sample:

The following downloads and loads the NFCorpus test split using the BEIR library, finds the first query with graded (multi-level) relevance labels, and prints the query ID, query text, and the first five document judgments — each showing the document ID, relevance grade, and the first 60 characters of the document title.

from beir import util
from beir.datasets.data_loader import GenericDataLoader

data_path = util.download_and_unzip(
    "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nfcorpus.zip",
    "/tmp/beir_datasets",
)
corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")

qid = next(q for q in qrels if len(set(qrels[q].values())) > 1)
print("NFCorpus — graded labels (same structure as MIRACL):")
print(f"  query_id : {qid!r}")
print(f"  query    : {queries[qid]!r}")
for did, grade in list(qrels[qid].items())[:5]:
    print(f"    doc {did}  grade={grade}  {corpus[did]['title'][:60]!r}…")