The Search Evaluation Catalog

Judgment list construction and pooling #

Source: TREC methodology (Voorhees, NIST); Manning et al., Introduction to Information Retrieval; Grainger, AI-Powered Search; OpenSource Connections methodology

Classification — The methodology for constructing the foundational judgment artifact — query selection, document pooling, judgment assignment.

Build a judgment list that supports reliable offline evaluation: representative queries that cover the production query distribution, document pools that capture the candidates any system might surface, judgments that are calibrated and reproducible across assessors.

A naive judgment list has predictable failure modes. Queries cherry-picked by the team don't reflect actual production traffic, so evaluation scores don't predict production quality. Documents judged are only those returned by the current system, so any candidate system surfacing different documents gets unfair scores (because the unjudged documents are treated as not-relevant by default). Judgments produced by a single assessor without calibration drift from any defensible relevance standard. The discipline of judgment list construction addresses these failure modes systematically.

Query sampling. The query list should reflect production traffic, not the team's intuitions about important queries. Sample from production query logs, stratified by query characteristics: head queries (high frequency, narrow set), torso queries (medium frequency, broader set), tail queries (low frequency, very diverse). A judgment list with only head queries misses the long-tail quality issues that drive user frustration. A typical production judgment list might have 200–500 queries spanning the frequency distribution.

Pooling. For each query, judgment lists need a pool of documents to judge. The naive approach — judge only what the current system returns — produces the bias described above. The TREC-style pooling approach: run multiple candidate systems against the query, take the top K from each, union the results, judge everything in the union. The pool includes documents the current system missed but candidates surface, eliminating the bias. Pool depth (K) typically 20–100; the deeper the pool, the better the evaluation supports systems that differ substantially from the baseline.

Relevance grading scale. The choice of scale shapes the judgment task. Binary (relevant / not relevant) is simple but loses information about how relevant. Graded scales (typically 0–4 or 0–5) capture more nuance but require clearer annotation guidelines. A common scale: 0 (not relevant), 1 (related but not what user wants), 2 (relevant), 3 (highly relevant), 4 (perfect match). The scale should match the relevance definition the team uses; consistency in interpretation matters more than the specific numeric range.

Annotation guidelines. Written guidelines explain what each grade means with concrete examples and edge cases. Without guidelines, different assessors interpret "relevant" differently and inter-annotator agreement is low. With guidelines, agreement is higher, judgments are reproducible, and disputes about specific judgments can be resolved by reference to the guideline. Annotation guideline development is itself a substantial discipline; production teams iterate on guidelines as edge cases are discovered.

Inter-annotator agreement. Measure how consistently different assessors grade the same query-document pairs. Common metrics: Cohen's kappa or weighted kappa for graded scales. Agreement scores below 0.6 suggest the annotation task or guidelines need work; scores above 0.8 are excellent. Low agreement means the judgment list is noisy; high agreement means it's reliable. Production teams typically maintain agreement targets and adjust guidelines when agreement drifts.

Maintenance. Judgment lists go stale. The corpus changes (new products, retired documents). User queries evolve (seasonal shifts, new terms). Production system changes alter what gets surfaced. A judgment list that worked well last quarter may not capture current quality concerns. Maintenance patterns: quarterly judgment refresh for active deployments; add new queries based on production trends; re-judge documents whose content changed; retire queries that no longer reflect current traffic.

Every production search team that does offline evaluation needs judgment lists. The investment is substantial but compounds across all subsequent tuning work; the alternative is each evaluation reinventing its own judgments. The pattern is foundational rather than optional.

Alternatives — pure online evaluation (A/B testing) for teams that can't justify judgment list investment but have enough traffic to support fast online experiments. LLM-as-judge (Section C) as a lower-cost approximation, with the caveats covered there. Most teams use judgment lists as the foundation and supplement with online evaluation; pure-online is rare in production search.

• Voorhees, "Variations in relevance judgments and the measurement of retrieval effectiveness" (TREC methodology, 1998)
• Manning, Raghavan, Schütze, Introduction to Information Retrieval, ch. 8 on evaluation
• Trey Grainger, AI-Powered Search, chapters on relevance judgment
• OpenSource Connections Quepid (quepid.com) for judgment list management

Code

// Example judgment list format (CSV)
// query, document_id, grade, judged_at, assessor_id
// Grade scale: 0=not relevant, 1=related, 2=relevant, 3=highly
relevant

query,document_id,grade,judged_at,assessor
"running shoes",prod_12345,3,2026-04-15,assessor_01
"running shoes",prod_12346,2,2026-04-15,assessor_01
"running shoes",prod_12347,0,2026-04-15,assessor_01
"running shoes",prod_98765,3,2026-04-15,assessor_02
"trail running shoes",prod_12345,2,2026-04-15,assessor_01
"trail running shoes",prod_45678,3,2026-04-15,assessor_01

// Example pooling script (Python pseudocode)
def build_judgment_pool(queries, candidate_systems, pool_depth=20):
pool = defaultdict(set)
for query in queries:
for system in candidate_systems:
results = system.search(query, top_k=pool_depth)
for doc_id in results:
pool[query].add(doc_id)
return pool

// Example inter-annotator agreement check (Python)
from sklearn.metrics import cohen_kappa_score
agreement = cohen_kappa_score(
assessor_a_grades,
assessor_b_grades,
weights=\'quadratic\' // for graded scales
)
print(f"Weighted kappa: {agreement:.3f}") // target > 0.6, ideally
> 0.8

NDCG and discounted gain metrics #

Source: Järvelin and Kekäläinen, "Cumulated gain-based evaluation of IR techniques" (ACM TOIS, 2002); ubiquitous in modern search evaluation

Classification — The dominant offline ranking metric — graded relevance combined with logarithmic position discount.

Score a ranked result list by combining the relevance grades of its results with a position discount that rewards relevant results appearing higher, normalized to enable comparison across queries with different ideal scores.

Search quality involves both "were good results found" and "were they at the top." Simpler metrics handle one or the other: precision-at-K handles "were good results found in the top K" but ignores position within the top K; reciprocal rank handles "where was the first good result" but ignores everything after. NDCG captures both dimensions in a single principled metric with graded relevance support, making it the default offline evaluation metric for modern search systems.

DCG computation. For a ranked result list, DCG (Discounted Cumulative Gain) sums the relevance gain at each position divided by a logarithmic position discount. The standard formula: DCG = sum over positions i of (rel_i / log₂(i + 1)), where rel_i is the relevance grade at position i. Position 1 gets full credit (divisor = log₂(2) = 1); position 2 gets 0.63 credit; position 10 gets 0.29; position 100 gets 0.15. The discount falls off but never reaches zero, so relevant documents anywhere in the list contribute.

Variant with exponential gain. Some implementations use 2^rel - 1 instead of rel directly, exponentially weighting high-grade results. The variant emphasizes finding highly-relevant documents over finding more moderately-relevant ones. The exponential variant is the original Järvelin-Kekäläinen formulation; the linear variant is also widely used. The choice affects scores but not relative rankings of systems on most workloads.

Normalization. Raw DCG isn't comparable across queries because queries with more relevant documents have higher possible DCG. NDCG normalizes by dividing by the ideal DCG (the DCG achieved by ranking all relevant documents in optimal order). NDCG = DCG / ideal-DCG produces scores in [0, 1] that are comparable across queries.

Truncation: NDCG@K. Most production evaluation uses NDCG@K — NDCG computed over only the top K positions. NDCG@10 is the most common variant: it reflects "quality of the first page of results" for typical search UX. NDCG@5 or NDCG@3 emphasize top-of-page even more. The truncation choice should match the use case; e-commerce typically uses NDCG@10 or NDCG@20; question-answering may use NDCG@3 or NDCG@5.

Aggregation across queries. The per-query NDCG scores are typically averaged across the query set to produce a single system score. Macro-averaging (mean of per-query scores) is the standard. Sometimes median or other robust statistics are used to reduce influence of outlier queries.

Limitations. NDCG depends on judgment quality — garbage in, garbage out. NDCG assumes the relevance grades are accurate; biased or noisy judgments produce biased or noisy NDCG. NDCG also doesn't directly measure user outcomes (click-through, conversion, revenue); it measures a proxy for relevance that correlates with outcomes but isn't identical to them. Production teams typically track NDCG alongside business metrics.

Default offline evaluation metric for ranking quality across most search use cases. Especially appropriate for e-commerce, web search, enterprise search, and content discovery where graded relevance and top-K focus both matter.

Alternatives — MRR for known-item search where only the first relevant result matters; MAP for exhaustive retrieval (legal, scientific); P@K for simpler interpretability; ERR for cases where user-stopping models matter. NDCG is the default; specific alternatives apply when their specific dimensions matter more.

• Järvelin and Kekäläinen, "Cumulated gain-based evaluation of IR techniques" (2002)
• Manning et al., Introduction to Information Retrieval, section 8.4
• Grainger, AI-Powered Search, chapters on evaluation

Code

// NDCG@K implementation (Python)
import math

def dcg(grades, k=None):
"""Compute DCG over a list of relevance grades (already in ranked
order)."""
if k is not None:
grades = grades[:k]
return sum(
grade / math.log2(i + 2) # i+2 because i is 0-indexed; position is
i+1
for i, grade in enumerate(grades)
)

def ndcg_at_k(grades, ideal_grades, k=10):
"""NDCG@K: DCG of actual ranking divided by DCG of ideal
ranking."""
dcg_actual = dcg(grades, k=k)
dcg_ideal = dcg(sorted(ideal_grades, reverse=True), k=k)
return dcg_actual / dcg_ideal if dcg_ideal > 0 else 0.0

// Example usage with the judgment list from Section A
// Run the system, get document IDs in ranked order, look up grades:
query = "running shoes"
results = system.search(query, top_k=10) // returns ranked doc IDs
grades_in_results = [judgments[query].get(doc_id, 0) for doc_id in
results]
all_grades_for_query = list(judgments[query].values())

score = ndcg_at_k(grades_in_results, all_grades_for_query, k=10)
print(f"NDCG@10 for \'{query}\': {score:.3f}")

// Aggregate across queries
import statistics
per_query_scores = [
ndcg_at_k(
[judgments[q].get(doc_id, 0) for doc_id in system.search(q,
top_k=10)],
list(judgments[q].values()),
k=10
)
for q in test_queries
]
print(f"Mean NDCG@10: {statistics.mean(per_query_scores):.3f}")
print(f"Median NDCG@10: {statistics.median(per_query_scores):.3f}")

MAP, MRR, and P@K — the alternative offline metrics #

Source: Classical IR literature; Manning et al., Introduction to Information Retrieval; ubiquitous in evaluation tooling (trec_eval, ranx, pytrec_eval)

Classification — Alternative offline metrics that handle specific cases NDCG doesn't cover well.

Apply the right metric for cases where NDCG isn't the best fit: MRR for known-item search, MAP for exhaustive retrieval, P@K for simpler interpretability, ERR for user-stopping models.

NDCG is the default but isn't always the best fit. Known-item searches ("nike air max 270") want a metric that scores high if the right item is at the top and ignores everything else; NDCG's position-discount handles this but isn't as crisp as MRR. Exhaustive retrieval (legal documents, scientific papers, e-discovery) wants a metric that scores recall across all relevant documents; NDCG's position discount makes it less suited than MAP. Communication with non-technical stakeholders favors metrics that are intuitive; P@K ("8 of the top 10 were good") is more communicable than NDCG. The discipline is matching the metric to the use case rather than defaulting to one metric for everything.

MRR — Mean Reciprocal Rank. For each query, find the position of the first relevant result. Compute 1/position (1.0 for position 1, 0.5 for position 2, 0.1 for position 10, 0 if no relevant result in top K). Average across queries. The metric is tailored to known-item search: "did you put the right answer at the top, and if not, how close?" Strong for question-answering and navigational search. Weak for discovery queries where users want to explore multiple results.

MAP — Mean Average Precision. For each query, walk through the ranked results; at each position where a relevant document appears, compute the precision at that position (fraction of top-N results that are relevant for that N). Average these per-relevant-document precisions to get the Average Precision for that query. Mean across queries. The metric handles binary relevance and rewards systems that find all relevant documents, not just the top few. Strong for exhaustive retrieval (legal, scientific, e-discovery). Weak when graded relevance matters or when only top-K matters.

P@K — Precision at K. The fraction of the top K results that are relevant. Binary relevance (a document is relevant or not). Simple and interpretable. P@1, P@5, P@10 are common. Doesn't care about position within the top K; doesn't care about anything past K. Good for high-level communication; less informative than NDCG for detailed evaluation.

ERR — Expected Reciprocal Rank. Models user stopping behavior: the probability that a user stops at each position based on the result's relevance. A highly relevant result early in the list "absorbs" the user's attention; documents below it have lower expected impact. ERR is more theoretically principled than NDCG for modeling user behavior; less widely adopted because the marginal benefit over NDCG is small for most workloads and the additional complexity isn't justified.

Recall@K. The fraction of all relevant documents that appear in the top K. Used in retrieval evaluation (first-stage retrieval in cascade architectures, Volume 1 Section D) where the question is whether relevant documents make it into the candidate set, not their final ranking. Recall@100 or Recall@1000 are common for first-stage retrieval; recall in the candidate set bounds what the reranker can achieve.

When to use multiple. Production teams typically track multiple metrics in parallel: NDCG@10 as the headline offline metric; MRR for navigational query subsets; P@5 for stakeholder communication; Recall@100 for first-stage retrieval evaluation. Each metric catches what the others miss; using them together produces fuller evaluation than any one alone.

MRR: known-item search, question-answering, navigational queries. MAP: exhaustive retrieval, legal/scientific search, e-discovery. P@K: communication with non-technical stakeholders, top-of-page evaluation. ERR: research-grounded evaluation methodology, cases where the user-stopping model matters. Recall@K: first-stage retrieval evaluation in cascade architectures.

Alternatives — NDCG@K as the default for graded-relevance, top-K-focused use cases (most modern production search). The metrics in this entry are alternatives for specific cases; for general search evaluation, NDCG remains the default.

• Manning et al., Introduction to Information Retrieval, ch. 8
• Chapelle and Zhang, "Expected Reciprocal Rank for Graded Relevance" (CIKM 2009)
• trec_eval reference implementation (github.com/usnistgov/trec_eval)
• pytrec_eval / ranx Python implementations

Explicit expert labeling #

Source: Search relevance practitioner methodology; Quepid (OpenSource Connections); enterprise search teams at major e-commerce and content companies

Classification — Judgment collection by in-house specialists or trained domain experts who assign relevance grades according to annotation guidelines.

Produce high-quality relevance judgments by using assessors who understand the domain, the relevance definition, and the edge cases, accepting higher cost in exchange for higher quality.

Crowdsourced labels are cheap but noisy. Implicit signals are scaleable but biased. LLM-as-judge is fast but encodes model biases. For some uses — small gold sets validating other judgment sources, high-stakes domains where lower-quality judgments are unacceptable, calibration of annotation guidelines — the quality of expert labeling is necessary. Expert labeling can't scale to the volume that crowdsourcing handles, but for the cases that need it, nothing else substitutes.

Assessor selection. Domain expertise matters: e-commerce search assessors should understand the product domain; legal search assessors should understand legal relevance; healthcare assessors should have appropriate clinical knowledge. The assessors are typically in-house staff (search quality team, product specialists) or contracted SMEs (consultants, retired experts). Throughput is low — 50–200 judgments per hour per assessor depending on domain complexity — and cost is high (assessor time at SME rates).

Annotation tooling. Quepid (OpenSource Connections, free open-source) is the leading judgment management tool. The tool presents query-document pairs to assessors with the document content displayed; assessors assign grades with single-key input; the tool tracks assessor identity, timestamps, and inter-annotator agreement. Custom tools built on internal infrastructure are common for teams with specific requirements. The tool matters: a well-designed tool can double assessor throughput vs. a poorly-designed one.

Calibration sessions. Before independent labeling, assessors work through shared examples together, discussing edge cases and arriving at consistent interpretations of the relevance scale. The session produces calibration: assessors who have done this together agree more often than assessors who haven't. Recalibration sessions monthly or quarterly maintain agreement over time as new edge cases emerge.

Quality measurement. Inter-annotator agreement (Cohen's kappa or weighted kappa for graded scales) measures how consistently different assessors grade the same items. A subset of items is judged by multiple assessors to enable the measurement. Agreement scores below 0.6 trigger review of guidelines or recalibration; scores above 0.8 indicate the labeling task is well-defined and assessors are aligned.

Workload management. Expert assessors are expensive; their time should be used on the highest-value judgments. Patterns: judge new queries that production logs surface; judge query-document pairs near decision boundaries (where current and candidate systems disagree); judge the gold set used to validate other judgment sources; spot-check crowdsourced or LLM-judge outputs. Production teams typically have explicit workload prioritization.

Documentation. Maintain written annotation guidelines that capture the relevance definition, scale interpretation, edge cases, and decision rules. The guidelines are versioned; changes to guidelines may require re-judgment of affected items. The guidelines are the institutional knowledge of the search team's relevance discipline; without them, expert labeling doesn't survive assessor turnover.

Small gold sets (50–500 queries) used to validate other judgment sources. High-stakes domains (legal, medical, regulated) where lower-quality judgments are unacceptable. Calibration of annotation guidelines that other methods will follow. Edge cases requiring domain expertise to judge correctly. Annual or semi-annual high-quality evaluation that gold-standard methods support.

Alternatives — crowdsourced labeling for larger judgment volume at lower cost (next entry). Implicit signals for very large scale. LLM-as-judge for fast iteration. Expert labeling is the highest-quality method; the alternatives substitute for scale or cost reasons.

• Quepid (quepid.com / github.com/o19s/quepid) for judgment management
• OpenSource Connections methodology writings
• Trey Grainger, AI-Powered Search, chapters on relevance judgment workflow

Implicit signals and click-based judgments #

Source: Joachims et al. on click-based relevance learning (KDD 2002 and successors); production practice at all major web search and e-commerce companies

Classification — Using production click data, dwell time, and conversion signals as judgment proxy.

Extract relevance signal from production user behavior at scale, accepting that the signal is biased and requires modeling to interpret correctly, in exchange for judgment volume that explicit labeling can't produce.

Production search produces enormous volumes of user behavior data: queries, clicked results, dwell times on landing pages, conversions, abandoned searches. This data could in principle replace explicit judgment lists — it's real user signal at scale. The challenge: the data is heavily biased. Users click what was shown to them in the order it was shown; click data reinforces the current system's biases rather than measuring an external truth. Section E covers click models and counterfactual evaluation — the methodology for extracting unbiased relevance signal. This entry covers the patterns for collecting and using the raw signal.

Signals to log. Beyond clicks, production search should log: query text, time of query, user identifier (if available), what was shown (results, their positions, presentation features like rich snippets), what was clicked (which results, in what order), dwell time on each click (how long before user returned to results), subsequent actions (next query if reformulating; abandonment if leaving; conversion if purchasing or completing the task). The richer the logging, the more analyses become possible.

Click as relevance proxy. The simplest interpretation: clicked results are more relevant than non-clicked results. This works partially — users do tend to click what they find relevant — but with heavy bias: users click position 1 more than position 10 regardless of relevance; users click results with rich snippets more than plain ones; users click results from known brands more than unknown ones. Naive click-as-relevance produces self-reinforcing failure modes.

Dwell time. Users who click a result and stay on the landing page typically found something useful; users who click and immediately return (bounce) often didn't. Dwell time partially compensates for the click signal's noise: a click followed by long dwell is stronger evidence of relevance than a click followed by quick return. The pattern is sometimes called "satisfied click" vs. "unsatisfied click."

Conversion as ground truth. For e-commerce, the conversion (purchase) is the strongest relevance signal: users converted because they found what they wanted. For enterprise search, the analogous signal might be "task completed without further searches" or "result shared / forwarded." Conversion-based judgments are sparse (only a fraction of searches result in conversion) but strong; they're typically used alongside denser click-based judgments.

Query reformulation as negative signal. Users who reformulate their query after seeing results typically didn't find what they wanted. Sequential queries ("running shoes" → "men's running shoes" → "nike running shoes") signal that earlier results weren't satisfactory. Reformulation patterns are a rich source of implicit negative judgment; analyzing them surfaces queries where the system is underperforming.

Aggregation patterns. Raw click data is per-impression; useful evaluation aggregates across many impressions. Patterns: click-through rate (CTR) per query per position; mean reciprocal rank of clicks; satisfied click rate; conversion rate per query class. The aggregation level depends on the analysis: query-level for tuning specific high-volume queries; position-level for identifying ranking biases; segment-level for personalization analysis.

Privacy and ethics. User behavior data is sensitive. Logging should comply with privacy regulations (GDPR, CCPA, sector-specific rules), retention policies, and user consent frameworks. Anonymization and aggregation matter; raw per-user behavior should be handled with appropriate access controls. The discipline overlaps with the agentic AI series' compliance volume; for search specifically, the ethical handling of user behavior data is foundational.

Production search systems with sufficient query volume to produce meaningful aggregate signals (typically thousands of queries per day or more). Continuous monitoring and evaluation that requires more data than explicit labeling can produce. Identification of underperforming queries that explicit labeling might not cover. Personalization and segment-specific evaluation.

Alternatives — explicit labeling (prior entry) for gold sets and high-stakes evaluation. Implicit signals supplement explicit labeling rather than replacing it; the bias correction (Section E) is essential when implicit signals drive decisions.

• Joachims, "Optimizing Search Engines using Clickthrough Data" (KDD 2002)
• Joachims et al., "Accurately Interpreting Clickthrough Data as Implicit Feedback" (SIGIR 2005)
• Production methodology writings from web search and e-commerce teams

LLM-as-judge for relevance labeling #

Source: Emerging methodology through 2023–2026; Thomas et al. (Microsoft) on LLM relevance assessment; vendor and practitioner reports

Classification — Using large language models to generate relevance judgments for query-document pairs.

Generate relevance judgments at scale using LLMs as automated assessors, accepting model-specific biases in exchange for low cost and high throughput, with explicit validation against expert gold sets.

Explicit labeling is high-quality but expensive and slow. Implicit signals are scaleable but biased and require modeling infrastructure. Crowdsourced labels are intermediate on both dimensions but require management overhead. LLM-as-judge fills a gap: lower cost than explicit labeling, faster than crowdsourcing, no production data dependency like implicit signals. The trade-off is that LLM judgments encode model biases that may not match the team's relevance definition; validation against expert labels is essential to know when LLM judgments are reliable enough to use.

Prompt design. The LLM is given the query, the document content, and the relevance scale with annotation guidelines. The model produces a grade. Prompt quality substantially affects judgment quality; vague prompts produce inconsistent grades, detailed prompts with examples produce more consistent and accurate grades. Production teams iterate on prompts using the gold set as validation.

Few-shot examples. Include examples in the prompt showing how each grade should be applied. The examples calibrate the model's interpretation of the scale. Without examples, models may interpret "relevant" differently than the team intends; with examples, the interpretation is more aligned. Example selection matters: include examples covering common cases, edge cases, and known difficult cases.

Model selection. Larger and more capable models typically produce better judgments. Claude Opus, GPT-5, Gemini family at full capability all produce reasonable judgments on standard relevance tasks. Smaller models can work for narrow domains with good prompting. The choice trades cost per judgment against judgment quality; the gold set validation tells you whether the quality is sufficient.

Validation against gold sets. Before relying on LLM-as-judge, validate against a small expert-labeled gold set: how often does the LLM's grade match the expert's grade? Agreement metrics (the same kappa metrics used for inter-annotator agreement) reveal whether the LLM is operating at expert-level consistency. Typical results in 2026: well-prompted strong models reach kappa 0.6–0.8 with experts on standard tasks, comparable to crowdsourced labels.

Drift over time. LLM models update; the same prompt may produce different grades after a model update. Production teams should: pin to specific model versions when reproducibility matters; rerun gold set validation after model updates; monitor for drift in judgment patterns.

Limitations and biases. LLMs encode biases from their training data. They may favor certain types of content, certain writing styles, certain document structures in ways that don't match the team's relevance definition. They may struggle with domain-specific relevance (legal, medical, technical) that requires expertise the model lacks. They may produce confident judgments on cases where uncertainty is appropriate. Production use of LLM-as-judge should always include monitoring for these failure modes.

Rapid iteration during development when fast judgment turnaround matters more than peak quality. Filling gaps in explicit-labeled judgment lists with comparable but cheaper labels. Pre-screening queries for expert labeling (LLM identifies likely-relevant candidates; experts confirm). Bootstrapping initial judgment lists for new domains where no labels exist yet.

Alternatives — explicit expert labeling for high-stakes judgments and gold sets. Crowdsourced labeling for cases where human assessor judgment is needed but expert is too expensive. Implicit signals at production scale. LLM-as-judge is best as a complement to these methods, not as a replacement.

• Thomas et al., "Large Language Models can Accurately Predict Searcher Preferences" (Microsoft, 2024)
• Various vendor and practitioner reports on LLM-as-judge methodology
• Anthropic and OpenAI documentation on building evaluation pipelines with LLMs

Code

// LLM-as-judge prompt template (production-quality)
// Returns a grade 0-3 for a query-document pair

const JUDGMENT_PROMPT = `
You are evaluating search result relevance for an e-commerce site
selling outdoor gear.
Rate the relevance of the product to the user\'s query on this scale:

0 - Not relevant. The product does not match what the user is looking
for.
1 - Related. The product is in a related category but is not what the
user wants.
2 - Relevant. The product matches the user\'s query and would likely
satisfy them.
3 - Highly relevant. The product is an excellent match for the query.

Examples:

Query: "running shoes"
Product: "Nike Pegasus 40 Men\'s Running Shoes"
Grade: 3 (highly relevant - directly matches the query)

Query: "running shoes"
Product: "Nike Pegasus 40 Women\'s Running Shoes"
Grade: 3 (highly relevant - matches except for gender specification)

Query: "running shoes"
Product: "Adidas Ultraboost Casual Sneakers"
Grade: 1 (related - shoes but not running shoes specifically)

Query: "running shoes"
Product: "Running Belt for Phone and Keys"
Grade: 0 (not relevant - accessory for running, not shoes)

Now rate the following:

Query: "{query}"
Product: "{document_title}"
Description: "{document_description}"

Respond with ONLY a single digit (0, 1, 2, or 3) and no other text.
`;

async function llmJudge(query, document) {
const response = await anthropic.messages.create({
model: \'claude-opus-4-7\',
max_tokens: 5,
messages: [{
role: \'user\',
content: JUDGMENT_PROMPT
.replace(\'{query}\', query)
.replace(\'{document_title}\', document.title)
.replace(\'{document_description}\', document.description)
}]
});
const text = response.content[0].text.trim();
const grade = parseInt(text);
if (\![0, 1, 2, 3].includes(grade)) {
console.warn(`Unexpected LLM response: ${text}`);
return null;
}
return grade;
}

// Validate against gold set
async function validateAgainstGold(goldJudgments) {
const llmGrades = [];
const expertGrades = [];
for (const { query, document, expertGrade } of goldJudgments) {
const llmGrade = await llmJudge(query, document);
if (llmGrade !== null) {
llmGrades.push(llmGrade);
expertGrades.push(expertGrade);
}
}
return cohenKappaWeighted(llmGrades, expertGrades); // target > 0.6
}

A/B testing for search #

Source: General A/B testing methodology adapted for search; Kohavi et al., "Trustworthy Online Controlled Experiments" (2020); production methodology at major search companies

Classification — Online evaluation pattern — split production traffic between systems, compare outcomes statistically.

Measure whether a candidate search system produces better real-user outcomes than the current system by splitting production traffic and comparing per-user metrics with statistical rigor.

Offline metrics correlate with user outcomes but don't guarantee them. A change that improves NDCG might not improve clicks or conversions; conversely, a change that doesn't affect NDCG might substantially improve user outcomes through subtle effects offline metrics miss. A/B testing measures actual user outcomes, providing the closest available approximation to ground truth about whether a change helps. The cost: time (experiments need to run long enough for statistical significance), users (some users see the candidate system, which may be worse than the current), and operational complexity (running two systems in parallel).

Experimental design. Define the metric of interest (CTR, conversion rate, revenue per session, satisfaction proxy). Define the user population to include (all users? specific segments?). Define the split (50/50 is standard; smaller candidate share for high-risk changes). Define the duration (long enough for statistical power; not so long that user experience degradation accumulates).

Statistical power calculation. Before running, calculate how many users (or sessions, or queries) are needed to detect a meaningful effect with sufficient statistical confidence. The calculation depends on baseline metric variance, expected effect size, desired significance level (typically p < 0.05), and desired power (typically 0.80). High-traffic search can detect 1% effects in days; low-traffic search may need weeks or months for the same detection.

Randomization. Users (or sessions, or other units) are assigned to control or treatment randomly. The randomization must be stable across the experiment (same user sees the same system throughout). Random assignment is what makes statistical inference valid; non-random assignment (e.g., showing the new system to early adopters) confounds the experiment.

Outcome measurement. Track the primary metric and a set of guardrail metrics that could indicate unintended consequences. Primary might be conversion rate; guardrails might be revenue per user, query reformulation rate, bounce rate, latency. Negative guardrail movement may make a positive primary movement unacceptable.

Statistical analysis. At experiment end, compute the metric for each arm; compute the difference; compute the statistical significance (typically via t-test, chi-square, or appropriate non-parametric test depending on metric type). Report effect size with confidence interval, not just p-value. The discipline of statistical interpretation matters; rushed conclusions from underpowered experiments produce false-positive deployments.

Sequential testing pitfalls. Running an experiment longer or peeking at results midway changes the statistical guarantees. Sequential testing methods (or Bayesian methods) address this; naive "watch the experiment and stop when significant" inflates false positive rates. Production teams should set the experiment duration upfront based on power analysis and resist the temptation to read results early.

Holdout populations. Some users (1–5%) may be excluded from experiments entirely as a long-term holdout: they see the current production system always. The holdout enables measurement of cumulative impact ("we've shipped 10 experiments; are users in the cumulative-shipped arm doing better than holdout?") that experiment-by-experiment analysis can't capture.

System-level changes that affect more than ranking (UI changes, presentation features, filter logic, complete architectural shifts). Cases where the change's impact on user outcomes is uncertain. High-stakes deployments where the cost of being wrong about a change is high. Validation before rolling out to 100% of users.

Alternatives — interleaving (next entry) for pure ranking comparisons, where its statistical efficiency reduces required sample size by an order of magnitude. Offline evaluation for changes that don't need online validation (small changes that offline metrics handle well). Multi-armed bandit methods for cases where minimizing exposure to worse variants matters more than rigorous statistical comparison.

• Kohavi, Tang, Xu, Trustworthy Online Controlled Experiments (2020)
• Daniel Tunkelang's writing on A/B testing for search
• Vendor documentation: Coveo experimentation, Algolia A/B testing

Interleaving (TDI and successors) #

Source: Joachims, "Evaluating Retrieval Performance using Clickthrough Data" (2003); Radlinski, Kurup, Joachims, "How Does Clickthrough Data Reflect Retrieval Quality?" (CIKM 2008); Schuth, "Multi-Leaved Comparisons for Fast Online Evaluation" (2014)

Classification — Online evaluation pattern — blend two systems' rankings into a single result set per user, track which system's contributions get clicked.

Compare two ranking systems with much higher statistical efficiency than A/B testing by having each user effectively serve as their own experiment — seeing results from both systems and clicking those they prefer.

A/B testing for ranking changes is statistically inefficient: each user sees only one system, so detecting which is better requires many users for statistical significance. Interleaving solves the inefficiency: each user sees results from both systems merged into one list, and clicks on each system's contributions are direct evidence of which system's ranking that user preferred. Statistical signal per user is much stronger; experiments reach significance with roughly an order of magnitude fewer users than equivalent A/B tests.

Team-Draft Interleaving (TDI). The canonical interleaving algorithm. Each ranking is treated as a team; the merged list is built by drafting: a coin flip determines which team picks first; that team adds its top result; the other team adds its top result; alternate until the result list is complete. Documents that both systems would have shown go to whichever team "drafted" them first; documents only one system would have shown go to that system's team. Each result in the merged list is attributable to one team.

Click attribution. When the user clicks a result, the click is credited to the team that contributed that result. Aggregate over many user impressions: if Team A's contributions get more clicks than Team B's, that's evidence A produced better results. Statistical inference uses the per-impression team-comparison as the unit of analysis.

Tie-breaking. Documents both systems rank highly create ties that need handling. TDI handles this via the draft order (first team to want a tied document gets it). Probabilistic Interleaving handles it differently. The choice affects experiment behavior; the canonical TDI handles most cases well.

Multi-leaved comparisons. The extension to more than two systems: instead of two teams, N teams contribute to the merged list. Each user sees results from N systems blended together; clicks are attributed across all N. The extension allows comparing many candidate rankings simultaneously, multiplying interleaving's efficiency advantage.

Production engineering. Interleaving requires real-time merging of two (or more) systems' rankings, real-time tracking of which system contributed each result, and click logging that associates clicks with contributing systems. The engineering is non-trivial; production teams that have implemented it consider the investment worthwhile because of the experimentation throughput it enables.

Limitations. Interleaving only compares rankings; non-ranking changes (different UX, different filters) can't be tested via interleaving. Interleaving assumes users perceive a single result list; if the UX separates contributions visually, the interleaving signal breaks down. Production teams typically use interleaving for ranking experiments and reserve A/B testing for system-level changes.

Comparing two or more candidate ranking algorithms. Comparing different parameter settings within the same algorithm. Quick experiments on smaller-traffic search systems where A/B testing would take too long. Sequential experimentation where running many ranking variants is feasible if each individual experiment is short.

Alternatives — A/B testing (prior entry) for non-ranking comparisons. Offline evaluation for changes that offline metrics handle adequately. The combination of interleaving for ranking and A/B for system-level changes is the working pattern for mature production search teams.

• Joachims, "Evaluating Retrieval Performance using Clickthrough Data" (2003)
• Radlinski, Kurup, Joachims, "How Does Clickthrough Data Reflect Retrieval Quality?" (CIKM 2008)
• Schuth et al., "Multi-Leaved Comparisons for Fast Online Evaluation" (CIKM 2014)
• Hofmann, Whiteson, de Rijke writings on interleaving and online learning

Click models for bias correction (PBM, Cascade, DBN) #

Source: Craswell et al., "An Experimental Comparison of Click Position-Bias Models" (WSDM 2008); Chapelle and Zhang, "Dynamic Bayesian Network Click Model" (WWW 2009); Chuklin, Markov, de Rijke, Click Models for Web Search (2015)

Classification — Probabilistic models of user click behavior that separate relevance signal from presentation bias.

Model the probability that a user clicks a result as a function of the result's relevance and its position (and other presentation features), so that observed clicks can be decomposed into the underlying relevance signal and the bias from how the result was presented.

Users click position 1 about 30–40% of the time on web search regardless of how relevant position 1 actually is, simply because users see it first. Position 10 gets clicked maybe 2–3% of the time, again regardless of relevance. The position effect dwarfs any actual relevance differences for non-top positions; aggregating raw click-through rates as relevance signal treats position bias as if it were relevance. The result: ranking models trained on raw click data learn to put already-high-ranked documents higher, ignoring the relevance signal underneath. Click models address this by explicitly modeling the position effect (and other biases) so the relevance signal can be extracted.

Position-Based Model (PBM). The simplest useful click model: P(click | rank, query, doc) = P(examine | rank) × P(click | examine, query, doc). The examine probability depends only on rank; the click-given-examine probability depends only on the query-document relevance. The decomposition lets you estimate per-position examination probabilities from production data, then divide observed clicks by the examination probabilities to recover relevance signal. PBM is widely used as a baseline; more sophisticated models extend it.

Cascade Model. Models the user as walking down the result list: they examine position 1; if it satisfies them, they click and stop; if not, they move to position 2 and repeat. The model captures the intuition that lower positions get less examination because higher positions absorb attention. Cascade is more accurate than PBM for navigational queries (where users typically click the first satisfying result and stop) but less accurate for queries where users explore multiple results.

Dynamic Bayesian Network (DBN). Chapelle and Zhang's more sophisticated model: separate variables for "user examined this position" and "this result was perceived relevant" and "user was satisfied". Captures the case where users continue past results they perceived as relevant if those results turned out not to fully satisfy. DBN handles a wider range of user behavior than simpler models at the cost of more parameters to estimate and more data needed for stable estimates.

Estimating model parameters. The models have parameters (per-position examination probabilities, etc.) that must be estimated from production data. Standard approach: expectation-maximization (EM) on production click logs. The EM iterates between estimating per-query-document relevances given current parameter estimates and estimating parameters given current relevance estimates. Convergence produces stable estimates of both relevances and biases.

Inverse Propensity Scoring (IPS). Once the per-position examination probability is estimated, observed clicks can be reweighted: a click at position 10 (low examination probability) is worth more evidence than a click at position 1 (high examination probability), because the position-10 click is more surprising and more likely to reflect genuine relevance. Joachims and colleagues developed the IPS approach to counterfactual evaluation; the methodology lets production click data drive offline evaluation in unbiased ways.

Limitations. Click models capture some biases but not all. Trust bias (users click results from known sources more), presentation bias (results with rich snippets get more clicks), and brand bias all add complexity that simple position-bias correction doesn't address. More sophisticated models (UBM, DCM, others) handle additional biases at the cost of more complexity. The methodology has limits; production teams calibrate models against ground-truth experiments where available.

Production search teams using implicit signals (Section C) at scale who need to correct for position and other biases. Offline log replay evaluation (Chapter 2) where click prediction is needed for systems that didn't generate the original logs. Counterfactual evaluation: "what would users have done if we'd shown them this candidate ranking instead?"

Alternatives — explicit judgment-based evaluation when implicit signals' biases are hard to handle. Online evaluation (A/B testing or interleaving) that doesn't need counterfactual reasoning because actual user behavior is observed. Click models are essential when implicit signals are the primary evaluation source; they're less needed when explicit judgments or online tests are available.

• Craswell et al., "An Experimental Comparison of Click Position-Bias Models" (2008)
• Chapelle and Zhang, "Dynamic Bayesian Network Click Model" (2009)
• Chuklin, Markov, de Rijke, Click Models for Web Search (Morgan & Claypool, 2015)
• Joachims, Swaminathan, Schnabel, "Unbiased Learning-to-Rank with Biased Feedback" (WSDM 2017)

Golden query sets and continuous evaluation #

Source: Production methodology at major search companies; OpenSource Connections methodology writings; Quepid for judgment set management

Classification — Pattern for continuous evaluation against curated query sets with alerting on regression.

Detect search quality regressions automatically by running curated query sets against the current system frequently (daily or per-deployment) and alerting when metrics fall outside expected ranges.

Production search quality degrades silently. Code changes, configuration changes, model updates, index changes, and corpus changes can all cause regressions that don't surface as user complaints for weeks. By the time the regression is visible in business metrics, the team has lost time and trust. Continuous evaluation catches the regression early: a daily run that fails alerts the team before users notice.

Golden query set composition. A small set (50–500 queries) of carefully chosen queries with known-good expected results. The queries cover: high-value queries (top traffic), edge cases (specific failure modes the team has fixed and wants to keep fixed), and representative queries across query classes. The set is smaller than the full judgment list (Section A) because continuous evaluation runs frequently and must be fast.

Hard query sets. A subset of the golden set focused specifically on queries known to be difficult — queries where the current system performs adequately but candidate changes have historically regressed. The hard set is the regression-test set: it catches the easy-to-introduce regressions that simpler queries might miss. New hard queries are added when a regression escapes other tests.

Run cadence. Daily for active production search; per-deployment for high-velocity teams; weekly for stable systems. The cadence should match the rate of change in the system; static systems need less frequent evaluation than systems under active development. The cadence shouldn't be so frequent that it overwhelms operations with false-positive alerts.

Per-query alerting. Aggregate metrics (mean NDCG@10) hide per-query regressions: one query dropping from 0.95 to 0.10 may not move the mean noticeably. Per-query alerting catches these: if any individual query's score drops by more than a threshold (e.g., 0.2), alert. The pattern catches regressions that aggregate alerting misses.

Aggregate alerting. Per-query alerting catches specific failures; aggregate alerting catches systemic ones. Track per-day aggregate NDCG over time; alert when the rolling average drops by more than a threshold. The combination of per-query and aggregate alerting catches both kinds of regression.

Alert routing. Alerts should reach the right people quickly. Search quality team or on-call engineer for immediate regression investigation; relevant feature team if the regression correlates with their deployment; broader notification if the regression affects business metrics. The routing infrastructure is operations practice, not specific to search; the application to search is what matters.

Investigation workflows. When an alert fires, the team needs to diagnose: was it a deployment? a model update? a corpus change? an external dependency? Searchable history of metric values over time supports the diagnosis. Tools like Splainer (OpenSource Connections) let engineers inspect why specific queries returned the results they did, supporting root-cause analysis. The diagnostic workflow is itself a substantial discipline; the future Search Operations Catalog covers it in depth.

Regression budget. Not every regression must be fixed immediately; some may be acceptable trade-offs for other improvements. Production teams typically maintain a regression budget: how much quality degradation is acceptable in exchange for what kind of capability gain? Explicit budgets enable principled decisions rather than ad-hoc "is this regression OK?" negotiations.

Production search above toy scale where quality matters. Teams with frequent deployments where each deployment could regress quality. Mature search systems where the cost of undetected regressions is significant. Any team with judgment lists (Section A) substantial enough to support frequent evaluation.

Alternatives — less frequent evaluation (weekly or monthly) for stable systems with low deployment frequency. Pure online monitoring (production click metrics) for teams without judgment list infrastructure. The continuous-evaluation pattern is best suited to teams with both judgment list infrastructure and active development; teams missing either may use lighter patterns.

• Production methodology writings from search teams at major companies
• OpenSource Connections methodology blog posts on continuous relevance testing
• Quepid documentation on judgment set management and test cases

Custom business metrics for search #

Source: Production methodology at e-commerce, enterprise search, and content search companies; Tunkelang's writing on search business metrics

Classification — Patterns for translating relevance evaluation into business outcomes.

Measure search quality through metrics that map directly to business outcomes — revenue, conversion, task completion, satisfaction — rather than only through academic proxy metrics.

Academic relevance metrics are useful proxies but not business outcomes. Stakeholders outside the search team typically don't care about NDCG; they care about revenue, conversion, customer satisfaction, productivity. Communicating search quality in business terms requires custom metrics that bridge the gap. Without these metrics, the search team's quality work is hard to justify to broader leadership, and trade-offs between search investment and other priorities can't be made on common ground.

E-commerce business metrics. Revenue per query (gross merchandise value generated per search session, attributed to search). Conversion rate (fraction of search sessions resulting in purchase). Add-to-cart rate. Search-to-purchase time (how long from query to purchase, lower is better for most use cases). Average order value for search-originated sessions. Repeat purchase rate for customers acquired via search. The business metrics combine with relevance metrics to give a fuller picture: revenue impact correlates with relevance but isn't identical to it.

Enterprise search business metrics. Task completion rate (fraction of searches that result in the user finding what they needed). Time to result (median time from query to clicking a result that satisfies). Repeat query rate (fraction of searches that lead to query reformulation, lower is better). Self-service deflection rate (fraction of searches that prevent a support ticket or escalation). Productivity proxies (estimated time saved across search users).

Content search business metrics. Click-through rate by content type. Dwell time on clicked content. Share rate (content shared after being found through search). Repeat visit rate (users returning to find content again). Subscription/conversion attribution when content drives signups.

Attribution modeling. Business metrics require attribution: which sales/completions came from search vs. other channels? Direct attribution (the user searched and immediately converted) is straightforward. Indirect attribution (the user searched, browsed, returned later, and converted) is harder. Attribution models (last-click, first-click, multi-touch) produce different answers; the choice affects how much credit search gets and shapes investment decisions.

A/B test alignment. Online evaluation (Section D) ideally measures both relevance metrics and business metrics. If the candidate system improves NDCG but degrades conversion, the offline-online correlation has broken down and the change shouldn't ship. Production teams typically require both relevance and business metrics to improve (or business to improve and relevance to not regress) before deploying changes from experiments.

Communication and reporting. Business metrics drive executive reporting and budget discussions. A monthly or quarterly search quality report should include: business metric trends, relevance metric trends, notable wins and losses, hypotheses about drivers, planned next steps. The report's audience is broader than the search team; the metrics chosen should communicate to that audience.

Custom dashboards. Production teams typically maintain dashboards combining academic and business metrics over time. Grafana or Datadog dashboards aggregating from production logging infrastructure. Coveo, Algolia, and similar vendors provide built-in dashboards as part of their platforms. Custom dashboards for specific needs supplement vendor-provided ones.

Any production search that supports a business with measurable outcomes (e-commerce, SaaS, content, enterprise). Teams that need to justify search investment to leadership. Cases where academic metrics aren't persuasive to stakeholders. Long-term tracking of search's business impact.

Alternatives — academic metrics alone may suffice for research or pure-relevance development contexts where business outcomes aren't the immediate concern. For all production contexts with business goals, custom business metrics are essential alongside academic ones.

• Daniel Tunkelang's writing on search business metrics
• Coveo and Algolia documentation on built-in business metric tracking
• Production methodology from major e-commerce search teams (Etsy, Wayfair, others have published in this area)

Resources for tracking search evaluation discipline #

Source: Multiple academic, practitioner, and vendor sources

Classification — Sources for staying current on search evaluation practice.

Provide pointers to the active sources of search evaluation knowledge: foundational texts, academic and industry conferences, practitioner blogs, tools, communities.

Search evaluation practice spans academic literature, industry methodology, vendor tooling, and emerging techniques (LLM-as-judge, advanced click models). Production teams need ongoing engagement with multiple sources to keep their evaluation discipline current.

Foundational texts. Manning, Raghavan, Schütze, Introduction to Information Retrieval (free online at nlp.stanford.edu/IR-book) — ch. 8 on evaluation remains the canonical academic reference. Grainger, AI-Powered Search (Manning, 2024) — extensive coverage of modern evaluation methodology. Turnbull and Berryman, Relevant Search (Manning, 2016) — practical relevance engineering with evaluation throughout. Chuklin, Markov, de Rijke, Click Models for Web Search (2015) — the canonical reference on click modeling. Kohavi, Tang, Xu, Trustworthy Online Controlled Experiments (Cambridge, 2020) — A/B testing methodology applied to search.

Academic conferences. SIGIR (ACM Special Interest Group on Information Retrieval) is the premier venue for evaluation research. CIKM (Conference on Information and Knowledge Management) covers evaluation methodology adjacent to broader IR. WSDM (Web Search and Data Mining) focuses on web-scale problems including evaluation. ECIR (European Conference on Information Retrieval) is the European counterpart. TREC (Text Retrieval Conference) is both a venue and a long-running evaluation infrastructure.

Industry conferences. Haystack (haystackconf.com) is the premier practitioner conference for relevance engineering, organized by OpenSource Connections; evaluation is heavily represented. AI-Powered Search Conference (related to Grainger's book). Berlin Buzzwords covers search and data engineering. Spark + AI Summit, MLOps World, and similar venues cover evaluation infrastructure alongside broader ML evaluation.

Practitioner writing. Daniel Tunkelang (dtunkelang.medium.com) on search evaluation strategy. OpenSource Connections (opensourceconnections.com) on practical relevance engineering. Doug Turnbull's writing across multiple venues. Trey Grainger's ongoing writing on AI-powered search. Search team blogs at major companies (Etsy, Wayfair, Spotify, GitHub) periodically publish detailed evaluation methodology.

Tools. Quepid (quepid.com, open source) for judgment set management and offline evaluation. Splainer (also from OpenSource Connections) for query explanation and debugging. trec_eval (github.com/usnistgov/trec_eval) for standard metric computation. pytrec_eval (Python bindings to trec_eval). ranx (github.com/AmenRa/ranx) modern Python evaluation library. BEIR benchmark suite (github.com/beir-cellar/beir) for retrieval evaluation. MS MARCO dataset for training and evaluation.

Communities. Relevancy Engineering Slack (via OpenSource Connections invitation) is the primary practitioner community. Reddit r/searchengines, r/elasticsearch for casual discussion. LinkedIn groups around search engineering for professional networking. Conference attendance (Haystack especially) produces network effects that async channels can't replicate.

Emerging areas to watch. LLM-as-judge methodology continues to mature; Microsoft and other research groups are publishing increasingly. Counterfactual evaluation and unbiased learning-to-rank continue to advance. Evaluation for RAG and agentic systems is an emerging area where search evaluation methodology informs broader practice. Multimodal search evaluation (image, video, audio in addition to text) is a frontier with limited consolidated methodology.

Search engineering teams building or maintaining evaluation infrastructure. Engineers transitioning into relevance engineering from adjacent fields. Continuous learning as the discipline evolves. Reference when specific evaluation needs go beyond what existing knowledge handles.

Alternatives — specialized consulting (RelevantSearch.AI, OpenSource Connections, others) for high-stakes evaluation engagements. Internal documentation for teams with mature practice. The combination of external tracking and internal knowledge is the working pattern.

• Manning et al., Introduction to Information Retrieval (free online)
• Trey Grainger, AI-Powered Search (Manning, 2024)
• Doug Turnbull and John Berryman, Relevant Search (Manning, 2016)
• Chuklin, Markov, de Rijke, Click Models for Web Search (2015)
• Kohavi, Tang, Xu, Trustworthy Online Controlled Experiments (2020)
• Haystack Conference (haystackconf.com); SIGIR proceedings
• Quepid (quepid.com); ranx (github.com/AmenRa/ranx); trec_eval
• BEIR (github.com/beir-cellar/beir); MS MARCO; MTEB