Source: Production methodology at e-commerce, enterprise search, and content search companies; Tunkelang's writing on search business metrics
Classification — Patterns for translating relevance evaluation into business outcomes.
Measure search quality through metrics that map directly to business outcomes — revenue, conversion, task completion, satisfaction — rather than only through academic proxy metrics.
Academic relevance metrics are useful proxies but not business outcomes. Stakeholders outside the search team typically don't care about NDCG; they care about revenue, conversion, customer satisfaction, productivity. Communicating search quality in business terms requires custom metrics that bridge the gap. Without these metrics, the search team's quality work is hard to justify to broader leadership, and trade-offs between search investment and other priorities can't be made on common ground.
E-commerce business metrics. Revenue per query (gross merchandise value generated per search session, attributed to search). Conversion rate (fraction of search sessions resulting in purchase). Add-to-cart rate. Search-to-purchase time (how long from query to purchase, lower is better for most use cases). Average order value for search-originated sessions. Repeat purchase rate for customers acquired via search. The business metrics combine with relevance metrics to give a fuller picture: revenue impact correlates with relevance but isn't identical to it.
Enterprise search business metrics. Task completion rate (fraction of searches that result in the user finding what they needed). Time to result (median time from query to clicking a result that satisfies). Repeat query rate (fraction of searches that lead to query reformulation, lower is better). Self-service deflection rate (fraction of searches that prevent a support ticket or escalation). Productivity proxies (estimated time saved across search users).
Content search business metrics. Click-through rate by content type. Dwell time on clicked content. Share rate (content shared after being found through search). Repeat visit rate (users returning to find content again). Subscription/conversion attribution when content drives signups.
Attribution modeling. Business metrics require attribution: which sales/completions came from search vs. other channels? Direct attribution (the user searched and immediately converted) is straightforward. Indirect attribution (the user searched, browsed, returned later, and converted) is harder. Attribution models (last-click, first-click, multi-touch) produce different answers; the choice affects how much credit search gets and shapes investment decisions.
A/B test alignment. Online evaluation (Section D) ideally measures both relevance metrics and business metrics. If the candidate system improves NDCG but degrades conversion, the offline-online correlation has broken down and the change shouldn't ship. Production teams typically require both relevance and business metrics to improve (or business to improve and relevance to not regress) before deploying changes from experiments.
Communication and reporting. Business metrics drive executive reporting and budget discussions. A monthly or quarterly search quality report should include: business metric trends, relevance metric trends, notable wins and losses, hypotheses about drivers, planned next steps. The report's audience is broader than the search team; the metrics chosen should communicate to that audience.
Custom dashboards. Production teams typically maintain dashboards combining academic and business metrics over time. Grafana or Datadog dashboards aggregating from production logging infrastructure. Coveo, Algolia, and similar vendors provide built-in dashboards as part of their platforms. Custom dashboards for specific needs supplement vendor-provided ones.
Any production search that supports a business with measurable outcomes (e-commerce, SaaS, content, enterprise). Teams that need to justify search investment to leadership. Cases where academic metrics aren't persuasive to stakeholders. Long-term tracking of search's business impact.
Alternatives — academic metrics alone may suffice for research or pure-relevance development contexts where business outcomes aren't the immediate concern. For all production contexts with business goals, custom business metrics are essential alongside academic ones.
- Daniel Tunkelang's writing on search business metrics
- Coveo and Algolia documentation on built-in business metric tracking
- Production methodology from major e-commerce search teams (Etsy, Wayfair, others have published in this area)