Source: Multiple academic, practitioner, and vendor sources
Classification — Sources for staying current on ranking and relevance practice.
Provide pointers to the active sources of ranking and relevance knowledge: foundational texts, academic and industry venues, practitioner writing, open-source tools, communities.
Ranking discipline spans academic literature (information retrieval, machine learning), industry methodology (production search teams), vendor tooling (commercial rerankers, LTR frameworks), and emerging techniques (LLM-based ranking, novel architectures). Production teams need ongoing engagement with multiple sources to keep current.
Foundational texts. Liu, Learning to Rank for Information Retrieval (Foundations and Trends in IR, 2009) — the canonical LTR survey, still the best comprehensive reference. Manning, Raghavan, Schütze, Introduction to Information Retrieval (free online) — broader IR with strong ranking coverage. Grainger, AI-Powered Search (Manning, 2024) — modern hybrid ranking in production depth. Turnbull and Berryman, Relevant Search (Manning, 2016) — practical relevance engineering with lexical focus.
Academic conferences. SIGIR (ACM Special Interest Group on Information Retrieval) — the premier IR venue, where most ranking research is published. CIKM, WSDM, ECIR — adjacent venues with substantial ranking content. NeurIPS, ICML, ICLR — ML conferences where neural ranking methods appear. The TREC competitions — historical evaluation infrastructure that produced much of the methodology in this volume.
Industry venues. Haystack Conference (haystackconf.com) — the practitioner conference where ranking work is heavily represented. Berlin Buzzwords — European search and data engineering. AI-Powered Search Conference (related to Grainger's book). Vendor developer conferences (Elastic, Algolia, Coveo) periodically cover ranking patterns.
Practitioner writing. Daniel Tunkelang — long-running blog on search strategy and ranking. OpenSource Connections — practical relevance engineering content. Doug Turnbull — writing across multiple venues. Trey Grainger — ongoing AI-Powered Search content. Search team blogs at Etsy, Wayfair, Spotify, GitHub, Pinterest, and other major search-driven companies periodically publish substantial ranking methodology.
Open-source tools and code. LightGBM ranking (lightgbm.readthedocs.io) and XGBoost ranking (xgboost.readthedocs.io) — production-grade GBDT LTR. TF-Ranking (github.com/tensorflow/ranking) — listwise neural LTR. RankLib (sourceforge.net/p/lemur/wiki/RankLib/) — Java implementations of multiple LTR algorithms. Sentence-transformers (sbert.net) — cross-encoders for reranking. ColBERT (github.com/stanford-futuredata/ColBERT) — late-interaction reference implementation. Elasticsearch Learning to Rank plugin and Solr LTR contrib — production integration.
Datasets and benchmarks. MS MARCO — the canonical web search dataset for training and evaluation. BEIR — retrieval benchmark across many domains. MTEB — embedding model benchmark relevant to vector ranking. TREC datasets — historical IR benchmarks. Production teams use these as comparison points for their own systems and as training data for fine-tuning.
Communities. Relevancy Engineering Slack (via OpenSource Connections invitation) — the primary practitioner community. Reddit r/searchengines, r/elasticsearch — casual discussion. Conference attendance (Haystack especially) produces network effects.
Emerging areas to watch. LLM-based ranking continues to evolve: monoT5 and successors, RankGPT-style direct LLM ranking, listwise LLM rerankers. Counterfactual learning to rank using bias-corrected production data. Multi-modal ranking (image + text, video + text). Multi-objective LTR with explicit constraint handling. The frontier is active; tracking SIGIR and Haystack catches most of the developments.
Ranking engineers building or maintaining production ranking systems. Engineers transitioning into ranking from adjacent fields. Continuous education as the discipline evolves. Reference when specific ranking needs go beyond what current knowledge handles.
Alternatives — specialized consulting (RelevantSearch.AI, OpenSource Connections, others) for high-stakes ranking engagements. Internal documentation for teams with mature practice. The combination of external tracking and internal knowledge is the working pattern.
- Liu, Learning to Rank for Information Retrieval (2009)
- Grainger, AI-Powered Search (2024); Turnbull/Berryman, Relevant Search (2016)
- Manning et al., Introduction to Information Retrieval (free online)
- SIGIR, Haystack, Berlin Buzzwords proceedings
- LightGBM, XGBoost, TF-Ranking, sentence-transformers, ColBERT documentation
- MS MARCO, BEIR, MTEB benchmarks
- Relevancy Engineering Slack