Source: spaCy, Stanford NER for classical NER; BERT-NER and transformer-based NER for modern methods; production methodology at major e-commerce search companies
Classification — Methods for identifying entity spans in queries and resolving them to structured IDs.
Extract structured entity information (brands, products, categories, attributes, locations) from natural-language queries and link the extracted entities to IDs in the company's catalog or knowledge base, producing signals that retrieval and ranking can use as filters, boosts, and features.
A query like "nike air max 270 size 10 mens" contains structured information: BRAND=nike, PRODUCT_LINE=air_max_270, SIZE=10, GENDER=mens. Without entity extraction, the query is treated as a bag of words; downstream stages can match against the words but can't apply structured filters or use the structure as features. With entity extraction, the brand/size/gender become filters that narrow retrieval correctly, and the recognized entities become high-quality ranking features. The same applies across domains: travel search needs origin/destination/date; legal search needs jurisdiction/date/case-type; enterprise search needs people/teams/projects.
The two-stage architecture. NER identifies the spans; entity linking resolves them to IDs. Span identification: "nike" → BRAND, "air max 270" → PRODUCT_LINE, "size 10" → SIZE, "mens" → GENDER. Linking: BRAND span "nike" → brand_id=8421 (the catalog's ID for Nike Inc.); PRODUCT_LINE span "air max 270" → line_id=AM270. The two stages can be implemented separately (a generic NER model plus a domain-specific linker) or jointly (a model trained to predict the linked ID directly).
Classical NER. Tools like spaCy and Stanford NER provide pretrained models for standard entity types (PERSON, ORGANIZATION, LOCATION, DATE, MONEY). The pretrained models handle general cases but miss domain-specific entity types (BRAND, PRODUCT, ATTRIBUTE). For domain-specific entities, train a custom NER model on labeled examples from the domain.
Transformer-based NER. Fine-tuned BERT models (BERT-NER) handle NER as token classification: each token gets a tag (B-BRAND for "beginning of brand", I-BRAND for "inside of brand", O for "outside any entity"). The IOB tagging scheme handles multi-token entities cleanly. Fine-tuning needs labeled data: a few thousand annotated queries with entity spans typically suffices for good domain performance. The transformer architecture handles context better than classical methods, especially for ambiguous entities.
LLM-based NER. Modern LLMs handle NER with zero-shot or few-shot prompting. Provide the query and ask the LLM to identify entities: "Extract brands, products, categories, sizes, and gender mentions from this query: [...]. Return as JSON." LLM-based NER handles unusual entity types without retraining; it adapts to new entity types via prompt updates. The trade-offs are latency and cost; production deployments use LLM NER when other methods' quality is insufficient and the cost is justified.
Entity linking methods. Lookup-based: maintain a dictionary mapping surface forms to IDs ("nike", "Nike", "NIKE" → brand_id=8421). Simple, fast, requires manual maintenance. Embedding-based: encode both query entities and catalog entities into the same vector space; find nearest neighbor in catalog. Handles surface variations automatically; needs an embedding model that captures entity semantics. Learned linking: train a model that takes a query span and produces a catalog ID directly, using labeled training data. Most accurate; needs the most engineering investment.
Slot filling pattern. The pattern of extracting structured query parameters from natural-language queries. "Flights from SFO to JFK on Tuesday" becomes [origin=SFO, destination=JFK, date=2026-05-26]. Slot filling combines NER (identifying spans) with role assignment (this span is origin, that span is destination) and value normalization ("Tuesday" becomes a specific date). Modern slot filling typically uses transformer-based or LLM-based methods; the patterns from this entry apply with the addition of role-assignment logic.
Domain-specific design. Each domain has its own entity types and linking targets. E-commerce: brands, products, categories, sizes, colors, materials, gender, age-group, occasion. Travel: origins, destinations, dates, cabin classes, passenger counts. Legal: jurisdictions, case types, dates, statutes, courts. Enterprise: people, teams, projects, documents, dates. The entity inventory should be designed deliberately based on what retrieval and ranking need; over-broad entity inventories produce noise, under-broad inventories miss structured signals.
Evaluation. NER evaluation: precision (of extracted entities, what fraction are correct), recall (of true entities in queries, what fraction were extracted), F1 (the harmonic mean). Linking evaluation: of correctly extracted entities, what fraction linked to the right ID. Production deployments track both metrics and the downstream impact: do queries with successful entity extraction produce better retrieval outcomes? The downstream measurement (Volume 5 Section G covers business metrics) is the ultimate validation.
E-commerce search where structured query understanding drives substantial business value. Travel search where slot-filled parameters are essential. Legal and medical search where domain entities matter. Enterprise search where people, teams, and projects need recognition. Any domain where queries contain structured information that retrieval and ranking could use.
Alternatives — no entity extraction for narrow domains with simple query patterns. Pure lexical matching where structured signals don't add value. Most production search with diverse queries benefits from entity extraction; the level of investment depends on query complexity and downstream value.
- spaCy NER documentation (spacy.io)
- Hugging Face transformers NER documentation
- Production methodology from e-commerce search teams (Etsy, Wayfair, Amazon)
- Coveo named entity recognition documentation
Code
# E-commerce NER with spaCy + custom entity linking
import spacy
from spacy.tokens import Span
# For production: train custom NER on labeled domain data
# This example uses spaCy\'s NER with custom entity types defined
via training
nlp = spacy.load("ecommerce_ner_model") # custom-trained model
# Catalog lookup tables (in production: backed by Redis or similar)
BRAND_LOOKUP = {
"nike": "brand_8421",
"adidas": "brand_8422",
"reebok": "brand_8423",
}
CATEGORY_LOOKUP = {
"running shoes": "cat_running",
"sneakers": "cat_running", # synonym mapping at link time
"trail shoes": "cat_trail",
}
def extract_and_link_entities(query: str) -> dict:
doc = nlp(query)
entities = {
"brands": [],
"categories": [],
"sizes": [],
"colors": [],
"gender": None,
}
for ent in doc.ents:
text = ent.text.lower().strip()
if ent.label_ == "BRAND":
linked = BRAND_LOOKUP.get(text)
if linked:
entities["brands"].append({"text": text, "id": linked})
elif ent.label_ == "CATEGORY":
linked = CATEGORY_LOOKUP.get(text)
if linked:
entities["categories"].append({"text": text, "id": linked})
elif ent.label_ == "SIZE":
entities["sizes"].append(text)
elif ent.label_ == "COLOR":
entities["colors"].append(text)
elif ent.label_ == "GENDER":
entities["gender"] = text
return entities
# Example usage
query = "nike air max 270 size 10 mens"
entities = extract_and_link_entities(query)
# Expected:
# {
# \'brands\': [{\'text\': \'nike\', \'id\': \'brand_8421\'}],
# \'categories\': [],
# \'sizes\': [\'10\'],
# \'colors\': [],
# \'gender\': \'mens\'
# }
# Convert entities to retrieval filters/boosts
def entities_to_query_clauses(entities):
clauses = {"filter": [], "boost": []}
for brand in entities["brands"]:
clauses["filter"].append({"term": {"brand_id":
brand["id"]}})
if entities["gender"]:
clauses["filter"].append({"term": {"gender":
entities["gender"]}})
for size in entities["sizes"]:
clauses["filter"].append({"term": {"size": size}})
return clauses