Source: Apache Lucene; Elasticsearch / OpenSearch / Solr analyzer documentation; production methodology
Classification — Patterns for running analyzer chains at index time, including symmetric (same chain at index and query) and deliberately asymmetric (different chains) configurations.
Apply analyzer chains at index time to produce the tokens that retrieval will match against, with deliberate choices about whether to use the same chain at query time (symmetric) or to use a different chain for specific match behaviors (asymmetric).
The default configuration runs the same analyzer chain at index and query time. This produces predictable behavior but may not produce the optimal match behavior for all query patterns. Specific patterns (autocomplete, synonym expansion, multi-language) benefit from asymmetric chains where index-time and query-time analysis differ deliberately. The discipline is knowing when symmetric is right and when asymmetric is warranted.
Symmetric analysis: the default and most common. Index and query use identical analyzer chains. Documents indexed with the chain produce tokens; queries processed by the same chain produce tokens that match. The configuration is the design intent of Lucene-based engines: the analyzer chain is associated with the field, and is applied automatically at both index time and query time. The team configures one chain per field; the engine handles applying it correctly at both times.
Asymmetric for autocomplete. Edge n-gram indexing with keyword query. At index time: tokenize and produce edge n-grams of the tokens ("nike" → [n, ni, nik, nike]). At query time: use keyword analyzer (no tokenization beyond lowercasing). The user typing "ni" produces query token [ni] which matches the indexed n-gram [ni] from "nike". The pattern supports prefix matching efficiently; without asymmetric chains, prefix matching requires expensive query-time substring operations.
Asymmetric for synonym expansion. Two options. Option 1: expand synonyms at index time (an indexed document containing "sneakers" gets indexed with tokens [sneakers, running, shoes, footwear]). Query time uses no synonym expansion (the variants are already in the index). Index size grows; query time is faster. Option 2: expand synonyms at query time (the indexed document has [sneakers]; query for "shoes" expands to [shoes, sneakers, footwear] which matches). Index stays lean; query time has more work. Production deployments choose based on whether index-side or query-side complexity is preferred.
Multi-field analysis from one source. The same source content indexed multiple times with different analyzers. Schema declares title with sub-fields title.exact and title.ngram; the indexer automatically produces three sets of tokens from each document's title. Queries can match against title (stemmed match), title.exact (exact phrase), or title.ngram (prefix). The pattern is the foundation of multi-mode matching; production schemas use it heavily for high-value text fields.
Language-aware index-time analysis. For multi-language corpora: detect language at index time per document; apply language-specific analyzer based on detection. A document in French gets the french_analyzer; a document in English gets english_analyzer. Tokens produced are language-appropriate. At query time, language detection on the query routes to the appropriate field. The pattern preserves per-language match quality but requires reliable language detection and per-language analyzer configuration.
Index-time validation. Test the analyzer chain against representative content before deploying schema changes. Use the engine's analyze API (Elasticsearch _analyze, Solr analysis tool) to see exactly what tokens a piece of content produces. Compare with expectations; investigate discrepancies. Production deployments typically maintain test suites of representative content with expected token outputs; schema changes that affect tokenization should pass these tests before deployment.
Operational implications. Index-time analyzer choices are persistent: documents are indexed once with the analyzer in effect at that time; subsequent analyzer changes require reindexing to take effect. The persistence means analyzer changes are not lightweight — they trigger reindexing operations (Section F) that have operational cost. The discipline is designing analyzer chains carefully upfront and treating changes as significant decisions.
All lexical search systems use index-time analyzer chains. Specific asymmetric patterns apply where they're justified by use case: edge n-grams for autocomplete; index-time synonym expansion for query-time performance; language-specific analyzers for multilingual content. The default symmetric pattern is correct for most cases; asymmetric patterns are deliberate optimizations.
Alternatives — keyword-only fields (no tokenization) for exact-match-only fields. Pure vector retrieval bypasses the analyzer chain entirely. The analyzer chain remains foundational for the lexical portion of any hybrid system.
- Apache Lucene analyzer documentation
- Elasticsearch / OpenSearch / Solr analyzer documentation
- Production methodology writings on multi-field schema design