BM25 tokenizers

A BM25 index splits the text of each fragment and of each query into tokens with the index's tokenizer, and ranks fragments by the BM25 formula over those tokens. This page lists the options of a tokenizer, the built-in models, the validation rules and the conditions for a tokenizer from Hugging Face. Integrators use this page to choose the tokenizer of a BM25 index, and operators use this page to plan BM25 indexes in an air-gapped deployment. Search and retrieval explains BM25 ranking.

Tokenizer object

A tokenizer is a JSON object with four optional fields. The tokenizer applies the fields in the following order: the character filters change the text, the pre-tokenizer splits the text into tokens, the token filters change or drop tokens, and the model maps each token to the token identifiers that the index stores.

Field	Type	Default	Description
model	String or object	bert_base_uncased	The model that maps tokens to token identifiers, described in Models
character_filters	Array	None	Filters applied to the text, in order, before the text is split
pre_tokenizer	String or object	None	Splits the text into tokens before the model. Without a pre-tokenizer, the model receives the whole text and splits the text itself.
token_filters	Array	None	Filters applied to the tokens, in order, before the model

A tokenizer is set in two places:

• Pipeline creation. bm25_tokenizer in POST /pipelines, with full_text_search set to bm25, as described in Pipelines.
• Added index. tokenizer in POST /pipelines/{id}/search-indexers, with type set to bm25, as described in Add, rebuild or switch a search index.

An omitted or null tokenizer, and the empty object {}, select the default tokenizer, {"model": "bert_base_uncased"}. The index response returns the tokenizer in tokenizer, and the pipeline response returns the tokenizer of each BM25 index in indexers, as {"Bm25": <tokenizer>}. The response omits empty arrays and an absent pre-tokenizer.

Models

model	Vocabulary	Languages
bert_base_uncased	Word pieces of the BERT base uncased model. The model lowercases the text.	English
wiki_tocken	Whole words, from a vocabulary trained on the WikiText-103 corpus	English
gemma2b	Pieces of the Gemma 2B model	Multilingual
llmlingua2	Pieces of the XLM-RoBERTa model of LLMLingua-2	Multilingual
{"hugging_face": {"repo": "<owner>/<name>", "revision": "<revision>"}}	The tokenizer.json file of a Hugging Face model repository. revision is optional and names a branch, a tag or a commit.	As the model

The four built-in models are part of the PostgreSQL tokenizer extension and need no download. A Hugging Face model is downloaded when the index is created, as described in Hugging Face tokenizers.

Character filters

Filter	JSON	Effect
Lowercase	"to_lowercase"	Lowercases the text
Unicode normalization	{"unicode_normalization": "nfc"}, or nfd, nfkc or nfkd	Converts the text to the named Unicode normalization form

Pre-tokenizers

Pre-tokenizer	JSON	Tokens
Regular expression	{"regex": "<pattern>"}	Every match of the pattern
Unicode segmentation	"unicode_segmentation"	The words of the text, by the word boundary rules of Unicode Standard Annex #29
Jieba	{"jieba": {"mode": "search", "enable_hmm": true}}	The words of Chinese text, by the Jieba segmenter. mode is full, precise or search, with search as the default, and enable_hmm defaults to true.

Token filters

Filter	JSON	Effect
Non-alphanumeric tokens	"skip_non_alphanumeric"	Drops tokens without a letter or a digit
Stemmer	{"stemmer": "<language>"}	Reduces each token to its stem with a Snowball stemmer
Stop words	{"stopwords": "lucene_english"}, or nltk_english or iso_english	Drops the words of a built-in English stop word list
PostgreSQL dictionary	{"pg_dict": "<name>"}	Passes each token through a PostgreSQL text search dictionary of the pg_catalog schema, such as english_stem

The stemmer takes one of the following languages: arabic, armenian, basque, catalan, danish, dutch, english_porter, english_porter2, estonian, finnish, french, german, greek, hindi, hungarian, indonesian, irish, italian, lithuanian, nepali, norwegian, portuguese, romanian, russian, serbian, spanish, swedish, tamil, turkish and yiddish.

Examples

The following tokenizers show the shape of common configurations. The same tokenizer processes fragments and queries, so a configuration that suits the content also suits the queries. A tokenizer is compared on a sample of the pipeline's content and queries before the index becomes the default, as described in Changing a tokenizer.

English text with the default model:

{"model": "bert_base_uncased"}

English text split into words, lowercased, without English stop words and reduced to stems:

{
  "model": "bert_base_uncased",
  "character_filters": ["to_lowercase"],
  "pre_tokenizer": "unicode_segmentation",
  "token_filters": [{"stopwords": "lucene_english"}, {"stemmer": "english_porter2"}]
}

Text in several languages:

{"model": "llmlingua2"}

Chinese text split into words by Jieba before a multilingual model:

{"model": "llmlingua2", "pre_tokenizer": {"jieba": {}}}

The default tokenizer keeps common words, such as the and and. BM25 gives a token that occurs in most fragments a low weight, but a query that consists only of common words still matches most fragments. A stop word filter drops such words from fragments and queries.

Validation

The API server checks a tokenizer before the index is created. An object with an unknown field or an unknown option value, such as a misspelled model or a stemmer for an unsupported language, returns HTTP 422. The following checks return HTTP 400 with the message Invalid parameter and the reason in details.error:

details.error	Cause
Invalid Hugging Face repository '<repo>': expected 'owner/name'	repo is not of the form owner/name
Invalid Hugging Face revision '<revision>'	revision holds a character that a branch, tag or commit cannot hold
Invalid pre-tokenizer regular expression: <reason>	The regex pattern is not a valid regular expression
Invalid text search dictionary name '<name>'	The pg_dict name holds a character other than lowercase letters, digits and underscores, or starts with a digit or an underscore
Unknown text search dictionary '<name>'	The pg_catalog schema has no text search dictionary with that name
Error downloading tokenizer.json of <repo> from Hugging Face: <reason>	The API server could not download the tokenizer, as described in Hugging Face tokenizers
bm25_tokenizer requires full_text_search to be bm25	A pipeline create request sets bm25_tokenizer without full_text_search set to bm25

A pipeline holds one BM25 index for each distinct tokenizer. A request that adds a BM25 index with a tokenizer identical to the tokenizer of an existing BM25 index returns HTTP 409 with details.error set to The pipeline already has a BM25 indexer with this tokenizer. Tokenizers that differ in any option, such as one token filter, are distinct.

Hugging Face tokenizers

A tokenizer with a hugging_face model uses the tokenizer.json file of a model repository on Hugging Face. The API server downloads the file from huggingface.co when the pipeline or the index is created, keeps a copy in the Hugging Face cache directory (paths.huggingface), and stores the tokenizer in PostgreSQL. Workers and searches use the stored tokenizer and download nothing.

The following conditions apply:

• Network access. The API server pod reaches huggingface.co over HTTPS when the index is created. No setting points the download at another address, so an air-gapped deployment uses the built-in models, as listed in Models, packages and air-gapped installs.
• Revision. Without revision, the API server downloads the file from the main branch at the time of creation. An index created later from the same repository can therefore receive a different tokenizer. A commit identifier in revision fixes the file.

Changing a tokenizer

The tokenizer of a BM25 index is fixed for the life of the index. A different tokenizer requires a new BM25 index:

1. Add a BM25 index with the new tokenizer. The workers fill the index from the stored fragment text, and no document is submitted again.
2. Wait until the new index is ready, and compare searches that name each index in params.index.
3. Make the new index the default text index, and delete the previous index.

Add, rebuild or switch a search index describes each request.