Full-text languages
This page lists the languages that a full-text index supports, how Foundation4 determines the language of fragments and queries, how text is reduced to searchable terms, the query syntax and the ranking. Integrators use this page to predict which fragments a full-text search on a full-text index matches. A BM25 index detects no language and splits text with the tokenizer of the index, as described in BM25 tokenizers. Search and retrieval explains full-text search, and Search requests describes the request.
Supported languages
Full-text search uses the text search of PostgreSQL. A PostgreSQL text search configuration defines how words are reduced to terms. Most detected languages have a configuration of their own, and the others use the simple configuration, which converts words to lowercase without stemming:
| Language | Configuration |
|---|---|
| Arabic | arabic |
| Armenian | armenian |
| Basque | basque |
| Catalan | catalan |
| Danish | danish |
| Dutch | dutch |
| English | english |
| Finnish | finnish |
| French | french |
| German | german |
| Greek | greek |
| Hindi | hindi |
| Hungarian | hungarian |
| Indonesian | indonesian |
| Irish | irish |
| Italian | italian |
| Lithuanian | lithuanian |
| Norwegian, Bokmål and Nynorsk | norwegian |
| Portuguese | portuguese |
| Romanian | romanian |
| Russian | russian |
| Serbian | serbian |
| Spanish | spanish |
| Swedish | swedish |
| Tamil | tamil |
| Turkish | turkish |
| Belarusian, Bulgarian, Czech, Marathi, Persian, Polish, Slovak, Ukrainian and Urdu | simple |
| No language detected | simple |
Several of these configurations exist only in recent PostgreSQL releases. When the database has no configuration for a detected language, Foundation4 indexes and parses the text of that language with simple. The PostgreSQL images that Foundation4 provides run PostgreSQL 18, which has every configuration in the table.
Language detection
Foundation4 detects languages with the Lingua library, restricted to the languages in the table: 36 languages, with Norwegian Bokmål and Norwegian Nynorsk counted separately. The language models are part of the Foundation4 binary, so detection needs no network access and no model files.
- Fragments. The worker detects the language of each fragment when the worker processes a document on a pipeline with full-text search, and stores the configuration with the fragment's index entry. The fragments of one document can therefore receive different configurations.
- Queries. The API server parses the query with the stored configuration of each fragment. The query words are therefore reduced to terms by the same rules as the words of each fragment, and the language that detection would assign to the query text alone does not affect matching.
- Fragments indexed by an earlier release. These fragments have no stored configuration. For them, the API server detects the language of the whole query text and parses the query with that language's configuration. A query detected as a different language from the content is stemmed with a different configuration, and fewer of these fragments match. For example, the queries
billingandBilling pagematch nothing in such an English fragment that contains "the billing page", whilethe billing pagematches the fragment. A new version of the document, or a full-text index that is deleted and added again, stores the configuration of each fragment. - Shared stems. Languages whose stemmers reduce a word to the same stem match each other's fragments. The French
facturesand the Spanishfacturasboth reduce tofactur, so either query matches French and Spanish fragments that contain the word. - No override. No request, pipeline setting or configuration key sets the language. The stored configuration of a fragment is visible in the database only: no API response returns the configuration.
Other languages
Because detection chooses among the 36 detected languages only, text in another language is treated as follows:
- Languages written in the script of a detected language. Detection assigns the closest detected language, such as Dutch for Afrikaans text. The text is then stemmed with the configuration of the assigned language.
- Nepali. Detection has no Nepali model, so Nepali text is detected as Hindi or Marathi and receives the
hindior thesimpleconfiguration. The PostgreSQL configurationnepaliis not used. - Other scripts and text without letters. Text in scripts that no detected language uses, such as Hebrew or Thai, and text that consists only of numbers, codes or punctuation, uses the
simpleconfiguration, which converts words to lowercase without stemming. - Chinese, Japanese and Korean. PostgreSQL text search does not divide these scripts into words, so a run of characters without spaces forms one term. Full-text search on a full-text index suits languages that separate words with spaces. A BM25 index whose tokenizer splits these scripts, such as the multilingual
llmlingua2model or the Jieba pre-tokenizer for Chinese, matches words inside a run of characters, as described in BM25 tokenizers. Similarity search with a multilingual embedding model also serves content in these languages.
Terms
PostgreSQL reduces the text of each fragment and each query to terms with the selected configuration:
- Case. Text is converted to lowercase, so matching is case-insensitive.
- Stems. Each word is reduced to a stem, so a query for
connectionsalso matchesconnectedandconnectingin English text. Thesimpleconfiguration does not stem. - Stop words. Common words, such as
theandandin English, are removed from the text and from the query in the configurations that define a stop word list. - Accents. Accents are kept, so
caféandcafeare different terms unless the stemmer of the language treats them as one. - Tokens. Words, numbers, hyphenated words, email addresses, URLs, host names and file paths each form terms. Other punctuation separates words.
PostgreSQL limits the size of a term and of the terms of one fragment. A word longer than 2,047 bytes is skipped, and a fragment whose terms exceed 1 megabyte cannot be indexed, which affects only very large fragments.
Query syntax
The query of a full-text search is plain text:
- All terms required. A fragment matches when the fragment contains every term of the query, in any order and at any distance.
- No operators. Quotation marks,
OR,-,NOT,*and other operators have no special meaning, and the characters separate words. - Empty queries. A query that consists only of stop words or punctuation produces no terms and matches no fragment.
A query of several specific words therefore narrows the results, and a query that contains a word absent from the relevant fragments returns nothing. A client application that needs looser matching runs a similarity search as well, as described in Search and retrieval.
Ranking
The score of a full-text result is the PostgreSQL ts_rank value of the fragment for the query, divided by 1 plus the logarithm of the fragment length:
- Frequency. A fragment that contains the query terms more often ranks higher.
- Length. A longer fragment ranks lower for the same matches.
- No corpus statistics. The rank does not consider how common a term is across the pipeline, so a rare term and a common term count the same.
- Direction. A higher score is a better match. The
thresholdparameter sets a minimum score. - Scale. Scores are small numbers without a fixed upper limit. A threshold is chosen from the scores of sample searches on the pipeline's content.