Skip to main content

Full-text languages

This page lists the languages that a full-text index supports, how Foundation4 determines the language of fragments and queries, how text is reduced to searchable terms, the query syntax and the ranking. Integrators use this page to predict which fragments a full-text search on a full-text index matches. A BM25 index detects no language and splits text with the tokenizer of the index, as described in BM25 tokenizers. Search and retrieval explains full-text search, and Search requests describes the request.

Supported languages​

Full-text search uses the text search of PostgreSQL. A PostgreSQL text search configuration defines how words are reduced to terms. Most detected languages have a configuration of their own, and the others use the simple configuration, which converts words to lowercase without stemming:

LanguageConfiguration
Arabicarabic
Armenianarmenian
Basquebasque
Catalancatalan
Danishdanish
Dutchdutch
Englishenglish
Finnishfinnish
Frenchfrench
Germangerman
Greekgreek
Hindihindi
Hungarianhungarian
Indonesianindonesian
Irishirish
Italianitalian
Lithuanianlithuanian
Norwegian, Bokmål and Nynorsknorwegian
Portugueseportuguese
Romanianromanian
Russianrussian
Serbianserbian
Spanishspanish
Swedishswedish
Tamiltamil
Turkishturkish
Belarusian, Bulgarian, Czech, Marathi, Persian, Polish, Slovak, Ukrainian and Urdusimple
No language detectedsimple

Several of these configurations exist only in recent PostgreSQL releases. When the database has no configuration for a detected language, Foundation4 indexes and parses the text of that language with simple. The PostgreSQL images that Foundation4 provides run PostgreSQL 18, which has every configuration in the table.

Language detection​

Foundation4 detects languages with the Lingua library, restricted to the languages in the table: 36 languages, with Norwegian Bokmål and Norwegian Nynorsk counted separately. The language models are part of the Foundation4 binary, so detection needs no network access and no model files.

  • Fragments. The worker detects the language of each fragment when the worker processes a document on a pipeline with full-text search, and stores the configuration with the fragment's index entry. The fragments of one document can therefore receive different configurations.
  • Queries. The API server parses the query with the stored configuration of each fragment. The query words are therefore reduced to terms by the same rules as the words of each fragment, and the language that detection would assign to the query text alone does not affect matching.
  • Fragments indexed by an earlier release. These fragments have no stored configuration. For them, the API server detects the language of the whole query text and parses the query with that language's configuration. A query detected as a different language from the content is stemmed with a different configuration, and fewer of these fragments match. For example, the queries billing and Billing page match nothing in such an English fragment that contains "the billing page", while the billing page matches the fragment. A new version of the document, or a full-text index that is deleted and added again, stores the configuration of each fragment.
  • Shared stems. Languages whose stemmers reduce a word to the same stem match each other's fragments. The French factures and the Spanish facturas both reduce to factur, so either query matches French and Spanish fragments that contain the word.
  • No override. No request, pipeline setting or configuration key sets the language. The stored configuration of a fragment is visible in the database only: no API response returns the configuration.

Other languages​

Because detection chooses among the 36 detected languages only, text in another language is treated as follows:

  • Languages written in the script of a detected language. Detection assigns the closest detected language, such as Dutch for Afrikaans text. The text is then stemmed with the configuration of the assigned language.
  • Nepali. Detection has no Nepali model, so Nepali text is detected as Hindi or Marathi and receives the hindi or the simple configuration. The PostgreSQL configuration nepali is not used.
  • Other scripts and text without letters. Text in scripts that no detected language uses, such as Hebrew or Thai, and text that consists only of numbers, codes or punctuation, uses the simple configuration, which converts words to lowercase without stemming.
  • Chinese, Japanese and Korean. PostgreSQL text search does not divide these scripts into words, so a run of characters without spaces forms one term. Full-text search on a full-text index suits languages that separate words with spaces. A BM25 index whose tokenizer splits these scripts, such as the multilingual llmlingua2 model or the Jieba pre-tokenizer for Chinese, matches words inside a run of characters, as described in BM25 tokenizers. Similarity search with a multilingual embedding model also serves content in these languages.

Terms​

PostgreSQL reduces the text of each fragment and each query to terms with the selected configuration:

  • Case. Text is converted to lowercase, so matching is case-insensitive.
  • Stems. Each word is reduced to a stem, so a query for connections also matches connected and connecting in English text. The simple configuration does not stem.
  • Stop words. Common words, such as the and and in English, are removed from the text and from the query in the configurations that define a stop word list.
  • Accents. Accents are kept, so café and cafe are different terms unless the stemmer of the language treats them as one.
  • Tokens. Words, numbers, hyphenated words, email addresses, URLs, host names and file paths each form terms. Other punctuation separates words.

PostgreSQL limits the size of a term and of the terms of one fragment. A word longer than 2,047 bytes is skipped, and a fragment whose terms exceed 1 megabyte cannot be indexed, which affects only very large fragments.

Query syntax​

The query of a full-text search is plain text:

  • All terms required. A fragment matches when the fragment contains every term of the query, in any order and at any distance.
  • No operators. Quotation marks, OR, -, NOT, * and other operators have no special meaning, and the characters separate words.
  • Empty queries. A query that consists only of stop words or punctuation produces no terms and matches no fragment.

A query of several specific words therefore narrows the results, and a query that contains a word absent from the relevant fragments returns nothing. A client application that needs looser matching runs a similarity search as well, as described in Search and retrieval.

Ranking​

The score of a full-text result is the PostgreSQL ts_rank value of the fragment for the query, divided by 1 plus the logarithm of the fragment length:

  • Frequency. A fragment that contains the query terms more often ranks higher.
  • Length. A longer fragment ranks lower for the same matches.
  • No corpus statistics. The rank does not consider how common a term is across the pipeline, so a rare term and a common term count the same.
  • Direction. A higher score is a better match. The threshold parameter sets a minimum score.
  • Scale. Scores are small numbers without a fixed upper limit. A threshold is chosen from the scores of sample searches on the pipeline's content.