Skip to main content

Search and retrieval

Foundation4 retrieves fragments from a pipeline with five search types: similarity search, maximal marginal relevance (MMR) search, full-text search, hybrid search and history search. This page describes the search request, how each search type selects and ranks fragments, what the score of a result means for each type, and the scope that applies to every search. Integrators need this page to choose a search type and to interpret search results.

Search request​

All five search types use one endpoint, POST /pipelines/{id}/search. The API key requires read and execute permission on the pipeline. The request body contains the following fields.

FieldRequiredDescription
queryYesThe search text, such as the end user's question or keywords. History search ignores the text, but the field is still required.
typeYesThe search type: similarity, mmr, search (full-text search), hybrid or history
classificationYesThe classifications that the reader holds. The Classifications page describes the accepted forms.
paramsNoThe parameters of the search type, described in each type's section, including the search index to read
filtersNoA metadata filter. The Metadata, filters and taxonomies page describes the filter language.
pathNoA path pattern, such as kb.billing.>, that limits the search to the documents under a path. Without the field, the search covers only the documents without a path. Document paths describes the patterns.
distance_strategyNoThe vector distance function for similarity, MMR and hybrid search, described in Distance strategies

The following request runs a similarity search for the five closest fragments:

{
"query": "How do I rotate the database credentials?",
"type": "similarity",
"classification": ["internal"],
"params": {"k": 5}
}

The response is an array of fragments, ordered from the best match to the weakest match. Each fragment carries the decrypted text in page_content and the match score in score. The Documents, versions and fragments page describes the other fragment fields.

A parameter that the request omits takes the default of the search type, whether the request omits the whole params object or one field inside the object. A field outside the table returns HTTP 400 with the message Unknown field `<name>` and the accepted fields in details.expected. A parameter that the search type does not define, such as a misspelled parameter name, returns HTTP 422. Search requests lists every field, parameter and error of the search request.

Search indexes​

Each search type except history search reads one or more search indexes of the pipeline. Similarity and MMR search read a vector index, full-text search reads a text index, a full-text index or a BM25 index, and hybrid search reads several indexes of either kind. The request names an index in params.index, or the indexes of a hybrid search in params.indexes. A request that names no index uses the pipeline's default index of the kind that the search type reads.

A search that names an index of another kind, or of another pipeline, returns HTTP 400. A search on an index that is still being built, or whose build failed, returns HTTP 409. Search indexes describes the index kinds, the defaults and the life cycle of an index.

Search path​

Rectangles are processing steps in the API server. Cylinders are search indexes in PostgreSQL.

A search that reads a vector index first converts the query into a vector with the embedding model of the index, so the latency of a vector search includes one embedding call. A hybrid search computes one vector for each embedding model that its vector indexes use. The API server then resolves the classifications named in the request, including inherited classifications when matching is hierarchical. PostgreSQL selects the matching fragments from each index, restricted to the resolved classifications, the metadata filter and current versions. A hybrid search fuses the lists of its indexes into one list, and other search types pass their single list on unchanged. The API server decrypts the text of the selected fragments and returns the fragments with their scores. History search reads no index and returns the most recent fragments.

Similarity search returns the k fragments whose vectors are closest to the query vector. Similarity search suits questions phrased in natural language, because the embedding model places texts with similar meaning close together even when the texts share no words.

ParameterDefaultDescription
indexThe default vector indexThe identifier of the vector index to search
k5The number of fragments to return, from 1 to 65535. A value of 0 returns HTTP 400.
thresholdNoneThe maximum distance. Fragments farther from the query than the threshold are excluded.

Each vector index finds the closest vectors through a distance index, an approximate nearest-neighbor index that trades a small loss of recall for query times that remain short as the pipeline grows.

MMR search returns k fragments that are relevant to the query and different from one another. MMR search suits content with many near-duplicate fragments, such as chat history or versioned documents, where the closest fragments often repeat the same information.

MMR search runs in two steps. The API server retrieves the fetch_k closest fragments, as in similarity search, and then selects k of those candidates one at a time. Each selection balances the candidate's similarity to the query against the candidate's similarity to the fragments already selected.

ParameterDefaultDescription
indexThe default vector indexThe identifier of the vector index to search
k5The number of fragments to return, from 1 to 65535. A value of 0 returns HTTP 400.
fetch_k10The number of candidates, at least k. A value smaller than k returns HTTP 400, so a request with a k above 10 also sets fetch_k.
lambda_mult0.5The balance between relevance and diversity, from 0 to 1. A value of 1 selects by relevance alone. A value of 0 maximizes diversity. A value outside 0 to 1 returns HTTP 400.
thresholdNoneThe maximum distance of a candidate from the query

MMR selection measures relevance and diversity with the distance strategy of the search, which can be any of the three strategies that a vector index serves. The selected fragments are returned in order of distance from the query.

Distance strategies​

The distance strategy determines how a vector search measures the distance between the query vector and a fragment vector. The request sets the strategy in distance_strategy, and the strategy applies to every vector index that the search reads. A strategy can be used only on a vector index that has a distance index for the strategy.

ValueDistanceScore range
cosineCosine distance: 1 minus the cosine similarity0 (identical direction) to 2
euclidean-distanceEuclidean (L2) distance0 (identical vectors) upward
dot-productNegative inner productNegative values for similar vectors. Lower is closer.

A request without distance_strategy uses the first strategy that the index has a distance index for, in the order cosine, euclidean-distance, dot-product. A new vector index has a distance index for cosine only, so cosine is the default in practice. A request that names a strategy without a distance index returns HTTP 400 rather than comparing the query with every vector of the pipeline. The value max-inner-product is not supported and returns HTTP 400.

The distance strategy should match the strategy for which the embedding model was trained. Most sentence embedding models, including the default FastEmbed model, are trained for cosine similarity. Search indexes describes how to add a distance index.

Full-text search returns up to k fragments that match the words of the query, ranked by relevance. Full-text search suits exact terms that an embedding model represents poorly: product codes, error messages, names and acronyms. The search reads a text index, and the kind of the index determines how fragments match and rank.

ParameterDefaultDescription
indexThe default text indexThe identifier of the full-text index or BM25 index to search
k5The number of fragments to return, from 1 to 65535. A value of 0 returns HTTP 400.
thresholdNoneThe minimum score. Fragments that score below the threshold are excluded.

A full-text index uses PostgreSQL text search. Language handling determines which fragments match:

  • Language detection at ingestion. The worker detects the language of each fragment, builds the fragment's index entry with the PostgreSQL text search configuration for that language and stores the configuration with the entry. The configuration reduces each word to a stem and removes common words, so the query approve refunds matches a fragment that contains "approving a refund".
  • Query parsing. The API server parses the query with the stored configuration of each fragment, so the query words are reduced to stems by the same rules as the words of the fragment. The language that detection would assign to the query text alone does not affect matching.
  • Supported languages. Arabic, Armenian, Basque, Catalan, Danish, Dutch, English, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Lithuanian, Norwegian, Portuguese, Romanian, Russian, Serbian, Spanish, Swedish, Tamil and Turkish have a configuration of their own. Belarusian, Bulgarian, Czech, Marathi, Persian, Polish, Slovak, Ukrainian and Urdu text, and text in scripts that no detected language uses, receives the simple configuration: words are lowercased without stemming. Text in another language receives the configuration of the closest detected language. Full-text languages describes detection in detail.
  • Word matching. Every remaining word of the query must appear in the fragment. The query syntax supports no phrase, prefix or Boolean operators.

Fragments indexed by an earlier release have no stored configuration. For these fragments, the API server parses the query with the language detected for the query text alone. A short query can then be detected as a different language from the content, in which case the query words are stemmed differently from the indexed words and fewer fragments match. A new version of the document stores the configuration of each fragment.

Full-text search on a BM25 index​

A BM25 index splits the text of each fragment and of each query into tokens with the index's tokenizer, and ranks fragments with the BM25 formula. BM25 tokenizers lists the tokenizer options. BM25 weighs each query token by the number of times the token occurs in the fragment and by the rarity of the token across the pipeline, and favors shorter fragments for the same matches.

A BM25 index ranks every fragment that shares at least one token with the query, so a query in natural language still returns fragments that match some of its words, ranked by how well each fragment matches. The tokenizer, not language detection, determines how words are split and normalized. The default tokenizer, bert_base_uncased, lowercases the text and splits words into the word pieces of an English vocabulary, and other tokenizers can add stemming, stop words or segmentation for other languages.

The statistics of a BM25 index, such as the number of fragments, their average length and the number of fragments that contain each token, keep counting the fragments of deleted documents. After deletions, the BM25 scores of the remaining fragments therefore differ from the scores that a newly built index gives, and a fixed threshold can keep a different set of fragments. A rebuild of the BM25 index recomputes the statistics, as described in Add, rebuild or switch a search index.

Hybrid search runs several searches for the same query and fuses the result lists into one list by rank position. Hybrid search suits a search box that serves both questions and exact terms: a fragment that is close in meaning and also contains the exact terms leads the fused list.

ParameterDefaultDescription
indexesThe default vector index and the default text indexThe identifiers of the search indexes to combine, in any number and mix of vector and text indexes. Order and repeated identifiers have no effect.
k5The number of fragments that each index returns, and the length of the fused list

Foundation4 searches each index for the k best fragments, without a threshold, and fuses the lists with reciprocal rank fusion (RRF). A fragment at rank r in a list, counting from 1, receives 1 / (60 + r) from that list, and the fused score of a fragment is the sum over all lists. A fragment that is missing from a list receives nothing from that list. The fused list is sorted by descending fused score and cut to k fragments. Hybrid search gives every list the same weight and takes no threshold: a hybrid search with threshold in params returns HTTP 422.

A hybrid search that names no index requires both a default vector index and a default text index, and returns HTTP 400 when the pipeline lacks either default. Set up hybrid search describes hybrid search with a worked example.

History search returns the most recent fragments of the pipeline, without ranking by the query. The search orders fragments by document version, newest first, and the fragments of one document version by position. The parameter k, default 5, sets the number of fragments. query is required but ignored, and the results carry no score field.

History search reads no search index, so the search works on every pipeline, including a pipeline without search indexes. The classifications, the metadata filter and current versions restrict history search in the same way as the other search types. A history search with a metadata filter therefore returns, for example, the latest fragments of the documents about one product.

Score semantics​

The meaning of score differs by search type, and scores from different types cannot be compared.

Search typeScoreBetter match
SimilarityDistance from the query, under the request's distance strategyLower
MMRDistance from the query, under the request's distance strategyLower
Full-text, full-text indexPostgreSQL ts_rank, divided by 1 plus the logarithm of the fragment lengthHigher
Full-text, BM25 indexBM25 score, greater than 0Higher
HybridFused RRF score: the sum of 1 / (60 + rank) over the lists, at most the number of lists divided by 61Higher
HistoryNo score fieldNot applicable

Result scope​

The following restrictions apply to every search type:

  • Classifications. Search returns only fragments that carry a classification resolved from the request.
  • Allow-list. The classifications that a request names must be within the classification allow-list of the API key, as described in Access control. A request that names a classification outside the allow-list returns HTTP 403.
  • Metadata filter. Search returns only fragments whose metadata satisfies the filter.
  • Path. Search returns only fragments of documents whose path matches the request's path pattern, or, without path, of documents without a path.
  • Current content. Search excludes expired versions. When a new version of a document is submitted, the previous version leaves search results at once, and the new version becomes searchable when processing of that version completes.

Search always operates on current content. Point-in-time reads are available on the document and fragment endpoints, as described in Documents, versions and fragments.