Skip to main content

Choose and tune a search mode

This guide selects a search type for a retrieval task and tunes the parameters of the type: the number of results, the maximal marginal relevance (MMR) parameters, score thresholds and the distance strategy. A worked sequence of searches on one pipeline compares similarity search, MMR search and full-text search. Integrators who design the retrieval of a client application need this guide. Search and retrieval explains how each search type ranks fragments, and Search requests lists every field of the request.

Search modes​

Each search type suits a different kind of query:

Retrieval taskSearch typeReason
Questions in natural language, paraphrases and synonymsSimilarity searchThe embedding model places texts with similar meaning close together, even when the texts share no words
Content with many near-duplicate fragments, such as chat history or successive versions of a policyMMR searchMMR search selects fragments that are relevant to the query and different from one another, so the results do not repeat one piece of information
Exact terms: product codes, error messages, names and acronymsFull-text search on a full-text indexThe fragment must contain every significant word of the query after stemming, independently of how the embedding model represents rare terms
Keywords where a partial match is still usefulFull-text search on a BM25 indexA BM25 index ranks every fragment that shares a token with the query, with the best matches first
Questions and exact terms in the same search boxHybrid searchHybrid search fuses the results of a vector index and a text index by rank, as described in Set up hybrid search
The most recent content, such as the latest messagesHistory searchHistory search returns the newest fragments without ranking by the query

The search indexes of the pipeline determine which types are available:

  • Vector index. Similarity and MMR search read a vector index. Each vector search converts the query into a vector, so the latency of a vector search includes one embedding call.
  • Text index. Full-text search reads a full-text index or a BM25 index. A pipeline without a text index returns HTTP 400 to a full-text search.
  • Both kinds. Hybrid search without named indexes reads the default vector index and the default text index.
  • No index. History search reads no index and works on every pipeline.

A missing index can be added to a pipeline that already holds documents, as described in Add, rebuild or switch a search index.

Example pipeline​

The examples use the support-kb pipeline and the three documents from First search, with the shell variables from that tutorial. The pipeline has a vector index for the all-MiniLM-L6-v2 embedding model and a full-text index, which are the pipeline's default indexes. Each document forms a single fragment. Every search names the classification internal, so the fragments of all three documents are candidates.

Similarity search baseline​

Run a similarity search for a question in natural language. The request omits params, so the default parameters apply and k is 5:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "How do customers get their money back?",
"type": "similarity",
"classification": "internal"
}'

Expected result: status 200 and the three fragments of the pipeline, ordered by ascending score:

[
{
"id": "01a0ed0d-e007-7c3d-8e4f-6a7b8c9d0e1f",
"classification": "internal",
"metadata": {"product": "billing"},
"path": null,
"document_id": "01a0ed0d-df3f-7f80-9a1b-2c3d4e5f6a7b",
"page_content": "Refunds for annual plans are prorated. Support agents must record the refund reason in the ticket before approving a refund.",
"version": 1790683504447128,
"expired_at": null,
"score": 0.6445,
"order_id": 0
},
{
"id": "01a0ed0d-d814-7d6c-8b5a-4c3d2e1f0a9b",
"classification": "internal",
"metadata": {"product": "billing"},
"path": null,
"document_id": "01a0ed0d-d75f-7d5e-8f60-718293a4b5c6",
"page_content": "Invoices are generated on the first day of each month. Finance staff can export invoices as PDF or CSV files from the Billing page.",
"version": 1790683502431906,
"expired_at": null,
"score": 0.8043,
"order_id": 0
},
{
"id": "01a0ed0d-d291-7c5d-8e4f-3a2b1c0d9e8f",
"classification": "public",
"metadata": {"product": "accounts"},
"path": null,
"document_id": "01a0ed0d-cf82-7c3d-8e4f-5a6b7c8d9e0f",
"page_content": "To reset a password, open Settings, choose Security and select Reset password. A reset link is sent to the registered email address and expires after 30 minutes.",
"version": 1790683500418273,
"expired_at": null,
"score": 0.8393,
"order_id": 0
}
]

The refund fragment is the closest match, although the question shares no significant word with the fragment. The invoice and password fragments are returned as well, because similarity search returns the k closest fragments whether or not a fragment answers the question. In this mode, score is the cosine distance from the query, so a lower score is a closer match.

Full-text search with a question​

Send the same question as a full-text search:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "How do customers get their money back?",
"type": "search",
"classification": "internal"
}'

Expected result: status 200 and an empty array:

[]

On a full-text index, a fragment matches only when the fragment contains every significant word of the query after stemming. Common words such as how and their are removed from the query, but the remaining words, such as customers and money, appear in no fragment. Questions in natural language therefore suit similarity search. A BM25 index, by contrast, would rank the fragments that share some words of the question.

Full-text search with exact terms​

Send a full-text search with terms that the content contains verbatim:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "refund reason ticket",
"type": "search",
"classification": "internal",
"params": {"k": 5}
}'

Expected result: status 200 and one fragment, from the kb-refund-policy document:

[
{
"id": "01a0ed0d-e007-7c3d-8e4f-6a7b8c9d0e1f",
"classification": "internal",
"metadata": {"product": "billing"},
"path": null,
"document_id": "01a0ed0d-df3f-7f80-9a1b-2c3d4e5f6a7b",
"page_content": "Refunds for annual plans are prorated. Support agents must record the refund reason in the ticket before approving a refund.",
"version": 1790683504447128,
"expired_at": null,
"score": 0.1059,
"order_id": 0
}
]

The fragment contains all three words, so the fragment matches. The other fragments lack at least one word and are not returned, however many results k allows. In a full-text search, score is a text search rank: a higher score is a better match, and the values are not comparable with the distances of a similarity search.

Number of results​

k sets the maximum number of fragments that a search returns, from 1 to 65535. A larger value returns HTTP 422, and a value of 0 returns HTTP 400 with details.error set to `k` must be greater than 0. The value depends on the use of the results:

  • Retrieval for an LLM. The number of fragments that the prompt can hold. The fragment size of the text splitter multiplied by k stays within the part of the LLM context that the prompt reserves for retrieved text.
  • Result lists. The number of results that the client application displays. The search request has no offset, so a request for more results repeats the search with a larger k.
  • Hybrid search. Each index of a hybrid search contributes up to k fragments, and the fused list is cut to k, so k sets both the depth of each list and the length of the result, as described in Set up hybrid search.

A parameter that the request omits inside params takes the default of the search type, so {"threshold": 0.7} searches with a k of 5. A search returns fewer than k fragments when fewer fragments satisfy the classifications, the filter and the threshold. A vector search with a selective filter can also return fewer than k fragments, because the vector index is approximate.

Similarity threshold​

A threshold excludes fragments that are too far from the query. Repeat the baseline search with a threshold of 0.7:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "How do customers get their money back?",
"type": "similarity",
"classification": "internal",
"params": {"k": 5, "threshold": 0.7}
}'

Expected result: status 200 and only the fragments whose score is 0.7 or lower. With the scores of the baseline, the response contains the refund fragment alone. The following response omits the fields other than document_id and score:

[
{"document_id": "01a0ed0d-df3f-7f80-9a1b-2c3d4e5f6a7b", "score": 0.6445}
]

For similarity and MMR search, threshold is the maximum distance from the query, and PostgreSQL applies the threshold before k. A threshold is chosen from the pipeline's own content:

  1. Run the search for a set of representative queries without a threshold and with a large k, such as 20.
  2. Record the scores of the fragments that answer each query and of the fragments that do not.
  3. Set the threshold between the two groups of scores, closer to the answering fragments when the client application must avoid irrelevant results.

The scores depend on the embedding model and on the distance strategy, so the threshold is chosen again after either changes. Under the cosine strategy, a minimum cosine similarity of s corresponds to a threshold of 1 minus s: a minimum similarity of 0.6 is a threshold of 0.4.

For full-text search, threshold is a minimum score instead: the rank on a full-text index, or the BM25 score on a BM25 index. Both scores have no fixed maximum, and a longer fragment scores lower for the same matches, so a full-text threshold is also chosen from sample searches and reviewed when the text splitter or the index changes. Without a threshold, a full-text search on a full-text index returns up to k fragments that contain every significant word of the query. Hybrid search takes no threshold. Full-text languages describes the rank.

MMR parameters​

Run an MMR search that selects 2 fragments from up to 10 candidates:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "How do customers get their money back?",
"type": "mmr",
"classification": "internal",
"params": {"k": 2, "fetch_k": 10, "lambda_mult": 0.5}
}'

Expected result: status 200 and 2 fragments, ordered by ascending score. The following response omits the fields other than document_id and score:

[
{"document_id": "01a0ed0d-df3f-7f80-9a1b-2c3d4e5f6a7b", "score": 0.6445},
{"document_id": "01a0ed0d-cf82-7c3d-8e4f-5a6b7c8d9e0f", "score": 0.8393}
]

MMR search first selects the refund fragment, the candidate closest to the query. For the second selection, MMR search weighs each remaining candidate's similarity to the query against the candidate's similarity to the refund fragment. In this response, the invoice fragment is closer to the query, but the invoice fragment and the refund fragment both describe billing, so MMR search selects the password fragment. The response lists the selected fragments in order of distance, and score is the distance from the query, not the MMR score.

The parameters are tuned as follows:

  • k. The number of fragments to return, chosen as for similarity search.
  • fetch_k. The number of candidates, at least k. A fetch_k smaller than k returns HTTP 400 with details.error set to `fetch_k` cannot be smaller than `k`, so a request with a k above 10 also sets fetch_k. A larger pool gives MMR search more fragments to choose from. The API server receives the vector of every candidate and compares the candidates in memory, so the cost of the selection grows with fetch_k and k. A request without params uses a k of 5 and a fetch_k of 10.
  • lambda_mult. The balance between relevance and diversity, from 0 to 1. A value of 1 selects by relevance alone and returns the same fragments as a similarity search over the candidates. A value of 0 maximizes diversity. The default is 0.5, and lower values suit content that repeats itself more. A value outside 0 to 1 returns HTTP 400.
  • threshold. The maximum distance of a candidate. The threshold applies to the candidates before the selection.
  • Distance strategy. The strategy of the search, chosen as for similarity search. MMR selection measures relevance and diversity with that strategy, under cosine, dot-product and euclidean-distance alike.

Distance strategy​

The distance strategy matches the similarity function for which the embedding model was trained, as stated in the documentation of the model. Most sentence embedding models, including all-MiniLM-L6-v2, are trained for cosine similarity, and cosine is the default.

distance_strategyUseScore at a cosine similarity of 0.5, for vectors of unit length
cosineModels trained for cosine similarity0.5
dot-productModels trained for inner product similarity-0.5
euclidean-distanceModels trained for Euclidean distance1.0

A vector index serves a strategy only through a distance index for that strategy, and a new vector index has a distance index for cosine only. A search with another strategy returns HTTP 400 until a distance index for the strategy is added, as described in Add, rebuild or switch a search index. A search without distance_strategy uses the first indexed strategy in the order cosine, euclidean-distance, dot-product, so a client application that uses another strategy sends the same distance_strategy value in every vector search while the cosine index exists.

For an embedding model that produces vectors of unit length, such as all-MiniLM-L6-v2, the three strategies return the same fragments in the same order, and only the scale of score differs. The baseline search scores the refund fragment 0.6445 under cosine, -0.3555 under dot-product and 1.1353 under euclidean-distance. A threshold chosen under one strategy therefore does not apply under another strategy. For a model whose vectors differ in length, a strategy other than the trained strategy changes the ranking. Search requests lists the score range of each strategy.

Full-text query language​

A full-text index reduces the words of each fragment and of each query to stems with the rules of a language. The language affects which fragments match:

  • Language per fragment. The worker stores the language of each fragment, and the API server parses the query with the language of each fragment. The language that detection would assign to a query of one or two words therefore does not change which fragments match. On the example pipeline, the queries Billing page, PDF CSV and invoices each match the invoice fragment.
  • Fragments indexed by an earlier release. These fragments have no stored language, so the API server parses the query with the language detected for the query text alone. A query of one or two words can be detected as a different language from the content, and the query then matches fewer of these fragments, or none. A new version of a document stores the language of each fragment.
  • Query length. Each added word must also appear in a matching fragment, so a query of a few specific words narrows the results.
  • Common words. A query that consists only of common words, such as the and and in English, produces no search terms and matches no fragment.
  • No language setting. No request field or pipeline setting selects the language of a full-text index. A BM25 index detects no language: the index's tokenizer splits and normalizes the words of every fragment and query in the same way.

A client application that sends one-word queries, such as product codes, runs a hybrid search, which also returns the closest fragments by meaning, as described in Set up hybrid search. Full-text languages describes detection, the supported languages and the treatment of other languages.