Skip to main content

Search requests

This page lists every field of a search request, every search type with the parameters and defaults of the type, the classification forms, the distance strategies, the response and the errors that a search request returns. Integrators use this page as the reference while writing search calls. Search and retrieval explains the search modes and when to use each mode.

Request​

POST /pipelines/{id}/search runs one search on the pipeline with the identifier id. The API key requires read and execute permission on the pipeline (permission 5). The request carries the authentication headers described in Authenticate and a JSON body. The example searches the support-kb pipeline from First search.

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "How do customers get their money back?",
"type": "similarity",
"classification": ["internal"],
"params": {"k": 5},
"filters": {"product": {"$eq": "billing"}}
}'

Expected result: status 200 and an array of up to 5 fragments, described in Response. On support-kb, the array holds two fragments, from the kb-refund-policy and kb-invoice-export documents, because the filter excludes the kb-password-reset document.

Body fields​

FieldTypeRequiredDefaultDescription
queryStringYesNoneThe search text. Similarity and MMR search convert the text into a vector, full-text search matches the words of the text, and hybrid search does both. History search ignores the text.
typeStringYesNoneThe search type: similarity, mmr, search, hybrid or history, described in Search types. Values are lowercase and case-sensitive.
classificationString, array or objectYesNoneThe classifications that the reader holds, described in Classification.
paramsObjectNoThe defaults of the search typeThe parameters of the search type, described in Parameters.
filtersObject or nullNoNo filterA metadata filter, described in Filter operators. An empty object {} applies no restriction.
distance_strategyString or nullNoThe first indexed strategyThe distance function for the vector indexes of the search, described in Distance strategies. Full-text and history search ignore the field.
pathString or nullNoDocuments without a pathA path pattern: a.b matches that path, a.b.* the paths one level below a.b, and a.b.> the paths at any depth below a.b. The search covers only the documents whose path matches. Without the field, or with null, the search covers only the documents without a path. Document paths describes the patterns.

A field that the body does not define returns an error, so a misspelled or misplaced field does not change the search silently. The following rules apply:

  • Top level. An unknown field at the top level of the body, such as filter instead of filters or k outside params, returns HTTP 400 with the message Unknown field `<name>`, and details.expected lists the fields of the body.
  • Parameters. A field inside params that the search type does not define, such as fetch_k in a similarity search or threshold in a hybrid search, returns HTTP 422 with a message that names the field and the fields of the search type, as in Failed to deserialize the JSON body: unknown field `fetch_k`, expected one of `embedding`, `index`, `k`, `threshold` for a similarity search. The list includes the parameters of earlier releases that the search type held, embedding and search_type, although a search that sends them returns HTTP 400, as described in Legacy parameters.
  • Classification object. The object form of classification ignores fields that the object does not define.

The request has no field for pagination, for a point in time or for the full-text language. A search always reads the current version of each document and returns one list of results.

Search types​

typeSearchSearch indexes readParameters
similarityThe k fragments whose vectors are closest to the query vectorOne vector indexindex, k, threshold
mmrMaximal marginal relevance (MMR) search: k fragments selected from the fetch_k closest fragments for relevance and diversityOne vector indexindex, k, fetch_k, lambda_mult, threshold
searchFull-text search: up to k fragments that match the words of the query, ranked by relevanceOne text index: a full-text index or a BM25 indexindex, k, threshold
hybridThe result lists of several search indexes, fused by reciprocal rank fusion (RRF)Any number and mix of vector and text indexesindexes, k
historyThe k most recent fragments, newest document version firstNonek

The API defines one further value, query, a placeholder type for agents, which returns HTTP 400 with the message Invalid search type or parameters in a search request. Any other value returns HTTP 422. Search and retrieval explains how each type selects and ranks fragments.

Parameters​

ParameterTypeSearch typesDefaultDescription
indexIdentifiersimilarity, mmr, searchThe pipeline's default index of the kindThe identifier of the search index to read: a vector index for similarity and mmr, a full-text or BM25 index for search.
indexesArray of identifiershybridThe default vector index and the default text indexThe identifiers of the search indexes to combine. Order and repeated identifiers have no effect.
kInteger, 1 to 65535All5The number of fragments to return. For hybrid, also the number of fragments that each index contributes.
fetch_kInteger, 1 to 65535, at least kmmr10The number of candidates from which MMR search selects k fragments.
lambda_multNumber, 0 to 1mmr0.5The balance between relevance and diversity. A value of 1 selects by relevance alone, and a value of 0 maximizes diversity.
thresholdNumbersimilarity, mmr, searchNoneFor similarity and MMR search, the maximum distance from the query. For full-text search, the minimum score. Fragments outside the threshold are excluded before k applies. Hybrid and history search do not define the field.

Every parameter that the request omits takes the default in this table, whether the request omits the whole params object or one field inside the object. The following request runs an MMR search for 5 fragments from 20 candidates on a named vector index:

{
"query": "How do customers get their money back?",
"type": "mmr",
"classification": "internal",
"params": {"index": "5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60", "k": 5, "fetch_k": 20}
}

Foundation4 checks the values before the search runs. A k or fetch_k of 0, a fetch_k smaller than k or a lambda_mult outside 0 to 1 returns HTTP 400 with Invalid parameter, and details.error names the rule, as listed in Errors. A value above 65535, or a value of another type, returns HTTP 422.

Legacy parameters​

Earlier releases selected an embedding model in params.embedding and a full-text method in params.search_type. A search that sends these parameters returns HTTP 400, so that a client application written for an earlier release fails visibly instead of searching other indexes than intended. The message names the replacement:

typeParameter sentMessage
similarity, mmrembedding`embedding` is not supported in pipeline searches: pass a search indexer id in `index`
searchsearch_type`search_type` is not supported in text searches: pass the text search indexer id in `index`
hybridA non-empty embedding or search_type array`embedding` and `search_type` are not supported in hybrid searches: pass the search indexer ids to combine in `indexes`

Classification​

The classification field names the classifications that the reader holds. Search returns only fragments whose classification is resolved from the request. Classifications describes inheritance and the role of the client application.

FormExampleMatching
String"classification": "internal"Hierarchical
Array of strings"classification": ["internal", "public"]Hierarchical
Object"classification": {"classifications": ["internal"], "search_type": 1}As set by search_type

In the object form, the array is named classifications, in the plural, and search_type is optional. Any other form returns HTTP 422: null or a number returns the message Failed to deserialize the JSON body: classification must be a string, an array of strings, or an object with classifications and search_type, and an object without classifications returns a message that names the missing field.

search_typeMatchingUndefined classification names
2 (default)Hierarchical: the named classifications and every classification that the named classifications inherit, transitivelyIgnored
1Exact: the named classifications onlyHTTP 404 with the message Classification not found
3Hierarchical with validation: as 2HTTP 404 with the message Classification not found

search_type is a bit field: the value 2 adds the inherited classifications, and the value 1 rejects undefined names. Any other value, including 0, returns HTTP 422 with the message Failed to deserialize the JSON body: invalid classification search type <value>: expected 1 (exact), 2 (hierarchical) or 3 (both).

An empty array, or an array in which no name is defined in the pipeline under hierarchical matching, resolves no classification, and the search returns an empty array with status 200.

The API key bounds the classifications that a request can name. Every named classification that is defined in the pipeline must be in the classification allow-list of the API key, or in a classification that the allow-list inherits, or the search returns HTTP 403, and details.error names the first classification outside the allow-list. A key whose allow-list is * can name every classification. Access control describes the allow-list.

Distance strategies​

The distance strategy determines how a vector search measures the distance between the query vector and each fragment vector. The strategy applies to every vector index that the search reads, and each strategy requires a distance index on the vector index, as described in Search indexes.

distance_strategyDistanceScore rangeBetter match
cosine1 minus the cosine similarity0 to 2Lower
euclidean-distanceEuclidean (L2) distance0 upwardLower
dot-productNegative inner productNegative for similar vectorsLower

A request without distance_strategy, or with null, uses the first strategy that the vector index has a distance index for, in the order cosine, euclidean-distance, dot-product. A new vector index has a distance index for cosine only. A request that names a strategy without a distance index returns HTTP 400, and the message lists the indexed strategies. The value max-inner-product is defined in the API but not supported, and returns HTTP 400.

The distance strategy matches the strategy for which the embedding model was trained, which is cosine for most sentence embedding models, including the default model.

Full-text search ("type": "search") has the following requirements and behavior:

  • Text index. The search reads the index named in params.index, or the pipeline's default text index. A pipeline without a text index returns HTTP 400, and POST /pipelines/{id}/search-indexers adds one, as described in Search indexes.
  • Full-text index. The query is plain text. After stemming and the removal of common words, every remaining word must appear in a fragment. The query syntax has no phrase, prefix, OR or NOT operators. The API server parses the query with the language configuration that each fragment was indexed with, so a query matches fragments of every indexed language. A fragment indexed by an earlier release, which stored no configuration, is matched with the configuration of the language that the API server detects in the query. Full-text languages lists the languages and describes detection. The score is the PostgreSQL rank of the fragment, divided by 1 plus the logarithm of the fragment length.
  • BM25 index. The index's tokenizer splits the query into tokens, and every fragment that shares a token with the query is ranked. The score is the BM25 score of the fragment, greater than 0.
  • Threshold. threshold sets a minimum score.
  • No embedding. Full-text search makes no embedding call, so distance_strategy does not apply.

Hybrid search ("type": "hybrid") searches each index in params.indexes, or the default vector index and the default text index, for the k best fragments. Foundation4 fuses the lists by reciprocal rank fusion: a fragment at rank r in a list, counting from 1, receives 1 / (60 + r), and the fused score is the sum over the lists. The response holds the first k fragments by descending fused score. The constant 60 is fixed, every list has the same weight, and a threshold in params returns HTTP 422. Set up hybrid search describes hybrid search with a worked example.

History search ("type": "history") returns the k most recent fragments that match the classifications and the filter, ordered by document version, newest first, and by position within a document version. query is required but ignored, the search reads no search index, and the fragments carry no score field.

Response​

A successful search returns status 200 and a JSON array of fragments. The array is not wrapped in the list envelope described in API conventions.

FieldTypeDescription
idIdentifierThe fragment identifier
classificationStringThe classification of the fragment's document
metadataObjectThe metadata of the fragment's document
pathString or nullThe path of the fragment's document, or null when the document has no path
document_idIdentifierThe identifier of the document
page_contentStringThe decrypted text of the fragment. The field is absent when the text is empty.
versionIntegerThe document version, in microseconds since the Unix epoch
expired_atTimestamp or nullAlways null in search results, because search reads current versions only
scoreNumberThe score of the fragment, described in the order rules below. Absent in history search.
order_idIntegerThe position of the fragment within the document version, starting at 0

The following rules govern the order and the number of results:

  • Similarity and MMR search. Fragments are ordered by ascending score, the distance from the query. MMR search returns the selected fragments in order of distance, not in order of selection, and score is the distance, not the MMR score.
  • Full-text search. Fragments are ordered by descending score, the rank or the BM25 score.
  • Hybrid search. Fragments are ordered by descending score, the fused score.
  • History search. Fragments are ordered by document version, newest first, and by order_id within a version.
  • Number of results. A search returns at most k fragments, and MMR search at most the smaller of k and fetch_k. A search returns fewer fragments when fewer fragments match the classifications, the filter and the threshold.
  • Comparison across types. Scores of vector search and full-text search run in opposite directions and cannot be compared. Hybrid search combines the two by rank position.

The following response contains the first fragment of the example in Request:

[
{
"id": "01a0ed0d-e007-7c3d-8e4f-6a7b8c9d0e1f",
"classification": "internal",
"metadata": {"product": "billing"},
"path": null,
"document_id": "01a0ed0d-df3f-7f80-9a1b-2c3d4e5f6a7b",
"page_content": "Refunds for annual plans are prorated. Support agents must record the refund reason in the ticket before approving a refund.",
"version": 1790683504447128,
"expired_at": null,
"score": 0.6445,
"order_id": 0
}
]

Each response carries the x-request-id header described in API conventions.

Errors​

The following errors are specific to search requests. Errors describes every error of the API.

StatusMessageCause
400Schema errorThe filter names a field that the pipeline schema does not declare, or compares a field with a value of another type
400Invalid parameterA search parameter is out of range, or a search index or a distance strategy cannot be used. details.error names the problem, as listed below.
400Unknown field `<name>`The body holds a top-level field that the search request does not define. details.expected lists the fields of the body.
400A legacy parameter messageparams contains embedding or search_type, as listed in Legacy parameters
400Invalid search type or parameterstype is query
403Unauthorized: execute access to Pipeline requiredA named classification is outside the classification allow-list of the API key. details.error names the classification, and details.id holds the pipeline identifier.
404Pipeline not foundThe pipeline does not exist, or the API key lacks read and execute permission on the pipeline. details.id holds the pipeline identifier.
404Classification not foundsearch_type is 1 or 3 and a named classification is not defined in the pipeline. The response does not name the classification.
409ConflictA search index or a distance index is not ready. details.error names the index, as listed below.
422Failed to deserialize the JSON body: <reason>A required field is missing, a field has the wrong type, type or distance_strategy has an unknown value, classification has an invalid form or search_type an invalid value, params holds a field that the search type does not define or a value above 65535, path is not a valid path pattern, or the filter does not follow the filter grammar. The reason names the problem without a line or column.
500Internal server errorThe embedding model could not convert the query into a vector, the query language could not be detected, or the database rejected the query, for example because stored metadata cannot be compared as the type that the schema declares. The API server log records the cause.

details.error of an Invalid parameter or Conflict error holds one of the following messages:

Statusdetails.errorCause
400`k` must be greater than 0, `fetch_k` must be greater than 0k or fetch_k is 0.
400`fetch_k` cannot be smaller than `k`An MMR search sets fetch_k below k.
400`lambda_mult` must be between 0 and 1 if specifiedAn MMR search sets lambda_mult outside 0 to 1.
400No search indexer given and the pipeline has no default vector search indexerA similarity or MMR search names no index, and the pipeline has no default vector index.
400No search indexer given and the pipeline has no default text search indexerA full-text search names no index, and the pipeline has no default text index.
400No default vector search indexer configured for this pipeline, No default text search indexer configured for this pipelineA hybrid search names no index, and the pipeline lacks the named default.
400Not a vector search indexer of this pipeline: <id>, Not a text search indexer of this pipeline: <id>, Not a search indexer of this pipeline: <id>The named index does not exist, belongs to another pipeline or is of the wrong kind for the search type.
400The <distance> distance strategy has no index on search indexer <id>, so searching it would scan the whole table; ...distance_strategy names a strategy without a distance index. The rest of the message lists the indexed strategies.
400Search indexer <id> has no distance index, so it cannot be searched; ...The vector index has no distance index at all.
400The max-inner-product distance strategy is not supporteddistance_strategy is max-inner-product.
409Search indexer <id> is still being built and cannot be searched yetThe index is building.
409Search indexer <id> failed to build and cannot be searchedThe index is failed.
409The index for the <distance> distance on search indexer <id> is still being builtThe distance index of the requested strategy is being built.

Other search operations​

Two other operations retrieve fragments with the same search engine:

  • Agent retrieval dry run. POST /agents/{id}/search runs the retrieval placeholders of an agent without calling an LLM. The search parameters come from the agent's placeholders, and filters maps each placeholder name to a filter. Agents and prompt templates describes the operation.
  • MCP search tool. The search_knowledge tool of the MCP server accepts a pipeline, a query, an array of classifications (matched hierarchically unless exact_classifications is true), a search type, search index identifiers, a path pattern, a metadata filter and the parameters of the search type as flat arguments. The tool names full-text search text and history search latest, and accepts no distance strategy. MCP tools describes the tool.