First search
This tutorial creates a pipeline, adds three documents and searches the documents four ways: a similarity search, the same search with a different classification, a search with a metadata filter and a full-text search. Integrators complete this tutorial to see the ingestion and retrieval path end to end. The tutorial uses the getting-started key and the shell variables from Authenticate.
Embedding model and text splitter
A pipeline names a default text splitter, which divides each document into fragments, and usually an embedding model, which converts the fragments and each query into vectors for the pipeline's vector index. Every installation includes an embedding model named all-MiniLM-L6-v2 and a recursive character text splitter, with the same identifiers in every installation. Embedding models and text splitters describes both object types.
Find the embedding model by name:
curl "$FOUNDATION4_URL/embedding-models?name=all-MiniLM-L6-v2" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"
Expected result: status 200. The following response omits the page_info and query_info fields that every list response carries:
{
"data": [
{
"id": "9ba4a409-8773-415b-aa9d-ae3a2bdbc775",
"created_at": "2026-09-29T11:40:00.512384Z",
"updated_at": "2026-09-29T11:40:00.512384Z",
"name": "all-MiniLM-L6-v2",
"description": null,
"provider": "FastEmbedEmbeddings",
"model": "Qdrant/all-MiniLM-L6-v2-onnx",
"size": 384,
"parameters": {}
}
]
}
Find the text splitter by provider. Text splitters have no name.
curl "$FOUNDATION4_URL/text-splitters?provider=RecursiveCharacterTextSplitter" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"
Expected result: status 200 and a data array with the splitter whose id is 019252e9-b4a0-7713-9a69-d701b4f4a2d1. Set the two variables:
export EMBEDDING_MODEL_ID=9ba4a409-8773-415b-aa9d-ae3a2bdbc775
export TEXT_SPLITTER_ID=019252e9-b4a0-7713-9a69-d701b4f4a2d1
Pipeline
The pipeline in this tutorial holds a small support knowledge base with the following settings:
- Classifications.
publicandinternal, whereinternalinheritspublic. A reader who holdsinternalseespubliccontent as well, and a reader who holds onlypublicdoes not seeinternalcontent. Classifications describes inheritance. - Search indexes.
embedding_model_idgives the pipeline a vector index for similarity search, and"full_text_search": "fts"gives the pipeline a full-text index. Each index becomes the pipeline's default index of its kind, which every search that names no index uses. Search indexes describes the index kinds and how to add an index later. - Metadata schema. The schema declares one field,
product, as a string. A metadata filter can reference only fields that the schema declares.
curl -X POST "$FOUNDATION4_URL/pipelines" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"name": "support-kb",
"description": "Support knowledge base",
"embedding_model_id": "'"$EMBEDDING_MODEL_ID"'",
"default_text_splitter_id": "'"$TEXT_SPLITTER_ID"'",
"classifications": ["public", ["internal", "public"]],
"full_text_search": "fts",
"schema": {
"type": "object",
"properties": {"product": {"type": "string"}}
}
}'
Expected result: status 201 and the new pipeline. The following response shortens the embedding model inside indexers to four fields:
{
"id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
"created_at": "2026-10-30T12:00:00.137529Z",
"updated_at": "2026-10-30T12:00:00.137529Z",
"name": "support-kb",
"description": "Support knowledge base",
"default_vector_embedding_index_id": "5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60",
"default_text_search_index_id": "8e3b6f1d-2c4a-4d7e-9b05-6a1f2e3d4c5b",
"default_text_splitter_id": "019252e9-b4a0-7713-9a69-d701b4f4a2d1",
"schema": {
"type": "object",
"properties": {"product": {"type": "string"}}
},
"indexers": {
"5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60": {
"type": "vchord",
"embedding_model": {
"id": "9ba4a409-8773-415b-aa9d-ae3a2bdbc775",
"name": "all-MiniLM-L6-v2",
"model": "Qdrant/all-MiniLM-L6-v2-onnx",
"size": 384
}
},
"8e3b6f1d-2c4a-4d7e-9b05-6a1f2e3d4c5b": {"type": "fts"}
},
"indexer_status": {
"5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60": "ready",
"8e3b6f1d-2c4a-4d7e-9b05-6a1f2e3d4c5b": "ready"
}
}
indexers maps the identifier of each search index to an object whose type holds the kind of the index: vchord with the embedding_model for the vector index, and fts for the full-text index. indexer_status reports both indexes as ready, because a new pipeline has no documents to index. default_vector_embedding_index_id and default_text_search_index_id hold the identifiers of the two default indexes.
Confirm that both default index fields hold an identifier. A request with a field that the create request does not define, such as a misspelled field name or the field name of earlier releases, has_full_text_search, returns HTTP 422 and creates no pipeline. The message names the unknown field. Set the variable from the id field:
export PIPELINE_ID=<id from the response>
Documents
Add three short documents. Each document carries one classification, a product metadata value and an external_identifier, the client application's own identifier for the document.
curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"classification": "public",
"external_identifier": "kb-password-reset",
"metadata": {"product": "accounts"},
"contents": "To reset a password, open Settings, choose Security and select Reset password. A reset link is sent to the registered email address and expires after 30 minutes."
}'
curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"classification": "internal",
"external_identifier": "kb-invoice-export",
"metadata": {"product": "billing"},
"contents": "Invoices are generated on the first day of each month. Finance staff can export invoices as PDF or CSV files from the Billing page."
}'
curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"classification": "internal",
"external_identifier": "kb-refund-policy",
"metadata": {"product": "billing"},
"contents": "Refunds for annual plans are prorated. Support agents must record the refund reason in the ticket before approving a refund."
}'
Expected result: status 201 for each request. The response to the first request is the following:
{
"id": "01a0ed0d-cf82-7c3d-8e4f-5a6b7c8d9e0f",
"created_at": "2026-09-29T12:05:00.418273Z",
"updated_at": "2026-09-29T12:05:00.418273Z",
"expired_at": null,
"external_identifier": "kb-password-reset",
"metadata": {"product": "accounts"},
"classification": "public",
"path": null,
"pipeline_id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
"text_splitter_id": "019252e9-b4a0-7713-9a69-d701b4f4a2d1",
"status": "pending",
"message": null,
"version": 1790683500418273
}
The pending status means that Foundation4 has accepted the document and queued the document for processing. version is the creation time in microseconds since the Unix epoch. path is null because the request names no document path. text_splitter_id holds the pipeline's default text splitter, which processes the document because the request names no text splitter. Set a variable from the id field of the last response:
export DOC_ID=<id from the last response>
Processing
A worker splits each document, computes the vectors and writes the fragments to every search index of the pipeline. Search results include a document only after processing completes. Retrieve the last document until the status is success:
curl "$FOUNDATION4_URL/documents/$DOC_ID" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"
Expected result: status 200 and the document, with status changed from pending to success.
The following request counts the documents of the pipeline that are still pending:
curl "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents?status=pending&count=true" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"
Expected result: status 200, with page_info.count equal to 0 when every document is processed.
A document whose processing fails remains pending while the worker retries the document, first after 10 seconds and then at growing intervals. After 12 failed attempts, about an hour, the status becomes failed and message holds the reason. When a document remains pending for more than a minute, the worker log shows the reason:
kubectl logs -n foundation4ai deploy/foundation4ai-api-server-worker -c worker
Similarity search
A similarity search embeds the query with the embedding model of the pipeline's default vector index and returns the fragments whose vectors are closest to the query vector. The search request names the reader's classifications; this search uses internal, so fragments from all three documents are candidates.
curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "How can finance export invoices?",
"classification": "internal",
"type": "similarity",
"params": {"k": 3}
}'
Expected result: status 200 and an array of 3 fragments, ordered by score: the invoice fragment with a score of about 0.2902, the refund fragment with about 0.8121 and the password fragment with about 1.1229. The following response is shortened to the first fragment:
[
{
"id": "01a0ed0d-d814-7d6c-8b5a-4c3d2e1f0a9b",
"classification": "internal",
"metadata": {"product": "billing"},
"path": null,
"document_id": "01a0ed0d-d75f-7d5e-8f60-718293a4b5c6",
"page_content": "Invoices are generated on the first day of each month. Finance staff can export invoices as PDF or CSV files from the Billing page.",
"version": 1790683502431906,
"expired_at": null,
"score": 0.2902,
"order_id": 0
}
]
In a similarity search, score is the distance between the fragment and the query, so a lower score is a closer match. Each document in this tutorial is short enough to form a single fragment, with order_id 0.
params sets the number of fragments with k. A parameter that the request omits takes the default of the search type, which is 5 for k. Search and retrieval describes every search type and parameter.
Classification scoping
Repeat the search with the classification public:
curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "How can finance export invoices?",
"classification": "public",
"type": "similarity",
"params": {"k": 3}
}'
Expected result: status 200 and an array with one fragment, from the kb-password-reset document. The two internal documents are not candidates, because public does not inherit internal. The fragment is returned although the fragment does not answer the question: a similarity search returns the closest fragments, however distant, unless the request sets a threshold.
Metadata filter
Search as internal again, restricted to documents whose product is billing:
curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "How can finance export invoices?",
"classification": "internal",
"type": "similarity",
"params": {"k": 3},
"filters": {"product": {"$eq": "billing"}}
}'
Expected result: status 200 and an array of two fragments, from the kb-invoice-export and kb-refund-policy documents. A filter on a field that the pipeline schema does not declare returns HTTP 400 with the message Schema error. Metadata, filters and taxonomies describes the filter operators.
Full-text search
A full-text search matches the words of the query against the text of the fragments. The search uses the pipeline's default text index, here the full-text index, which works with PostgreSQL text search:
curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "record the refund reason",
"classification": "internal",
"type": "search",
"params": {"k": 3}
}'
Expected result: status 200 and an array with one fragment, from the kb-refund-policy document, with a score of about 0.104, in the same format as a similarity search result.
Full-text results differ from similarity results in two ways:
- Score direction. In a full-text search,
scoreis a text search rank, so a higher score is a better match. Scores from the two search types cannot be compared. - Matching. A fragment matches only when the fragment contains every significant word of the query, after stemming. Foundation4 stems the query words with the language detected for each fragment, so the words of the query and of the fragment reduce to the same stems.
Set up hybrid search combines the two searches in one request. First RAG answer continues with the support-kb pipeline.