Skip to main content

First search

This tutorial creates a pipeline, adds three documents and searches the documents four ways: a similarity search, the same search with a different classification, a search with a metadata filter and a full-text search. Integrators complete this tutorial to see the ingestion and retrieval path end to end. The tutorial uses the getting-started key and the shell variables from Authenticate.

Embedding model and text splitter​

A pipeline names an embedding model, which converts text into vectors, and a default text splitter, which divides each document into fragments. Every installation includes an embedding model named all-MiniLM-L6-v2 and a recursive character text splitter, with the same identifiers in every installation. Embedding models and text splitters describes both object types.

Find the embedding model by name:

curl "$FOUNDATION4_URL/embedding-models?name=all-MiniLM-L6-v2" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200. The following response omits the page_info and query_info fields that every list response carries:

{
  "data": [
    {
      "id": "9ba4a409-8773-415b-aa9d-ae3a2bdbc775",
      "created_at": "2026-09-29T11:40:00.512384Z",
      "updated_at": "2026-09-29T11:40:00.512384Z",
      "name": "all-MiniLM-L6-v2",
      "description": null,
      "provider": "FastEmbedEmbeddings",
      "model": "Qdrant/all-MiniLM-L6-v2-onnx",
      "size": 384,
      "parameters": {}
    }
  ]
}

Find the text splitter by provider. Text splitters have no name.

curl "$FOUNDATION4_URL/text-splitters?provider=RecursiveCharacterTextSplitter" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and a data array with the splitter whose id is 019252e9-b4a0-7713-9a69-d701b4f4a2d1. Set the two variables:

export EMBEDDING_MODEL_ID=9ba4a409-8773-415b-aa9d-ae3a2bdbc775
export TEXT_SPLITTER_ID=019252e9-b4a0-7713-9a69-d701b4f4a2d1

Pipeline​

The pipeline in this tutorial holds a small support knowledge base with the following settings:

  • Classifications. public and internal, where internal inherits public. A reader who holds internal sees public content as well, and a reader who holds only public does not see internal content. Classifications describes inheritance.
  • Full-text search. has_full_text_search is true. The setting cannot be changed after the pipeline is created.
  • Metadata schema. The schema declares one field, product, as a string. A metadata filter can reference only fields that the schema declares.
curl -X POST "$FOUNDATION4_URL/pipelines" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "support-kb",
    "description": "Support knowledge base",
    "embedding_model_id": "'"$EMBEDDING_MODEL_ID"'",
    "default_text_splitter_id": "'"$TEXT_SPLITTER_ID"'",
    "classifications": ["public", ["internal", "public"]],
    "has_full_text_search": true,
    "schema": {
      "type": "object",
      "properties": {"product": {"type": "string"}}
    }
  }'

Expected result: status 201 and the new pipeline. The embedding_models object maps the pipeline's vector embedding identifier to the embedding model. The following response shortens the embedding model to four fields:

{
  "id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
  "created_at": "2026-09-29T12:00:00.137529Z",
  "updated_at": "2026-09-29T12:00:00.137529Z",
  "name": "support-kb",
  "description": "Support knowledge base",
  "default_vector_embedding_id": "5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60",
  "default_text_splitter_id": "019252e9-b4a0-7713-9a69-d701b4f4a2d1",
  "schema": {
    "type": "object",
    "properties": {"product": {"type": "string"}}
  },
  "has_full_text_search": true,
  "embedding_models": {
    "5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60": {
      "id": "9ba4a409-8773-415b-aa9d-ae3a2bdbc775",
      "name": "all-MiniLM-L6-v2",
      "model": "Qdrant/all-MiniLM-L6-v2-onnx",
      "size": 384
    }
  }
}

Confirm that has_full_text_search is true and that embedding_models contains the model. Foundation4 ignores unknown fields in a request body, so a misspelled field name produces a pipeline without the feature rather than an error. Set the variable from the id field:

export PIPELINE_ID=<id from the response>

Documents​

Add three short documents. Each document carries one classification, a product metadata value and an external_identifier, the client application's own identifier for the document.

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "classification": "public",
    "external_identifier": "kb-password-reset",
    "metadata": {"product": "accounts"},
    "contents": "To reset a password, open Settings, choose Security and select Reset password. A reset link is sent to the registered email address and expires after 30 minutes."
  }'

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "classification": "internal",
    "external_identifier": "kb-invoice-export",
    "metadata": {"product": "billing"},
    "contents": "Invoices are generated on the first day of each month. Finance staff can export invoices as PDF or CSV files from the Billing page."
  }'

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "classification": "internal",
    "external_identifier": "kb-refund-policy",
    "metadata": {"product": "billing"},
    "contents": "Refunds for annual plans are prorated. Support agents must record the refund reason in the ticket before approving a refund."
  }'

Expected result: status 201 for each request. The response to the first request is the following:

{
  "id": "01a0ed0d-cf82-7c3d-8e4f-5a6b7c8d9e0f",
  "created_at": "2026-09-29T12:05:00.418273Z",
  "updated_at": "2026-09-29T12:05:00.418273Z",
  "expired_at": null,
  "external_identifier": "kb-password-reset",
  "metadata": {"product": "accounts"},
  "classification": "public",
  "pipeline_id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
  "text_splitter_id": null,
  "status": "pending",
  "message": null,
  "version": 1790683500418273
}

The pending status means that Foundation4 has accepted the document and queued the document for processing. version is the creation time in microseconds since the Unix epoch. Set a variable from the id field of the last response:

export DOC_ID=<id from the last response>

Processing​

A worker splits each document, computes the vectors and indexes the fragments. Search results include a document only after processing completes. Retrieve the last document until the status is success:

curl "$FOUNDATION4_URL/documents/$DOC_ID" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and the document, with status changed from pending to success.

The following request counts the documents of the pipeline that are still pending:

curl "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents?status=pending&count=true" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200, with page_info.count equal to 0 when every document is processed.

A document whose processing fails remains pending, and the worker retries the document every 10 seconds. When a document remains pending for more than a minute, the worker log shows the reason:

kubectl logs -n foundation4ai deploy/foundation4ai-api-server-worker -c worker

A similarity search embeds the query with the pipeline's embedding model and returns the fragments whose vectors are closest to the query vector. The search request names the reader's classifications; this search uses internal, so fragments from all three documents are candidates.

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How can finance export invoices?",
    "classification": "internal",
    "type": "similarity",
    "params": {"k": 3}
  }'

Expected result: status 200 and an array of up to 3 fragments, ordered by score. The following response is shortened to the first fragment, and the score is illustrative:

[
  {
    "id": "01a0ed0d-d814-7d6c-8b5a-4c3d2e1f0a9b",
    "classification": "internal",
    "metadata": {"product": "billing"},
    "document_id": "01a0ed0d-d75f-7d5e-8f60-718293a4b5c6",
    "page_content": "Invoices are generated on the first day of each month. Finance staff can export invoices as PDF or CSV files from the Billing page.",
    "version": 1790683502431906,
    "expired_at": null,
    "score": 0.31,
    "order_id": 0
  }
]

In a similarity search, score is the distance between the fragment and the query, so a lower score is a closer match. Each document in this tutorial is short enough to form a single fragment, with order_id 0.

params sets the number of fragments with k. A request that includes params always sets k: a params object without k returns no fragments. Search and retrieval describes every search parameter.

Classification scoping​

Repeat the search with the classification public:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How can finance export invoices?",
    "classification": "public",
    "type": "similarity",
    "params": {"k": 3}
  }'

Expected result: status 200 and an array with one fragment, from the kb-password-reset document. The two internal documents are not candidates, because public does not inherit internal. The fragment is returned although the fragment does not answer the question: a similarity search returns the closest fragments, however distant, unless the request sets a threshold.

Metadata filter​

Search as internal again, restricted to documents whose product is billing:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How can finance export invoices?",
    "classification": "internal",
    "type": "similarity",
    "params": {"k": 3},
    "filters": {"product": {"$eq": "billing"}}
  }'

Expected result: status 200 and an array of two fragments, from the kb-invoice-export and kb-refund-policy documents. A filter on a field that the pipeline schema does not declare returns HTTP 400 with the message Schema error. Metadata, filters and taxonomies describes the filter operators.

Full-text search​

A full-text search matches the words of the query against the text of the fragments, using PostgreSQL text search:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "record the refund reason",
    "classification": "internal",
    "type": "search",
    "params": {"k": 3}
  }'

Expected result: status 200 and an array with one fragment, from the kb-refund-policy document, in the same format as a similarity search result.

Full-text results differ from similarity results in two ways:

  • Score direction. In a full-text search, score is a text search rank, so a higher score is a better match. Scores from the two search types cannot be compared.
  • Matching. A fragment matches only when the fragment contains every significant word of the query, after stemming. Foundation4 detects the language of the query separately from the language of the documents, so a query of several words matches more reliably than a query of one or two words.

First RAG answer continues with the support-kb pipeline.