Pipelines
A pipeline is the container for one body of content and the rules that govern how Foundation4 processes and retrieves that content. This page describes the settings that define a pipeline, which settings can change after creation, and how Foundation4 stores the content of each pipeline. Integrators and administrators need this page before creating a pipeline, because two settings cannot change after creation.
Pipeline settings
A pipeline is created with POST /pipelines. The request defines the following settings.
| Setting | Field | Required | Description |
|---|---|---|---|
| Name | name | Yes | A name that is unique across the deployment |
| Description | description | No | Free text for administrators |
| Embedding model | embedding_model_id | No | The embedding model that converts fragments and queries into vectors. Required for similarity and MMR search. |
| Full-text search | has_full_text_search | No | Enables full-text search on the pipeline. The default is false. |
| Default text splitter | default_text_splitter_id | Yes | The text splitter applied to each document that does not name a text splitter |
| Classifications | classifications | Yes | The classifications that documents in the pipeline can carry, with any inheritance between those classifications |
| Metadata schema | schema | No | A JSON Schema that document metadata must satisfy. The default is an empty schema, which accepts any metadata. |
The embedding model and the text splitter must exist before the pipeline is created. A request that omits a required setting returns HTTP 422, a request that names an embedding model or a text splitter that does not exist returns HTTP 404, and a name that another pipeline already uses returns HTTP 409 with the message Conflict unique error. In each case, no pipeline is created. A new installation includes one embedding model and three text splitters, so a pipeline can be created without configuring a provider first. Embedding models and text splitters describes both object types.
A pipeline supports similarity and maximal marginal relevance (MMR) search when the pipeline has an embedding model, and supports full-text search when full-text search is enabled. A pipeline can support both. A pipeline created without an embedding model and with full-text search enabled is a full-text-only pipeline: Foundation4 splits and stores the documents but computes no vectors, which removes the embedding cost from ingestion. A similarity or MMR search on a pipeline without an embedding model returns HTTP 400 with the message Vector embedding not found. The search_knowledge tool of the MCP server requires an embedding model for every search type, so an AI assistant cannot search a full-text-only pipeline through the MCP server.
Settings fixed at creation
The embedding model and the full-text search setting determine the storage structure of the pipeline, so neither setting can change after the pipeline is created. A PATCH /pipelines/{id} request that includes either setting returns HTTP 200 and leaves the setting unchanged.
- Embedding model. Every vector in a pipeline is produced by the same embedding model, and vectors from different models cannot be compared. A different embedding model requires a new pipeline and a new ingestion of the content.
- Full-text search. Foundation4 builds the full-text index for each fragment at ingestion time. A pipeline created without full-text search has no full-text index, and a full-text search request to that pipeline returns HTTP 501.
The choice of these two settings is therefore part of pipeline design. A pipeline that may need full-text search later should enable full-text search at creation, at the cost of additional storage for the index.
Settings that can change
PATCH /pipelines/{id} changes the following settings:
- Name and description. The name must remain unique.
- Default text splitter. The new text splitter applies to documents submitted after the change. Fragments of documents already processed are not recomputed.
- Metadata schema. The new schema applies to the metadata of documents submitted after the change and to every subsequent metadata filter. Metadata already stored is not validated again.
The response to PATCH /pipelines/{id} shows the settings as they were before the change, so a client application reads the pipeline with GET /pipelines/{id} to confirm the new values. A second PATCH request by the same API key within 5 minutes of the first can restore the settings that the first request changed. A client application therefore sends every change to a pipeline in one request, and waits at least 5 minutes before the next change with the same key.
Classifications are managed through a separate endpoint, POST /pipelines/{id}/classifications, which adds and removes classifications and inheritance relationships. The Classifications page describes the effect of each change, including the deletion of documents when a classification is removed.
Storage
Foundation4 creates a dedicated set of tables and indexes in PostgreSQL for each pipeline. The size and query load of one pipeline therefore do not affect the indexes of another pipeline, and deleting a pipeline removes the pipeline's tables in one operation.
| Table | Contents | Index |
|---|---|---|
| Classifications | The pipeline's classifications, each with a dedicated encryption key | Unique index on the classification name |
| Hierarchy | The inheritance relationships between classifications | Primary key on each relationship |
| Texts | One row per fragment: encrypted fragment text, document identifier, version, position, expiry time and metadata | Indexes on document and version, and on classification and metadata |
| Vectors | One vector per fragment, created when the pipeline has an embedding model | Hierarchical navigable small world (HNSW) indexes for approximate nearest-neighbor search, one for each supported distance strategy |
| Full-text | One text search vector per fragment, created when full-text search is enabled | Generalized inverted index (GIN) for word lookup |
Fragment text is encrypted with the key of the fragment's classification. Vectors, full-text index entries and metadata are stored unencrypted, because PostgreSQL compares these values directly during search. Security at a glance summarizes the encryption model.
Deleting a pipeline
DELETE /pipelines/{id} deletes the pipeline, every document and version in the pipeline, and the pipeline's tables. The deletion is permanent. Foundation4 keeps no copy of the raw text of a processed document, so recovering deleted content requires a new ingestion from the source system or a database restore.