Documents, versions and fragments
A document is the unit of content that a client application submits to a pipeline. Foundation4 splits each document into fragments, which are the unit that search returns, and records each update to a document as a new version. This page describes the document lifecycle, versioning, expiry and deletion, point-in-time reads, and the structure of a fragment. Integrators need this page to design ingestion and to keep a pipeline synchronized with a source system.
Document content
A document consists of plain text, one classification, and optional metadata. Foundation4 does not parse files: a client application that ingests PDF, Office or HTML files extracts the text before submission.
A document is submitted with POST /pipelines/{id}/documents, or with POST /documents and a pipeline_id field in the body. The request body contains the following fields.
| Field | Required | Description |
|---|---|---|
contents | Yes | The plain text of the document |
classification | Yes | One classification defined in the pipeline |
external_identifier | No | The identifier of the content in the client application's own system. Used to submit new versions of the same document. |
metadata | No | A JSON object. Validated against the pipeline's metadata schema. |
path | No | The location of the document in the pipeline, such as kb.billing.tax.vat, described in Document paths. Each version carries its own path. |
text_splitter_id | No | A text splitter to use instead of the pipeline's default text splitter |
expire_older_versions | No | Whether the new version expires the previous versions. The default is true. |
The API server validates the classification and the metadata before accepting the document. A classification that the pipeline does not define returns HTTP 404, a classification outside the API key's allow-list returns HTTP 403, and metadata that does not satisfy the schema returns HTTP 400. None of these requests creates a document.
Processing lifecycle
Document processing is asynchronous. The create request returns HTTP 201 with the document identifier, the version and the status pending before any splitting or embedding takes place.
Each rectangle is a value of the document's status field. The circle marks the create request.
- Acceptance. The API server records the document with the status
pending, stores the raw text in the NATS JetStream object store, and places a processing job on the queue. If the job cannot be placed on the queue, the document status becomesfailed. - Processing. A worker reads jobs in batches of up to 100 documents. For each document, the worker splits the text with the document's text splitter, writes each fragment to every search index of the pipeline, with a vector from the embedding model of each vector index and the entries of each text index, encrypts the fragment text, and writes the fragments to PostgreSQL. A document that cannot be split or embedded fails alone, and the other documents of the batch continue.
- Completion. The worker sets the status to
successand deletes the raw text from the object store. A document with empty contents completes with the statussuccessand no fragments. - Retry. If processing fails, the document remains
pending, and the queue delivers the job again after a delay that starts at 10 seconds and doubles after each attempt, up to 10 minutes. - Failure. After the 12th failed attempt, about an hour after submission, the worker sets the status to
failed, records the reason in the document'smessagefield and deletes the raw text. A text splitter whose package or data is missing from the gRPC service records that cause, such asProcessing failed after 12 attempts: Provider unavailable: NLTK data 'tokenizers/punkt_tab' was not found under /data/nltk; mount the NLTK data there. Another internal cause, such as a database error, readsProcessing failed after 12 attempts: internal error; see the worker logs. - Cancellation. A version that a newer version replaces, or that is expired, while the version waits in the queue is never processed. The worker sets its status to
canceledand deletes the raw text.
When a pipeline is deleted, the workers drop the queued jobs of its documents. The client application determines when processing is complete by retrieving the document with GET /pipelines/{id}/documents/{document_id} and reading the status field. A document is searchable only after the status is success.
Raw text retention
Foundation4 retains the raw text of a document only until processing completes, and for at most 24 hours. After processing, the only stored copy of the text is the set of encrypted fragments. Foundation4 therefore cannot return a submitted document verbatim. Fragments can be retrieved in order, but text splitters that overlap adjacent fragments repeat text at each boundary. The client application's source system remains the authoritative copy of the original content.
Versions
Every submission creates a version. The version identifier is the creation time of the version, expressed in microseconds since the Unix epoch, and is returned in the version field.
The external_identifier field determines whether a submission creates a new document or a new version of an existing document:
- Without an external identifier. Each submission creates a new document with a new document identifier.
- With a new external identifier. The submission creates a new document.
- With an existing external identifier. The submission creates a new version of the document that has that external identifier in the same pipeline. The new version keeps the document identifier of the existing document.
A new version carries its own path and does not inherit the path of the previous version, so a version without path has no path. A new version with a different path moves the document, as described in Document paths.
A new version must carry the same classification as the existing document. A submission with a different classification returns an error, even when every earlier version has expired. Content that moves to a different classification requires deleting the document and submitting a new document.
When expire_older_versions is true, the new version replaces the previous versions. Foundation4 expires the previous versions and their fragments when the new version is submitted, so the document leaves search results at once and returns with the new text when the new version reaches success. During processing, a search returns no fragment of the document. If the new version fails, the document has no current version until the client application submits another version. When expire_older_versions is false, the previous versions remain current, and search returns fragments from every current version.
GET /documents/{document_id}/versions lists the current versions of a document, newest first. With include_expired=true, the list also includes the expired versions, each with its expiry time in expired_at. A document identifier that does not exist, or whose versions have all been deleted, returns HTTP 404 Document not found. GET /documents/{document_id} returns the latest current version, or the version named in the version query parameter, even when that version has expired. Without version, both GET /documents/{document_id} and GET /pipelines/{id}/documents/{document_id} return 404 for an expired document unless the request sets include_expired=true.
Expiry and deletion
Foundation4 provides two ways to remove content from search.
| Operation | Request | Effect |
|---|---|---|
| Expire | DELETE with expire=true | Marks every current version as expired. Search and document lists exclude expired versions. Version history remains readable. |
| Delete | DELETE without expire | Removes every version and every fragment of the document permanently |
Both operations apply to a single document (/pipelines/{id}/documents/{document_id} or /documents/{document_id}) or to every document in a pipeline (/pipelines/{id}/documents). The optional as_of query parameter limits either operation to versions created at or before the given time. An expired document leaves search results from the next request.
Expiry suits content that has been withdrawn from use but must remain available for audit. Deletion suits content that must no longer exist in the deployment, such as content subject to a retention limit or an erasure request.
Point-in-time reads
Document and fragment reads accept an as_of query parameter, expressed in microseconds since the Unix epoch. With as_of, Foundation4 returns each document as the document existed at that time: the latest version created at or before that time that had not yet expired. Point-in-time reads apply to the following endpoints:
GET /pipelines/{id}/documentsGET /pipelines/{id}/documents/{document_id}GET /pipelines/{id}/documents/{document_id}/fragmentsGET /documents/{document_id}/versions
A version returned for an earlier time shows its expiry time as it is now, so expired_at can be later than as_of. The fragment list of a document follows the same rule as the document: it returns the fragments of the version that the document read returns, and HTTP 404 for an expired document unless the request sets include_expired=true. The expired_at field of each fragment shows the expiry.
Search always operates on current content. Point-in-time reads support audit, reproduction of an earlier answer, and comparison between versions.
Fragments
A fragment is a section of a document produced by the text splitter. Search returns fragments, and each fragment carries the information that the client application needs to display, cite or re-check a result.
| Field | Description |
|---|---|
id | The fragment identifier |
document_id | The identifier of the source document |
version | The version of the source document, in microseconds since the Unix epoch |
order_id | The position of the fragment within the document version, starting at 0 |
classification | The classification of the source document |
metadata | A copy of the source document's metadata |
path | The path of the source document version, or null when the version has no path |
page_content | The decrypted fragment text, when the request includes it |
expired_at | The expiry time, or null for a current fragment |
score | The score for the query. Present in search results only. |
Every fragment inherits the classification and metadata of the document, so classification scoping and metadata filters apply to fragments without additional configuration. Every read of documents, versions and fragments returns only content whose classification the API key can read, as Access control describes. The Search and retrieval page describes the meaning of score in each search mode.
GET /pipelines/{id}/documents/{document_id}/fragments lists the fragments of a document version in order. The fragment text is included when the request sets contents=true. GET /pipelines/{id}/fragments retrieves specific fragments by identifier, which allows a client application to resolve fragment identifiers stored with an earlier answer. The response is a JSON array in the order of the requested identifiers, and each fragment includes its text in page_content. A request names at most 1,000 identifiers; a longer list returns HTTP 400. The response omits fragments of expired versions unless the request sets include_expired=true, and omits identifiers whose fragments have been deleted or whose classification the API key cannot read.