Skip to main content

Ingest documents reliably

This guide describes how a client application submits documents to a pipeline, confirms that processing completed, finds documents that failed or did not complete and submits those documents again. The guide also covers retries, idempotent updates, pacing, request size, text preparation, metadata design and license limits. Integrators need this guide to build an ingestion service, and operators need the troubleshooting entry to diagnose documents that remain unprocessed. Documents, versions and fragments describes the document model.

Requirements​

The examples use the support-kb pipeline from First search, with the classifications public and internal and a metadata schema that declares the string field product. $PIPELINE_ID holds the pipeline identifier. The shell variables for the base URL and the credentials are described in Authenticate.

The API key of the ingestion service holds read and execute permission on the pipeline, execute permission on the pipeline's text splitter and a classification allow-list that includes the classification of each document. A service that also expires and deletes documents holds permission 7 on the pipeline. Permissions by operation lists the permissions of each document operation.

Document status​

Document processing is asynchronous: the create request places a processing job in NATS JetStream, the queue between the API server and the workers, and returns before a worker splits and embeds the document. The status field of a document tells the client application what to do next.

StatusMeaningClient application action
pendingThe document is queued, or processing failed and the worker retries the documentCheck the document again later. Submit the document again when the document is still pending 24 hours after creation.
successThe fragments are stored and search returns the documentNone
failedProcessing failed 12 times, and message holds the reason; or Foundation4 recorded the document but could not place the processing job on the queue, and message is nullRemove the cause, then submit the document again
canceledA newer version of the document replaced this version, or the version was expired, before a worker processed itNone: the newer version is the one to check

Processing lifecycle describes each stage of processing.

Retries and redelivery​

The workers and the queue retry processing without action by the client application:

  • Processing failure. When processing of a document fails, the worker returns the job to the queue, and the queue delivers the job again after a delay. The delay starts at 10 seconds and doubles after each attempt, up to 10 minutes, so the attempts take place about 10 seconds, 30 seconds, 70 seconds, 2.5 minutes, 5 minutes and 10.5 minutes after the first attempt, and every 10 minutes after that.
  • Attempt limit. After the 12th failed attempt, about an hour after the first attempt, the worker sets the document status to failed, records the reason in message and removes the document's raw text. A text splitter whose package or data is missing from the gRPC service records that cause, such as Processing failed after 12 attempts: Provider unavailable: NLTK data 'tokenizers/punkt_tab' was not found under /data/nltk; mount the NLTK data there. Another cause inside the deployment, such as a database error, reads Processing failed after 12 attempts: internal error; see the worker logs, and the worker log holds the details.
  • Independent documents. A document that fails does not hold back other documents. The other documents of the same pipeline and the same worker batch are processed.
  • Queue age limit. The queue removes a job that is still waiting 24 hours after submission, and the object store removes the document's raw text after the same period. A document whose job is removed this way, for example because no worker ran for a day, remains pending and is not processed again.
  • Worker stop. A job that a worker has taken but not acknowledged, for example because the worker pod stopped, returns to the queue after the acknowledgment wait of 10 minutes and is delivered to another worker.
  • Repeated delivery. A job for a document that is no longer pending is acknowledged without processing, so a repeated delivery does not duplicate fragments. A job for a document that was deleted before processing, or for a pipeline that was deleted, is acknowledged without processing as well.
  • Replaced versions. A job for a version that a newer version replaced while the job waited is not processed. The worker sets the version's status to canceled.

A client application therefore submits a document again in two cases only: the document status is failed, or the document is still pending 24 hours after creation. A document that is pending for a shorter time is still in the queue, and the worker processes the document once the cause of the failure is removed, provided that the cause is removed before the last attempt.

Document submission​

Submit a document with POST /pipelines/{id}/documents and an external_identifier, the identifier of the content in the source system. The identifier makes the submission safe to repeat, as described in Idempotent submission.

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"classification": "public",
"external_identifier": "kb-account-lockout",
"metadata": {"product": "accounts"},
"contents": "Account lockout. After five failed sign-in attempts, an account is locked for 15 minutes. An administrator can unlock the account from the Users page."
}'

Expected result: status 201 and the new document with status set to pending:

{
"id": "01a0f18a-ca80-754e-ba2b-46e126f7cf9e",
"created_at": "2026-09-30T09:00:00.722235Z",
"updated_at": "2026-09-30T09:00:00.722235Z",
"expired_at": null,
"external_identifier": "kb-account-lockout",
"metadata": {"product": "accounts"},
"classification": "public",
"path": null,
"pipeline_id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
"text_splitter_id": "019252e9-b4a0-7713-9a69-d701b4f4a2d1",
"status": "pending",
"message": null,
"version": 1790758800722235
}

A status of failed in this response means that the processing job was not queued; submit the document again. Record the identifier and the version of the document:

export DOCUMENT_ID=<id from the response>
export VERSION=<version from the response>

A client application that writes to several pipelines can use POST /documents instead, with the pipeline identifier in the body. The request returns the same response:

curl -X POST "$FOUNDATION4_URL/documents" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"pipeline_id": "'"$PIPELINE_ID"'",
"classification": "public",
"external_identifier": "kb-account-lockout",
"metadata": {"product": "accounts"},
"contents": "Account lockout. After five failed sign-in attempts, an account is locked for 15 minutes. An administrator can unlock the account from the Users page."
}'

Processing check​

Retrieve the submitted version with GET /documents/{document_id} and read the status field:

curl "$FOUNDATION4_URL/documents/$DOCUMENT_ID?version=$VERSION" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and the document version. After processing, status is success:

{
"id": "01a0f18a-ca80-754e-ba2b-46e126f7cf9e",
"created_at": "2026-09-30T09:00:00.722235Z",
"updated_at": "2026-09-30T09:00:01.016043Z",
"expired_at": null,
"external_identifier": "kb-account-lockout",
"metadata": {"product": "accounts"},
"classification": "public",
"path": null,
"pipeline_id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
"text_splitter_id": "019252e9-b4a0-7713-9a69-d701b4f4a2d1",
"status": "success",
"message": null,
"version": 1790758800722235
}

The version parameter makes the check refer to the submitted version, even after a later submission creates a newer version or expires this version. Without the parameter, the request returns the latest current version, and HTTP 404 once the document has expired, unless the request sets include_expired=true.

A check more often than every 10 seconds returns no new information for a failing document, because the worker waits at least 10 seconds between two attempts. A client application that submits many documents checks the number of pending and failed documents instead of each document, as described in Initial load and in Failed documents.

Pending documents​

The document list, GET /pipelines/{id}/documents, filters by status and by creation time. The following request lists the documents that are still pending more than 24 hours after creation, oldest first, for a check made at 2026-09-30T09:15:00Z. The $ of each operator suffix is escaped for the shell, as described in Filters and ordering.

curl "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents?status=pending&created_at\$lt=2026-09-29T09:15:00Z&count=true&first=100&order_by=created_at&order_by=id" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and one page of pending documents. page_info.count holds the total number of matching documents, and query_info repeats the filters that the API server applied:

{
"data": [
{
"id": "01a0e8e3-3700-7bb0-a499-3343833c03f8",
"created_at": "2026-09-28T16:40:00.945114Z",
"updated_at": "2026-09-28T16:40:00.945114Z",
"expired_at": null,
"external_identifier": "kb-sso-setup",
"metadata": {"product": "accounts"},
"classification": "internal",
"path": null,
"pipeline_id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
"text_splitter_id": "019252e9-b4a0-7713-9a69-d701b4f4a2d1",
"status": "pending",
"message": null,
"version": 1790613600945114
}
],
"page_info": {
"after": "WyIyMDI2LTA5LTI4VDE2OjQwOjAwLjk0NTExNFoiLCIwMWEwZThlMy0zNzAwLTdiYjAtYTQ5OS0zMzQzODMzYzAzZjgiXQ",
"before": "WyIyMDI2LTA5LTI4VDE2OjQwOjAwLjk0NTExNFoiLCIwMWEwZThlMy0zNzAwLTdiYjAtYTQ5OS0zMzQzODMzYzAzZjgiXQ",
"count": 1,
"has_next": false,
"has_prev": false,
"order_by": ["created_at", "id"]
},
"query_info": {
"created_at$lt": "2026-09-29T09:15:00Z",
"path": null,
"status": ["pending"]
}
}

Ordering by created_at and then id keeps cursor paging stable, as described in Pagination. When has_next is true, the next page is requested with after set to page_info.after. The list excludes expired versions, so a document that was submitted again leaves this list. Without the created_at filter, the same request returns every pending document, including documents that the workers have not yet processed. The list covers the documents without a path; a pipeline whose documents have paths repeats the request with path=%3E, the URL-encoded pattern >, as described in Document paths.

Record the identifier of the pending document:

export STUCK_DOCUMENT_ID=<id of the pending document>

Failed documents​

The same list with status=failed returns the documents whose processing failed. The message field of each document gives the reason:

curl "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents?status=failed&count=true&first=100&order_by=created_at&order_by=id" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and one page of failed documents, in the format of Pending documents. The following excerpt shows the status and message fields of a document that failed after its attempts because the NLTK data of its text splitter was missing:

{
"status": "failed",
"message": "Processing failed after 12 attempts: Provider unavailable: NLTK data 'tokenizers/punkt_tab' was not found under /data/nltk; mount the NLTK data there"
}

A failed document with message set to null was never queued. The causes of processing failures are in the worker log, as described in Documents remaining in pending or failed status. After the cause is removed, each failed document is submitted again, as described in Resubmission. The list excludes expired versions, so a document that was submitted again leaves this list.

Resubmission​

Submit the document again with the same external_identifier, the same classification and the text from the source system:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"classification": "internal",
"external_identifier": "kb-sso-setup",
"metadata": {"product": "accounts"},
"contents": "Single sign-on setup. Administrators configure single sign-on on the Security page by uploading the identity provider metadata file and mapping the email attribute."
}'

Expected result: status 201 and a new version of the same document. The id is unchanged, and version identifies the new version:

{
"id": "01a0e8e3-3700-7bb0-a499-3343833c03f8",
"created_at": "2026-09-30T09:20:00.606425Z",
"updated_at": "2026-09-30T09:20:00.606425Z",
"expired_at": null,
"external_identifier": "kb-sso-setup",
"metadata": {"product": "accounts"},
"classification": "internal",
"path": null,
"pipeline_id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
"text_splitter_id": "019252e9-b4a0-7713-9a69-d701b4f4a2d1",
"status": "pending",
"message": null,
"version": 1790760000606425
}

The new version expires the version that remained pending. The version history, GET /documents/{document_id}/versions, shows both versions, newest first, when the request includes expired versions:

curl "$FOUNDATION4_URL/documents/$STUCK_DOCUMENT_ID/versions?include_expired=true" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and a JSON array. After processing, the new version has the status success, and the earlier version carries an expired_at time:

[
{
"id": "01a0e8e3-3700-7bb0-a499-3343833c03f8",
"created_at": "2026-09-30T09:20:00.606425Z",
"updated_at": "2026-09-30T09:20:00.786112Z",
"expired_at": null,
"external_identifier": "kb-sso-setup",
"metadata": {"product": "accounts"},
"classification": "internal",
"path": null,
"pipeline_id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
"text_splitter_id": "019252e9-b4a0-7713-9a69-d701b4f4a2d1",
"status": "success",
"message": null,
"version": 1790760000606425
},
{
"id": "01a0e8e3-3700-7bb0-a499-3343833c03f8",
"created_at": "2026-09-28T16:40:00.945114Z",
"updated_at": "2026-09-28T16:40:00.945114Z",
"expired_at": "2026-09-30T09:20:00.606425Z",
"external_identifier": "kb-sso-setup",
"metadata": {"product": "accounts"},
"classification": "internal",
"path": null,
"pipeline_id": "c4a7e2d1-6b3f-4f8e-a2d9-7e1b5c3f9a06",
"text_splitter_id": "019252e9-b4a0-7713-9a69-d701b4f4a2d1",
"status": "pending",
"message": null,
"version": 1790613600945114
}
]

The earlier version keeps its status with expired_at set: pending here, because the queue removed its job, or failed for a document that failed. An earlier version whose job still waits in the queue is never processed: at the job's next delivery, the worker sets the version's status to canceled. The resubmission of a failed document follows the same steps. When the document itself caused the failure, for example because it named a text splitter that cannot run, the new version corrects the cause, such as with another text_splitter_id.

A pending document without an external identifier cannot receive a new version. The client application deletes such a document with DELETE /pipelines/{id}/documents/{document_id} and submits the text as a new document, as described in Expiry and deletion.

Idempotent submission​

The following rules make every create request safe to repeat:

  • External identifier. Each document carries the source system's identifier in external_identifier. A repeated submission with the same identifier in the same pipeline creates a new version of the same document instead of a second document, so a request whose response was lost can be sent again.
  • Older versions. expire_older_versions keeps the default true, so the latest version replaces the previous versions in search. The previous versions leave search when the new version is submitted, and the document returns to search when the new version is processed. Versions describes both settings.
  • Classification. A new version carries the classification of the existing document. A different classification returns HTTP 400 with the message Classification mismatch.
  • Path. A new version carries the path of the document again, because a version without path has no path. Document paths describes paths across versions.
  • Order of versions. The client application sends the versions of one document one at a time and waits for each response. When two requests for the same identifier arrive together, the API server records one after the other, and the version recorded last becomes the current version, whatever the order in which the client application sent the requests.
  • Documents without an identifier. Each submission creates a new document. A client application that omits external_identifier records each returned id, and deletes the duplicate when a repeated request creates a second document.

Pacing and batching​

Foundation4 has no bulk create request: each document is one create request. Each create request places one job in the queue, and each worker takes up to 100 jobs at a time. A deployment runs 3 workers by default.

  • Rounds. The client application submits documents in rounds, such as 100 documents per round, and reads the pending count before the next round.
  • Backlog limit. The client application keeps the pending count at a level that the workers clear well within 24 hours, because the queue removes jobs older than 24 hours. The decrease of the pending count per minute during the first rounds measures the throughput of the deployment.
  • Parallel requests. Requests for different documents are independent and can run in parallel. Requests for the same external identifier follow the order rule of Idempotent submission.
  • Worker capacity. Throughput depends on the text splitter, the embedding model and the number of workers. The operator scales the workers as described in Scaling and performance.

The Processing metrics of the API server and the workers show submitted, processed and failed documents during a load.

Request size​

The API server accepts request bodies of up to 10 mebibytes (10,485,760 bytes) by default, and the operator changes the limit with the configuration key app.max_body_bytes. The limit applies to the whole JSON body, including the metadata and the escape characters that JSON adds to quotes, backslashes and line breaks. The limit counts bytes, not characters: accented letters and characters of non-Latin scripts take two to four bytes each. A larger request returns HTTP 413 with the message Failed to buffer the request body: length limit exceeded and creates no document.

A source document larger than the limit is divided into several documents. Each part carries an external identifier derived from the source identifier, such as kb-install-guide-part-1, and the metadata of the source document. When a revised source document produces fewer parts than before, the client application deletes the parts that no longer exist.

Text preparation​

A document consists of plain text, as described in Document content. The client application prepares the text before submission:

  • Extraction. The client application extracts the text of PDF, Office and HTML files. Foundation4 does not parse files.
  • Boilerplate. The client application removes navigation, page headers, page footers and repeated notices, because such text produces fragments that match many queries.
  • Titles. The document text begins with the title of the source, and the text of a long document keeps the section headings. A fragment carries only the text and the document metadata, so the title identifies the source in search results and in LLM prompts. Content preparation applies these rules to a knowledge base.
  • Empty text. A document with empty contents completes with the status success and no fragments. The client application skips sources whose extracted text is empty, such as scanned PDF files without a text layer.

Metadata design​

The following rules apply to document metadata:

  • Schema. The API server validates metadata against the pipeline's metadata schema. A document that does not satisfy the schema returns HTTP 400 with the message Invalid metadata, and details describes each failure under the name of the failing field. A required field that is missing is reported under its own name, for example {"region": "\"region\" is a required property"}. Metadata schema describes the schema rules.
  • Top-level fields. Values that search requests filter on are top-level fields of metadata where possible. Filters also reach nested fields by names with dots, but metadata indexes cover top-level fields only, as described in Filters.
  • Source reference. The metadata carries the values that a client application displays with a result, such as the title and the address of the source.
  • Stable values. Every fragment carries a copy of the document metadata, and a change of metadata requires a new version. Values that change often, such as view counts or workflow states, remain in the source system.

License limits​

Document creation requires a valid license. The create request returns HTTP 403 with the message Invalid license: Expired after the license expires, and with Invalid license: Exceeded("Documents") or Invalid license: Exceeded("Fragments") when the number of documents or fragments exceeds the license limit. HTTP 403 lists the license messages.

The API server refreshes the document and fragment counts every 10 seconds. Document creation is refused from the first refresh at which a count reaches the limit, and the refusal ends at the first refresh after the count falls below the limit.

Workers stop when the license expires, and a worker does not start with an expired license. Queued documents remain pending and are processed once a renewed license is installed and the workers run again, provided that their jobs are still in the queue, which keeps a job for 24 hours. After the renewal, the client application submits again the documents that remain pending more than 24 hours after creation.

Error handling​

The status code determines whether a failed create request is repeated:

  • HTTP 400, 404, 413 and 422. The request is invalid, for example because of metadata outside the schema, a classification mismatch, an unknown pipeline, a classification that the pipeline does not define, a body over the size limit or a missing field. The client application corrects the request, because the same request fails again.
  • HTTP 401 and 403. The key, the key's permissions, the key's classification allow-list or the license refuse the request. The client application stops submitting and reports the error, because repetition changes nothing until an administrator acts.
  • HTTP 500 and 503. A component of the deployment, such as the database or the queue, failed or is unavailable. The client application repeats the request with an increasing delay; with an external identifier, the repetition creates at most a new version.
  • Timeouts and connection errors. The document may or may not exist. The client application repeats the request with the same external identifier.

Errors lists every message with the cause.

Troubleshooting​

Documents remaining in pending or failed status​

  • Symptoms. The pending count does not fall, documents remain pending long after submission, or documents become failed.

  • Diagnosis. The requests in Pending documents and Failed documents list the documents concerned, and message gives the reason of each failure. The worker log names the cause of each processing failure:

    kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=500 --prefix \
    | grep -E "Error processing documents|Failed processing document|Document processing failed|Giving up|license is no longer valid"

    A line with Failed processing document names one document, the pipeline and the reason, and a line with Document processing failed gives the details of an internal error. A line with Error processing documents for pipeline names a failure of a whole pipeline group, such as a database error. A line with Giving up on a document after repeated processing failures marks a document that became failed. A line with The license is no longer valid: the worker stops; queued documents wait for a renewed license marks a worker that stopped because the license expired. When the log shows none of these lines, kubectl get pods -n foundation4ai shows whether the worker pods are running.

  • Cause. Processing fails for the document, the license has expired or no worker is running. The worker retries a failing document with growing delays and marks the document failed after 12 attempts, about an hour. The queue removes the job of a document that remains pending for 24 hours, for example while no worker runs, so that document is not processed again.

  • Resolution. Remove the cause shown in the log: restore the gRPC service or the embedding model, install a valid license, or start the workers. Documents that are still pending within their attempts are then processed without action. Submit again, as described in Resubmission, each failed document and each document that is pending more than 24 hours after creation, then confirm that the pending and failed counts fall. A document that fails because of its own content or settings is submitted with the cause corrected.

  • Actions to avoid. Submitting documents again while the cause persists, which adds versions that fail in the same way. Submitting again with expire_older_versions set to false, which keeps the failed version current. Purging or recreating the DOCUMENTS stream in NATS JetStream, which removes the jobs of every queued document and leaves those documents pending.