Skip to main content

Troubleshooting

This page lists the problems that operators meet while they install, start, reach, load and upgrade Foundation4, in that order. Each entry names the observed symptom, the commands that confirm the cause, the cause, the resolution and the actions that make the problem worse or hide the cause. The commands use the namespace foundation4ai, the releases foundation4ai-core and foundation4ai, and the values file foundation4ai.values.yaml of Install on Kubernetes. Operator command reference describes each command and the output of a healthy deployment.

First checks​

The following commands show the state of every pod, Job and volume claim, the latest events and the state of both releases:

kubectl get pods,jobs,pvc -n foundation4ai
kubectl get events -n foundation4ai --sort-by=.lastTimestamp | tail -20
helm list -n foundation4ai

Expected result on a healthy deployment: every pod is Running with all containers ready, every volume claim is Bound, and both releases show deployed. Helm deletes each installation Job after the Job succeeds, so the list shows no installation Job after a completed installation or upgrade. A listed installation Job is running or has failed, and the failed Job keeps its log until the next installation or upgrade of the application release.

Installation​

Pods or volume claims in pending status​

  • Symptoms. After the core release installation, NATS JetStream, Prometheus server or bundled PostgreSQL pods remain Pending, kubectl get pvc -n foundation4ai shows claims in Pending status, and Helm waits until the timeout.

  • Diagnosis.

    kubectl get storageclass
    kubectl describe pvc -n foundation4ai foundation4ai-core-nats-js-foundation4ai-core-nats-0
    kubectl describe pod -n foundation4ai foundation4ai-core-nats-0

    A list without a class marked (default), or claim events that report that no storage class is set, point to storage. Pod events that report Insufficient cpu or Insufficient memory point to node capacity instead.

  • Cause. The NATS JetStream claims (3 of 10 GiB), the Prometheus claim (8 GiB) and, in the evaluation profile, the bundled PostgreSQL claim (8 GiB) name no storage class unless the values name one, so the cluster needs a default StorageClass.

  • Resolution. In the evaluation profile, mark a StorageClass as the default or install a local volume provisioner. A production installation names the class in foundation4ai.values.yaml before the first installation, then runs the core release installation again:

    nats:
    config:
    jetstream:
    fileStore:
    pvc:
    storageClassName: <storage-class>
    prometheus:
    server:
    persistentVolume:
    storageClass: <storage-class>
    postgres:
    storage:
    className: <storage-class>

    Missing node capacity is resolved with more nodes or larger nodes; Scaling and performance describes the resource requests of each component.

  • Actions to avoid. Deleting the NATS JetStream volume claims after documents have been submitted, which deletes the queued documents, or the bundled PostgreSQL claim, which deletes the database. Changing the storage class of an existing installation in the values, which the StatefulSets do not apply to existing claims.

Image pull errors​

  • Symptoms. Hook Job pods or application pods report ErrImagePull, ImagePullBackOff or InvalidImageName, and the installation stops at the timeout. On a running deployment, new or restarted pods fail to pull while older pods keep running. A missing tag value stops the application installation before any pod starts, with a message such as set global.serverImageTag (or api-server.image.tag) to the API server image tag.

  • Diagnosis.

    kubectl describe pod -n foundation4ai <pod-name>
    kubectl get pod -n foundation4ai <pod-name> -o jsonpath='{.spec.imagePullSecrets}{"\n"}'

    The events show the image reference and the response of the registry, such as not found or unauthorized. The second command lists the pull secrets of the pod.

  • Cause. A repository or tag value is missing or wrong, the registry requires credentials that no pull secret provides, or the registry token in the pull secret has expired. Some registries issue tokens that expire after a few hours. The charts provide no default image tags, so a release without global.serverImageTag, global.grpcImageTag and global.dashboardImageTag, or the tag values of each chart, does not render.

  • Resolution. Set the tag values to the tags of the release, such as <release tag>, and correct the repository values. Create the pull secret and list the secret under global.imagePullSecrets of the application release, which the API server, worker, Job and dashboard pods use. A pull secret listed under a model or package image also reaches the API server, worker and Job pods. The core release takes pull secrets from the per-chart values listed in Charts, images and installation bundle. For short-lived registry tokens, refresh the pull secret on a schedule or pull from a mirror registry, as described on the same page. Then delete the failing pods, or repeat the failed installation step.

  • Actions to avoid. Setting the pull policy to Always with short-lived tokens, which makes every pod restart depend on a valid token. Omitting the tags in the expectation of a default.

Image volume errors on older Kubernetes versions​

  • Symptoms. Helm refuses the application chart with chart requires kubeVersion: >=1.31.0-0, followed by the version of the cluster. On a later version with api-server.models or api-server.packages set, the application installation fails with a validation error on a volume of the API server or worker Deployment, or the API server and worker pods remain in ContainerCreating.

  • Diagnosis.

    kubectl version
    kubectl get nodes -o wide
    kubectl describe pod -n foundation4ai -l app.kubernetes.io/name=api-server-worker

    kubectl version shows the server version, and the CONTAINER-RUNTIME column shows the runtime of each node. A Helm error that a volume must specify a volume type means that the Kubernetes API server dropped the image volume source. Pod events that report a failure to mount an image volume point to the kubelet or the container runtime.

  • Cause. Model and package images use the Kubernetes image volume source, which needs support in the Kubernetes API server, the kubelet and the container runtime. The application chart requires Kubernetes 1.31 or later, the first version that provides image volumes. A version that provides image volumes behind the ImageVolume feature gate accepts the volumes only where the gate is enabled.

  • Resolution. Install on a Kubernetes version and a container runtime that support image volumes, as listed in Deployment overview and requirements, or enable the ImageVolume feature gate on a version that provides the feature behind the gate.

  • Actions to avoid. Removing models from the values of a deployment whose pipelines use the models from the images. Those pipelines then fail to embed text.

Credentials missing after installation through a GitOps tool​

  • Symptoms. A GitOps tool, helm template or a client-side dry run fails to render the core chart with The Secret foundation4ai-secrets was not found in namespace <namespace>, followed by the advice to use helm install, helm upgrade or --dry-run=server. Where a GitOps tool renders the application release, the API server and the workers keep a previous credential after a change.

  • Diagnosis.

    kubectl get secret foundation4ai-secrets -n foundation4ai
    kubectl get secret foundation4ai-core -n foundation4ai \
    -o jsonpath='{.data.FOUNDATION4AI_DATABASE_URL}' | base64 -d | cut -d@ -f2; echo

    The first command shows whether the Secret of the secrets file exists in the namespace. The second command prints the host, the database and the parameters of the database URL without the credentials, or reports that the Secret foundation4ai-core is not found when the core release has not rendered. A render error that begins with Set names a key that foundation4ai-secrets lacks, and the reason.

  • Cause. The core chart reads foundation4ai-secrets from the cluster while Helm renders the templates, to build the connection URLs. Tools that render charts without cluster access, such as helm template, client-side dry runs and Argo CD, receive no Secret data, and the core chart stops the rendering. The application chart renders without cluster access, but the restart checksums of the credentials then stay constant, so a credential change does not restart the API server and the workers.

  • Resolution. Create foundation4ai-secrets before the core release, and install and upgrade the core release with the Helm command line against the cluster, or check the rendering with --dry-run=server, as described in Secrets and keys. Where a GitOps tool renders the application release, restart the API server and the workers after each credential change:

    kubectl rollout restart -n foundation4ai \
    deployment/foundation4ai-api-server deployment/foundation4ai-api-server-worker

    Expected result: both Deployments report restarted.

  • Actions to avoid. Storing credentials in a values file in a Git repository to replace the cluster lookup, which exposes every credential to the readers of the repository.

Database migration failure​

  • Symptoms. The application installation or upgrade stops in the pre-install or pre-upgrade hook, and the pods of job/foundation4ai-api-server-db-migration end in Error.

  • Diagnosis.

    kubectl logs -n foundation4ai job/foundation4ai-api-server-db-migration
    kubectl get secret foundation4ai-core -n foundation4ai \
    -o jsonpath='{.data.FOUNDATION4AI_DATABASE_URL}' | base64 -d | cut -d@ -f2; echo

    Error: "Failed connecting to database: <reason>" means that the connection failed, and the reason names the failure, such as a host name that cannot be resolved, a rejected password, a certificate error or a rule of pg_hba.conf. The second command prints the host, the database and the parameters of the URL without the credentials, or an empty line when the URL is missing. Error: "Failed applying migrations: <reason>" means that the connection succeeded and a migration failed:

    Reason containsMeaning
    A line that names the vector, vchord, pg_tokenizer and vchord_bm25 extensionsA migration failed on a database extension
    the schema has a schema_migrations table of another tooldatabase.schema holds the migration table of another tool
    the database was migrated by a newer Foundation4 releaseThe image is older than the database schema, for example after helm rollback
  • Cause. The URL is empty, because postgres.enabled is false and POSTGRES_URL is missing. A password inside POSTGRES_URL contains characters that are not percent-encoded; the core release encodes the passwords of the bundled services itself. The network, DNS, a network policy or pg_hba.conf blocks the connection. The TLS settings do not match the server, such as sslmode=verify-full with a server certificate from a private certificate authority that database.ca_cert does not contain. The database lacks the four search extensions, which only a superuser can create. Another tool keeps its migration table in the same schema. An older image runs against a database that a newer release has migrated.

  • Resolution. Correct the value in foundation4ai.secrets.env, the TLS settings described in Security hardening, the network path, or the database prerequisites described in Database, or give Foundation4 a schema of its own with database.schema. Apply the Secret, upgrade the core release and repeat the application installation or upgrade. A database that a newer release has migrated runs only with that release, or after the restore of the backup taken before the upgrade, as described in Backup, restore and upgrades.

  • Actions to avoid. Deleting the bundled PostgreSQL volume claim to start over, which creates an empty database with a new system ID. Granting the Foundation4 database user superuser rights to create the extensions, where an administrator can create the extensions once. Deleting rows from schema_migrations.

Installation stopped by the license check​

  • Symptoms. helm upgrade --install of the application release stops after the database migration and reports that the pre-install or pre-upgrade hook failed. job/foundation4ai-api-server-license-check remains with one failed pod.

  • Diagnosis.

    kubectl logs -n foundation4ai job/foundation4ai-api-server-license-check

    The log shows the system ID and the result of the check:

    SystemID: <system ID>
    Checking for a valid license... Invalid
    Error: "License check failed: Invalid"

    When the system ID cannot be computed, the first line reads SystemID: unavailable (<reason>). The second line ends with one of the following results:

    ResultMeaning
    OKThe license is valid; the hook completes
    MissingFOUNDATION4AI_APP_LICENSE is empty
    InvalidThe license is malformed or truncated, or was issued for another system ID
    ExpiredThe license expiry date has passed
    The system id for this license version could not be computedThe database does not provide the values that the license version binds to
  • Cause. After any result other than OK, the check exits with an error, the Job fails without a further attempt, and Helm stops the installation or upgrade. The failed Job keeps its log, with the system ID, until the next installation or upgrade of the application release replaces the Job. A new database, including a logical restore, has a new system ID, so a license issued before the change reads as Invalid.

  • Resolution. Send the system ID to the Foundation4 provider, as described in Licensing. Replace the value of FOUNDATION4AI_APP_LICENSE in foundation4ai.secrets.env, apply the Secret and upgrade the core release:

    kubectl kustomize . | kubectl apply -n foundation4ai -f -
    helm upgrade foundation4ai-core ./charts/foundation4ai-core \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

    Expected result: secret/foundation4ai-secrets configured and STATUS: deployed for foundation4ai-core. Then repeat the application installation or upgrade, which replaces the failed Job. A first installation that failed is removed with helm uninstall foundation4ai -n foundation4ai before the application release is installed again. After a successful check, Helm deletes the license check Job and continues with the master key Job. license check also runs through kubectl exec in a running API server pod and returns at once with the result.

  • Actions to avoid. Raising the Helm timeout, which does not change the result. Recreating the database, which changes the system ID again.

Master key Job failure​

  • Symptoms. The installation or upgrade stops after the license check, the pods of job/foundation4ai-api-server-create-admin-api-key end in Error, and Helm reports that the hook failed.

  • Diagnosis.

    kubectl logs -n foundation4ai job/foundation4ai-api-server-create-admin-api-key

    While PostgreSQL or the cache does not answer, the Job logs Waiting for the database or cache; retrying in <delay> and tries again, with a delay that doubles up to 10 seconds, for about four minutes. The last failure ends the attempt with one of the following errors:

    ErrorComponent
    Failed connecting to database: <reason>PostgreSQL: address, credentials, TLS or access rules
    Failed connecting to the cache: <reason>Redis-compatible cache: address or password
    Failed creating the master key: <reason>Storage of the master key in the database
    Failed to parse config: <reason>A configuration value that cannot be parsed
  • Cause. The master key Job connects to PostgreSQL and to the cache, then makes the stored master key match the configured identifier and secret. The Job needs no license, no NATS JetStream and no application secret, so a failure points to PostgreSQL, the cache or the stored key.

  • Resolution. Correct the component named in the log, then repeat the application installation or upgrade, which replaces the failed Job. Secrets and keys describes the values and the change procedure. A successful run prints Master key created successfully., Master key secret updated to match the configuration. or Master key already matches the configuration., and Helm then deletes the Job.

  • Actions to avoid. Creating a replacement administration key with admin generate-admin-api-key, because the API server and the workers authenticate with the configured master key at startup.

BM25 index creation failing after installation​

  • Symptoms. Creating a pipeline with full_text_search set to bm25, or adding a BM25 index, returns HTTP 500 Internal server error, while vector and full-text indexes are created.
  • Diagnosis. kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c server shows permission denied for table tokenizer at the time of the request.
  • Cause. The Foundation4 database user cannot write the schema tokenizer_catalog, where each BM25 index registers its tokenizer. The initialization script of the 1.1.0 core chart grants read access only, and so does a database prepared with read grants only.
  • Resolution. A superuser grants write access to tokenizer_catalog once, as described in BM25 index creation failing with a permission error, then the client application repeats the request.
  • Actions to avoid. Granting the Foundation4 database user superuser rights, where two grants suffice.

Startup​

API server or worker containers restarting​

  • Symptoms. After an installation, an upgrade or a restart, the server or worker container restarts repeatedly and the pod shows CrashLoopBackOff.

  • Diagnosis.

    kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c server --previous
    kubectl logs -n foundation4ai deploy/foundation4ai-api-server-worker -c worker --previous

    A license failure produces a warning with the system ID, followed by the error that stops the process (timestamps omitted):

    WARN foundation4ai_lib::connection: Failed to validate license for system id: <system ID>
    Error: "Failed to initialize server: License check failed: Invalid license"

    The worker reports the same failures without the prefix Failed to initialize server: . The text after the prefix names the cause:

    Log textCause
    License check failed: Invalid license, License check failed: Expired licenseThe license does not match the system ID in the warning, or has expired
    Failed to connect to Foundation4.ai's services: ORM error:PostgreSQL connection or migration
    Failed to connect to Foundation4.ai's services: The database was migrated by a newer Foundation4 releaseThe image is older than the database schema, for example after helm rollback
    Failed to connect to Foundation4.ai's services: Redis error: or Invalid parameter: Invalid Redis URLRedis-compatible cache
    Failed to connect to Foundation4.ai's services: Nats error:NATS JetStream
    Failed to connect to Foundation4.ai's services: Decryption error: Error decoding secret keyApplication secret format
    Failed to initialize with the master key: (API server), Failed to initialize Foundation4.ai: (worker)Master key
    Failed to parse config:Configuration file or value
    Failed to bind to addressListen address or port of the API server
    Failed to bind the metrics address, app.metrics_port must differ from app.portMetrics port of the API server

    Lines with Waiting for the database, cache or queue; retrying in <delay> before the error show that PostgreSQL, the cache or NATS JetStream did not answer.

  • Cause. At startup, the API server and each worker connect to PostgreSQL, apply pending migrations, check the license, connect to the cache and NATS JetStream, then authenticate with the master key. While PostgreSQL, the cache or NATS JetStream does not answer, the processes try again, with a delay that doubles up to 10 seconds, for about four minutes, and the startup probe gives the API server up to about 5 minutes before Kubernetes restarts the container. A license failure, any other failure and the end of the retries stop the process, and Kubernetes restarts the container. A master key that is expired, inactive or missing stops the workers; the API server keeps running, logs an error that names admin update-admin-api-key --clear-expiration and refuses requests made with the master key. A master key secret that does not match the stored key stops both.

  • Resolution. Correct the component named in the log: request a license for the system ID in the warning, as described in Licensing; restore PostgreSQL, the cache or NATS JetStream; restore the original application secret; restore an expired or inactive master key with admin update-admin-api-key, as described in Command-line interface; or correct the configuration. A value in foundation4ai.secrets.env, including a new master key secret, changes through the change procedure in Secrets and keys, whose upgrade of the application release lets the master key Job store the new values. An image older than the database is replaced by the current version, as described in Backup, restore and upgrades.

  • Actions to avoid. Generating a new application secret to clear a decryption error, which leaves the stored credentials of LLMs and embedding models unreadable. Changing the master key values in foundation4ai.secrets.env without an upgrade of the application release. Deleting the bundled PostgreSQL volume claim, which creates an empty database with a new system ID.

Access​

Login returning HTTP 401​

  • Symptoms. POST /login returns HTTP 401, and the dashboard sign-in fails.

  • Diagnosis.

    curl -sS -X POST "$FOUNDATION4_URL/login" \
    -H "x-api-key: $FOUNDATION4_API_KEY" \
    -H "x-api-key-secret: $FOUNDATION4_API_SECRET"
    kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \
    ./foundation4ai database check-connection

    The message of the error body names the cause, as listed in Errors: a missing header, a key identifier that is not a UUID, or Invalid API Key or Secret. The connection check prints Ok. when the configuration of the API server reaches PostgreSQL, and Error: "Failed connecting to database: <reason>" otherwise.

  • Cause. Invalid API Key or Secret covers an unknown key, a wrong secret, and an inactive or expired key. A database or cache failure while the API server checks the key returns HTTP 503 Unable to validate the API key; try again later instead, and the API server logs Could not validate the API key. A master key that is expired or inactive returns HTTP 401 while the API server runs, and the API server log shows an error at startup that names admin update-admin-api-key --clear-expiration. A license problem does not produce HTTP 401: an invalid license stops the API server at startup, and an expired license returns HTTP 403 Invalid license: Expired on the operations that check the license.

  • Resolution. Send the key identifier in x-api-key and the secret in x-api-key-secret, as described in Authenticate. The master key identifier and secret are the values of FOUNDATION4AI_APP_MASTER_KEY and FOUNDATION4AI_APP_MASTER_SECRET in foundation4ai.secrets.env. Restore an expired or inactive master key with admin update-admin-api-key, as described in Command-line interface.

  • Actions to avoid. Replacing the master key values in foundation4ai.secrets.env without an upgrade of the application release: the master key Job stores new values only at an upgrade, and a restart before the upgrade stops the workers and can stop the API server.

Ingress configured but unreachable​

  • Symptoms. Requests to the host name time out, or the ingress controller returns HTTP 404 or 503.

  • Diagnosis.

    kubectl get ingressclass
    kubectl describe ingress foundation4ai -n foundation4ai
    kubectl get endpoints foundation4ai-api-server foundation4ai-dashboard -n foundation4ai
    kubectl get httproute foundation4ai -n foundation4ai -o yaml

    kubectl describe ingress shows the class, the host and the backends of the Ingress. An endpoints entry without addresses means that no ready pod serves the Service. The status of the HTTPRoute lists each parent with an Accepted condition; False with a reason such as NoMatchingParent means that the parent reference matches no Gateway listener.

  • Cause. ingress.className names a class that no controller serves. The request host differs from ingress.hosts, whose chart default is chart-example.local. No API server pod is ready. The default httpRoute.parentRefs (Gateway gateway, listener http) match no Gateway. A network policy blocks the controller.

  • Resolution. Set className to an IngressClass from the list, set the host names in ingress.hosts or httpRoute.hostnames and send requests to those names, and set httpRoute.parentRefs to the Gateway and the listener. Allow the controller namespace in the network policies described in Security hardening. Check the API with GET /healthz or POST /login; the Ingress and the HTTPRoute send /dashboard to the dashboard and every other path to the API.

  • Actions to avoid. Changing the API server Service to LoadBalancer or NodePort to bypass the controller, which also bypasses the TLS of the ingress.

Dashboard sign-in failure through a port forward​

  • Symptoms. A port forward to the dashboard Service shows the dashboard page, but the sign-in fails with a correct key and the message Login failed: no API server answers at this address. Open the dashboard through the address (ingress or proxy) that also serves the API.
  • Diagnosis. The network panel of the browser's developer tools shows the sign-in request POST /login sent to the forwarded local address, where the dashboard server answers instead of the API server.
  • Cause. The dashboard is served under /dashboard/ and calls the API on the origin of the dashboard page, so the dashboard works only where one host name serves both /dashboard and the API. The chart's Ingress and HTTPRoute provide that host name; a port forward to one Service does not.
  • Resolution. Open the dashboard through the Ingress or the HTTPRoute host name, or through a local reverse proxy in front of port forwards to both Services, as described in API and dashboard access.
  • Actions to avoid. Exposing the dashboard or the API server through a LoadBalancer or NodePort Service to replace the port forward, which bypasses the TLS of the ingress.

Ingestion​

Documents remaining in pending or failed status​

  • Symptoms. The pending count does not fall, documents remain pending long after submission, or documents become failed.

  • Diagnosis.

    kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=500 --prefix \
    | grep -E "Error processing documents|Failed processing document|Document processing failed|Giving up|Undecodable"
    kubectl get pods -n foundation4ai -l app.kubernetes.io/name=api-server-worker
    kubectl logs -n foundation4ai deploy/foundation4ai-api-server-worker -c grpc --tail=100
    NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
    kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \
    nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" \
    consumer info DOCUMENTS document-processor

    A line with Failed processing document names one document, the pipeline and the reason, and a line with Document processing failed gives the details of an internal error of one document. A line with Error processing documents for pipeline names a failure of a whole pipeline group, such as a database error; the documents of the group then store the reason internal error; see the worker logs for an internal error. A line with Giving up on a document after repeated processing failures marks a document that became failed. A line with Undecodable queue message; terminating it marks a job that no worker can read; the queue drops that job, and the document stays pending. The gRPC service log shows the errors of Python text splitters and embedding providers, and the consumer information shows the jobs that wait and the jobs delivered more than once.

  • Cause. Processing fails for the document, for the text splitter or embedding model of the pipeline, or for the license, or no worker runs. A document that fails does not hold back the other documents of its batch. The worker retries a failing document after 10 seconds, then with a delay that doubles up to 10 minutes, and sets the status to failed after 12 attempts, about an hour after the first. The queue removes a job 24 hours after submission, so a document that waits that long, for example while no worker runs, stays pending and is not processed again.

  • Resolution. Remove the cause shown in the log: restore the gRPC service or the model server, install a valid license as described in the next entry, or start the workers. Documents that are still within their attempts are then processed without action. The client application submits again the failed documents and the documents that are pending more than 24 hours after creation, as described in Resubmission.

  • Actions to avoid. Submitting documents again while the cause persists, which adds versions that fail in the same way. Purging or recreating the DOCUMENTS stream, which removes the jobs of every queued document and leaves those documents pending.

Processing stopped after license expiry​

  • Symptoms. The workers log The license is no longer valid: the worker stops; queued documents wait for a renewed license and stop with Error: "License check failed: Expired license", the restarted worker containers stop at startup with the same error, and the pending count stops falling. Operations that check the license return HTTP 403 Invalid license: Expired, and a restarted API server container stops at startup with License check failed: Expired license.

  • Diagnosis.

    kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=200 --prefix \
    | grep -c "Expired license"
    LICENSE=$(grep '^FOUNDATION4AI_APP_LICENSE=' foundation4ai.secrets.env | cut -d= -f2-)
    kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \
    ./foundation4ai license details "$LICENSE"

    A count above zero and the output Error: "Expired" confirm the cause. The license details command needs a running API server pod.

  • Cause. The workers check the license before each batch, and each worker stops when the license has expired. The documents that the workers have not taken stay in the queue. The processes read the license at startup, so a renewed license takes effect only after a restart, and a process that restarts with an expired license stops at startup. The queue removes a job 24 hours after submission, so a queued document is processed only when a renewed license is deployed within 24 hours of its submission.

  • Resolution. Obtain a renewed license, as described in Licensing, and apply the license with the change procedure in Secrets and keys, whose upgrade of the application release restarts the API server and the workers. The workers then process the queued documents. Then submit again the failed documents and the documents that are still pending more than 24 hours after creation, as described in Resubmission.

  • Actions to avoid. Upgrading a release for other changes before the renewal, which stops at the license check. Purging the DOCUMENTS stream.

Workers timing out on NATS JetStream​

  • Symptoms. Worker containers log Waiting for the database, cache or queue; retrying in <delay> for about four minutes, then restart with Error: "Failed to connect to Foundation4.ai's services: Nats error: <reason>", where the reason reports a JetStream timeout or that JetStream is unavailable. The API server may report the same error after Failed to initialize server: . A running API server returns submitted documents with the status failed and logs Failed queueing document for processing.

  • Diagnosis.

    kubectl get pods -n foundation4ai -l app.kubernetes.io/component=nats
    kubectl logs -n foundation4ai foundation4ai-core-nats-0 -c nats --tail=50
    NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
    kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \
    nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" account info
    kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \
    nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" stream info DOCUMENTS

    The 3 NATS pods are expected Running with every container ready. nats account info shows whether JetStream is available to the account and how much storage is used, and nats stream info shows the server that holds the stream. A NATS log that reports no JetStream leader, or storage limits reached, points to the NATS cluster.

  • Cause. At startup, each worker and API server opens or creates the documents object store and the DOCUMENTS stream. JetStream answers only while a majority of the 3 NATS servers runs, and refuses new data when the JetStream volume is full, at 10 GiB per server by default. A network policy that blocks port 4222 or 6222 has the same effect.

  • Resolution. Restore the NATS pods, free or enlarge the JetStream storage described in Deployment overview and requirements, and allow ports 4222 and 6222 in the network policies. Then restart the workers:

    kubectl rollout restart deployment/foundation4ai-api-server-worker -n foundation4ai
    kubectl rollout status deployment/foundation4ai-api-server-worker -n foundation4ai

    Expected result: successfully rolled out, and the worker log shows no further NATS error.

  • Actions to avoid. Deleting the NATS JetStream volume claims or the DOCUMENTS stream, which removes the queued jobs. Reducing NATS to one replica on an existing installation, which can remove the server that holds the stream.

Worker errors while reading the queue​

  • Symptoms. The worker log shows Error retrieving messages from message queue, Error receiving message from message queue, Undecodable queue message; terminating it, Processing result does not match any queued message or Processing a batch panicked, while the worker containers keep running.

  • Diagnosis.

    kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=500 --prefix \
    | grep -E "message queue|Undecodable|does not match any queued|panicked"

    A retrieval error that repeats every 2 seconds points to NATS JetStream. Each Undecodable queue message line also counts one job in foundation4ai_document_queue_failed.

  • Cause. The worker logs queue errors and goes on. After a failed read from the queue, the worker waits 2 seconds and reads again. A job message that cannot be decoded is removed from the queue, because no worker can process the job, and the document of that job stays pending. A batch that stops on an internal error is not acknowledged, and the queue delivers its jobs again after the acknowledgement timeout.

  • Resolution. After repeated retrieval errors, check NATS JetStream as in the previous entry; the workers resume reading without a restart. When decoding errors or batch errors repeat, collect the worker logs and the output of nats stream info DOCUMENTS and nats consumer info DOCUMENTS document-processor, and contact the Foundation4 provider. The client application submits again the documents that stay pending, as described in Resubmission.

  • Actions to avoid. Purging the DOCUMENTS stream, which removes every queued job. Scaling the workers to 0 for more than 24 hours during the investigation, after which the queue has removed every waiting job.

Load​

Workers restarted for exceeding memory​

  • Symptoms. Worker pods restart during ingestion, and kubectl describe pod shows Last State: Terminated with Reason: OOMKilled and exit code 137, or the pod status is Evicted.

  • Diagnosis.

    kubectl get pods -n foundation4ai -l app.kubernetes.io/name=api-server-worker \
    -o jsonpath='{range .items[*]}{.metadata.name}{": "}{range .status.containerStatuses[*]}{.name}={.lastState.terminated.reason}{" "}{end}{"\n"}{end}'
    kubectl top pods -n foundation4ai --containers

    worker=OOMKilled or grpc=OOMKilled names the container that exceeded the memory limit. kubectl top, which needs the metrics server, shows the memory of each container during a batch.

  • Cause. Each worker takes up to 100 jobs per batch, and the memory of a batch grows with the size of the documents and with the embedding model. FastEmbed models run in the worker process, and Python embedding providers and text splitters run in the gRPC service. The charts set no resources. api-server.worker.resources, api-server.grpc.resources, api-server.server.resources and api-server.jobs.resources set the resources of each container, and api-server.resources applies to each container without a value of its own. A worker that stops during a batch leaves the jobs unacknowledged, the queue delivers the jobs again after the acknowledgement timeout, and the same batch can exceed the limit again.

  • Resolution. Set requests and a memory limit for the container named in the diagnosis, then upgrade the application release, which restarts the pods with the new values. The values below are an example, not a sizing recommendation; api-server.grpc.resources takes the same form for the gRPC service container, and Scaling and performance describes how to size the containers:

    api-server:
    worker:
    resources:
    requests:
    cpu: "1"
    memory: 2Gi
    limits:
    memory: 4Gi
  • Actions to avoid. Removing the memory limit on shared nodes, which lets one batch take memory from other pods. Adding worker replicas to reduce memory use, which does not change the size of a batch.

Low ingestion throughput​

  • Symptoms. The pending count grows during steady submission, and the message count of the DOCUMENTS stream rises.

  • Diagnosis. Read the stream state, then forward the metrics port of a worker in a separate terminal and read the worker metrics:

    NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
    kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \
    nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" stream info DOCUMENTS
    kubectl port-forward -n foundation4ai deploy/foundation4ai-api-server-worker 9090:9090
    curl -s http://localhost:9090/metrics | grep foundation4ai_document_queue

    The counters foundation4ai_document_queue_processed, _success, _skip, _removed, _retries and _failed count processing attempts and documents. _processed_latency is a histogram of the seconds per pipeline group, whose _sum divided by _count gives the average time of a group. The gauges _text_splitter_latency and _embedding_latency hold the seconds that the last pipeline group spent in the text splitter and the embedding model. Two readings a few minutes apart give the documents per second of that worker. A rising _retries or _failed counter means that the backlog comes from failures, as described in the entry on pending and failed documents. The port forward reaches one worker pod, and Monitoring and logging describes the metrics.

  • Cause. Each worker processes one batch at a time, and the pipeline groups of a batch one after another. Throughput is limited by the number of workers, by the CPU available to the worker and gRPC service containers, and by the time the embedding model needs per fragment.

  • Resolution. Add worker replicas with api-server.replicaCount.worker, set CPU requests for the containers, or use an embedding model that runs on a dedicated model server, as described in Scaling and performance.

  • Actions to avoid. Adding API server replicas, which do not process documents. Submitting more documents than the workers process in 24 hours, because the queue removes jobs after 24 hours.

HTTP 500 responses under load​

  • Symptoms. Under load, requests return HTTP 500 Internal server error after about 10 seconds, and the API server log shows Internal server error lines with kind="database" and an error that ends with Failed to acquire connection from pool: Connection pool timed out. A request whose key check waits for a connection returns HTTP 503 Unable to validate the API key; try again later instead, with Could not validate the API key in the log.

  • Diagnosis.

    psql "<administrator-connection-url>" -c "SHOW max_connections;"
    psql "<administrator-connection-url>" -c \
    "SELECT usename, state, count(*) FROM pg_stat_activity WHERE datname = '<database>' GROUP BY 1, 2;"
    kubectl get deployments,hpa -n foundation4ai

    The connection counts per state show whether the API server pools are full, and the replica counts give the number of pools.

  • Cause. Each API server process holds at most database.pool_size connections, 3 in the chart defaults, and a request waits up to 10 seconds for a free connection. Bursts or long searches exhaust the pool. Each additional API server replica adds a pool, so replicas, including those that an autoscaler adds, can reach max_connections of PostgreSQL, which then refuses new connections. Each worker holds 1 connection.

  • Resolution. Raise the pool size in a configs file, keep the total below max_connections as described in Database, then upgrade the application release, which restarts the API server and the workers with the changed configuration:

    api-server:
    configs:
    f4ai-10-database-pool.yaml: |
    database:
    pool_size: 10
    helm upgrade foundation4ai ./charts/foundation4ai \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
    kubectl rollout status deployment/foundation4ai-api-server -n foundation4ai

    Expected result: Helm reports STATUS: deployed, then deployment "foundation4ai-api-server" successfully rolled out.

  • Actions to avoid. Raising the API server replica count or the autoscaler maximum without a connection budget, which moves the failure to PostgreSQL.

Upgrades and changes​

Pods using a previous value after a change​

  • Symptoms. After a renewed license, a new password, a new URL or a change of api-server.configs, the API server and the workers behave as before.

  • Diagnosis.

    kubectl get pods -n foundation4ai -l app.kubernetes.io/instance=foundation4ai
    helm history foundation4ai -n foundation4ai

    The AGE of the API server and worker pods is older than the change.

  • Cause. Foundation4 reads configuration once at startup. The Deployments carry checksums of the configuration, of the api-server.secrets entries and of the mounted keys of foundation4ai-secrets and foundation4ai-core, so an upgrade of the application release that changes one of these values restarts the pods. A Secret changed without a later upgrade of the application release, such as a Secret applied after the last upgrade or a core release upgraded alone, does not restart the pods. An application release rendered without cluster access, such as through a GitOps tool, keeps the checksums of the Secrets constant.

  • Resolution. Run the change procedure in Secrets and keys, which ends with the upgrade of the application release, or restart the pods:

    kubectl rollout restart -n foundation4ai \
    deployment/foundation4ai-api-server deployment/foundation4ai-api-server-worker

    Expected result: both Deployments report restarted, and the new pods show the changed behavior.

  • Actions to avoid. Editing the Secrets foundation4ai-core or foundation4ai-api-server directly, which the next upgrade of the core or application release replaces.