Troubleshooting
This page lists the problems that operators meet while they install, start, reach, load and upgrade Foundation4, in that order. Each entry names the observed symptom, the commands that confirm the cause, the cause, the resolution and the actions that make the problem worse or hide the cause. The commands use the namespace foundation4ai, the releases foundation4ai-core and foundation4ai, and the values file foundation4ai.values.yaml of Install on Kubernetes. Operator command reference describes each command and the output of a healthy deployment.
First checks
The following commands show the state of every pod, Job and volume claim, the latest events and the state of both releases:
kubectl get pods,jobs,pvc -n foundation4ai
kubectl get events -n foundation4ai --sort-by=.lastTimestamp | tail -20
helm list -n foundation4ai
Expected result on a healthy deployment: every pod is Running with all containers ready, every volume claim is Bound, and both releases show deployed. Helm deletes each installation Job after the Job succeeds, so the list shows no installation Job after a completed installation or upgrade. A listed installation Job is running or has failed, and the failed Job keeps its log until the next installation or upgrade of the application release.
Installation
Pods or volume claims in pending status
-
Symptoms. After the core release installation, NATS JetStream, Prometheus server or bundled PostgreSQL pods remain
Pending,kubectl get pvc -n foundation4aishows claims inPendingstatus, and Helm waits until the timeout. -
Diagnosis.
kubectl get storageclasskubectl describe pvc -n foundation4ai foundation4ai-core-nats-js-foundation4ai-core-nats-0kubectl describe pod -n foundation4ai foundation4ai-core-nats-0A list without a class marked
(default), or claim events that report that no storage class is set, point to storage. Pod events that reportInsufficient cpuorInsufficient memorypoint to node capacity instead. -
Cause. The NATS JetStream claims (3 of 10 GiB), the Prometheus claim (8 GiB) and, in the evaluation profile, the bundled PostgreSQL claim (8 GiB) name no storage class unless the values name one, so the cluster needs a default StorageClass.
-
Resolution. In the evaluation profile, mark a StorageClass as the default or install a local volume provisioner. A production installation names the class in
foundation4ai.values.yamlbefore the first installation, then runs the core release installation again:nats:config:jetstream:fileStore:pvc:storageClassName: <storage-class>prometheus:server:persistentVolume:storageClass: <storage-class>postgres:storage:className: <storage-class>Missing node capacity is resolved with more nodes or larger nodes; Scaling and performance describes the resource requests of each component.
-
Actions to avoid. Deleting the NATS JetStream volume claims after documents have been submitted, which deletes the queued documents, or the bundled PostgreSQL claim, which deletes the database. Changing the storage class of an existing installation in the values, which the StatefulSets do not apply to existing claims.
Image pull errors
-
Symptoms. Hook Job pods or application pods report
ErrImagePull,ImagePullBackOfforInvalidImageName, and the installation stops at the timeout. On a running deployment, new or restarted pods fail to pull while older pods keep running. A missing tag value stops the application installation before any pod starts, with a message such asset global.serverImageTag (or api-server.image.tag) to the API server image tag. -
Diagnosis.
kubectl describe pod -n foundation4ai <pod-name>kubectl get pod -n foundation4ai <pod-name> -o jsonpath='{.spec.imagePullSecrets}{"\n"}'The events show the image reference and the response of the registry, such as
not foundorunauthorized. The second command lists the pull secrets of the pod. -
Cause. A repository or tag value is missing or wrong, the registry requires credentials that no pull secret provides, or the registry token in the pull secret has expired. Some registries issue tokens that expire after a few hours. The charts provide no default image tags, so a release without
global.serverImageTag,global.grpcImageTagandglobal.dashboardImageTag, or the tag values of each chart, does not render. -
Resolution. Set the tag values to the tags of the release, such as
<release tag>, and correct the repository values. Create the pull secret and list the secret underglobal.imagePullSecretsof the application release, which the API server, worker, Job and dashboard pods use. A pull secret listed under a model or package image also reaches the API server, worker and Job pods. The core release takes pull secrets from the per-chart values listed in Charts, images and installation bundle. For short-lived registry tokens, refresh the pull secret on a schedule or pull from a mirror registry, as described on the same page. Then delete the failing pods, or repeat the failed installation step. -
Actions to avoid. Setting the pull policy to
Alwayswith short-lived tokens, which makes every pod restart depend on a valid token. Omitting the tags in the expectation of a default.
Image volume errors on older Kubernetes versions
-
Symptoms. Helm refuses the application chart with
chart requires kubeVersion: >=1.31.0-0, followed by the version of the cluster. On a later version withapi-server.modelsorapi-server.packagesset, the application installation fails with a validation error on a volume of the API server or worker Deployment, or the API server and worker pods remain inContainerCreating. -
Diagnosis.
kubectl versionkubectl get nodes -o widekubectl describe pod -n foundation4ai -l app.kubernetes.io/name=api-server-workerkubectl versionshows the server version, and theCONTAINER-RUNTIMEcolumn shows the runtime of each node. A Helm error that a volume must specify a volume type means that the Kubernetes API server dropped the image volume source. Pod events that report a failure to mount an image volume point to the kubelet or the container runtime. -
Cause. Model and package images use the Kubernetes image volume source, which needs support in the Kubernetes API server, the kubelet and the container runtime. The application chart requires Kubernetes 1.31 or later, the first version that provides image volumes. A version that provides image volumes behind the
ImageVolumefeature gate accepts the volumes only where the gate is enabled. -
Resolution. Install on a Kubernetes version and a container runtime that support image volumes, as listed in Deployment overview and requirements, or enable the
ImageVolumefeature gate on a version that provides the feature behind the gate. -
Actions to avoid. Removing
modelsfrom the values of a deployment whose pipelines use the models from the images. Those pipelines then fail to embed text.
Credentials missing after installation through a GitOps tool
-
Symptoms. A GitOps tool,
helm templateor a client-side dry run fails to render the core chart withThe Secret foundation4ai-secrets was not found in namespace <namespace>, followed by the advice to usehelm install,helm upgradeor--dry-run=server. Where a GitOps tool renders the application release, the API server and the workers keep a previous credential after a change. -
Diagnosis.
kubectl get secret foundation4ai-secrets -n foundation4aikubectl get secret foundation4ai-core -n foundation4ai \-o jsonpath='{.data.FOUNDATION4AI_DATABASE_URL}' | base64 -d | cut -d@ -f2; echoThe first command shows whether the Secret of the secrets file exists in the namespace. The second command prints the host, the database and the parameters of the database URL without the credentials, or reports that the Secret
foundation4ai-coreis not found when the core release has not rendered. A render error that begins withSetnames a key thatfoundation4ai-secretslacks, and the reason. -
Cause. The core chart reads
foundation4ai-secretsfrom the cluster while Helm renders the templates, to build the connection URLs. Tools that render charts without cluster access, such ashelm template, client-side dry runs and Argo CD, receive no Secret data, and the core chart stops the rendering. The application chart renders without cluster access, but the restart checksums of the credentials then stay constant, so a credential change does not restart the API server and the workers. -
Resolution. Create
foundation4ai-secretsbefore the core release, and install and upgrade the core release with the Helm command line against the cluster, or check the rendering with--dry-run=server, as described in Secrets and keys. Where a GitOps tool renders the application release, restart the API server and the workers after each credential change:kubectl rollout restart -n foundation4ai \deployment/foundation4ai-api-server deployment/foundation4ai-api-server-workerExpected result: both Deployments report
restarted. -
Actions to avoid. Storing credentials in a values file in a Git repository to replace the cluster lookup, which exposes every credential to the readers of the repository.
Database migration failure
-
Symptoms. The application installation or upgrade stops in the pre-install or pre-upgrade hook, and the pods of
job/foundation4ai-api-server-db-migrationend inError. -
Diagnosis.
kubectl logs -n foundation4ai job/foundation4ai-api-server-db-migrationkubectl get secret foundation4ai-core -n foundation4ai \-o jsonpath='{.data.FOUNDATION4AI_DATABASE_URL}' | base64 -d | cut -d@ -f2; echoError: "Failed connecting to database: <reason>"means that the connection failed, and the reason names the failure, such as a host name that cannot be resolved, a rejected password, a certificate error or a rule ofpg_hba.conf. The second command prints the host, the database and the parameters of the URL without the credentials, or an empty line when the URL is missing.Error: "Failed applying migrations: <reason>"means that the connection succeeded and a migration failed:Reason contains Meaning A line that names the vector,vchord,pg_tokenizerandvchord_bm25extensionsA migration failed on a database extension the schema has a schema_migrations table of another tooldatabase.schemaholds the migration table of another toolthe database was migrated by a newer Foundation4 releaseThe image is older than the database schema, for example after helm rollback -
Cause. The URL is empty, because
postgres.enabledisfalseandPOSTGRES_URLis missing. A password insidePOSTGRES_URLcontains characters that are not percent-encoded; the core release encodes the passwords of the bundled services itself. The network, DNS, a network policy orpg_hba.confblocks the connection. The TLS settings do not match the server, such assslmode=verify-fullwith a server certificate from a private certificate authority thatdatabase.ca_certdoes not contain. The database lacks the four search extensions, which only a superuser can create. Another tool keeps its migration table in the same schema. An older image runs against a database that a newer release has migrated. -
Resolution. Correct the value in
foundation4ai.secrets.env, the TLS settings described in Security hardening, the network path, or the database prerequisites described in Database, or give Foundation4 a schema of its own withdatabase.schema. Apply the Secret, upgrade the core release and repeat the application installation or upgrade. A database that a newer release has migrated runs only with that release, or after the restore of the backup taken before the upgrade, as described in Backup, restore and upgrades. -
Actions to avoid. Deleting the bundled PostgreSQL volume claim to start over, which creates an empty database with a new system ID. Granting the Foundation4 database user superuser rights to create the extensions, where an administrator can create the extensions once. Deleting rows from
schema_migrations.
Installation stopped by the license check
-
Symptoms.
helm upgrade --installof the application release stops after the database migration and reports that the pre-install or pre-upgrade hook failed.job/foundation4ai-api-server-license-checkremains with one failed pod. -
Diagnosis.
kubectl logs -n foundation4ai job/foundation4ai-api-server-license-checkThe log shows the system ID and the result of the check:
SystemID: <system ID>Checking for a valid license... InvalidError: "License check failed: Invalid"When the system ID cannot be computed, the first line reads
SystemID: unavailable (<reason>). The second line ends with one of the following results:Result Meaning OKThe license is valid; the hook completes MissingFOUNDATION4AI_APP_LICENSEis emptyInvalidThe license is malformed or truncated, or was issued for another system ID ExpiredThe license expiry date has passed The system id for this license version could not be computedThe database does not provide the values that the license version binds to -
Cause. After any result other than
OK, the check exits with an error, the Job fails without a further attempt, and Helm stops the installation or upgrade. The failed Job keeps its log, with the system ID, until the next installation or upgrade of the application release replaces the Job. A new database, including a logical restore, has a new system ID, so a license issued before the change reads asInvalid. -
Resolution. Send the system ID to the Foundation4 provider, as described in Licensing. Replace the value of
FOUNDATION4AI_APP_LICENSEinfoundation4ai.secrets.env, apply the Secret and upgrade the core release:kubectl kustomize . | kubectl apply -n foundation4ai -f -helm upgrade foundation4ai-core ./charts/foundation4ai-core \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result:
secret/foundation4ai-secrets configuredandSTATUS: deployedforfoundation4ai-core. Then repeat the application installation or upgrade, which replaces the failed Job. A first installation that failed is removed withhelm uninstall foundation4ai -n foundation4aibefore the application release is installed again. After a successful check, Helm deletes the license check Job and continues with the master key Job.license checkalso runs throughkubectl execin a running API server pod and returns at once with the result. -
Actions to avoid. Raising the Helm timeout, which does not change the result. Recreating the database, which changes the system ID again.
Master key Job failure
-
Symptoms. The installation or upgrade stops after the license check, the pods of
job/foundation4ai-api-server-create-admin-api-keyend inError, and Helm reports that the hook failed. -
Diagnosis.
kubectl logs -n foundation4ai job/foundation4ai-api-server-create-admin-api-keyWhile PostgreSQL or the cache does not answer, the Job logs
Waiting for the database or cache; retrying in <delay>and tries again, with a delay that doubles up to 10 seconds, for about four minutes. The last failure ends the attempt with one of the following errors:Error Component Failed connecting to database: <reason>PostgreSQL: address, credentials, TLS or access rules Failed connecting to the cache: <reason>Redis-compatible cache: address or password Failed creating the master key: <reason>Storage of the master key in the database Failed to parse config: <reason>A configuration value that cannot be parsed -
Cause. The master key Job connects to PostgreSQL and to the cache, then makes the stored master key match the configured identifier and secret. The Job needs no license, no NATS JetStream and no application secret, so a failure points to PostgreSQL, the cache or the stored key.
-
Resolution. Correct the component named in the log, then repeat the application installation or upgrade, which replaces the failed Job. Secrets and keys describes the values and the change procedure. A successful run prints
Master key created successfully.,Master key secret updated to match the configuration.orMaster key already matches the configuration., and Helm then deletes the Job. -
Actions to avoid. Creating a replacement administration key with
admin generate-admin-api-key, because the API server and the workers authenticate with the configured master key at startup.
BM25 index creation failing after installation
- Symptoms. Creating a pipeline with
full_text_searchset tobm25, or adding a BM25 index, returns HTTP 500Internal server error, while vector and full-text indexes are created. - Diagnosis.
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c servershowspermission denied for table tokenizerat the time of the request. - Cause. The Foundation4 database user cannot write the schema
tokenizer_catalog, where each BM25 index registers its tokenizer. The initialization script of the 1.1.0 core chart grants read access only, and so does a database prepared with read grants only. - Resolution. A superuser grants write access to
tokenizer_catalogonce, as described in BM25 index creation failing with a permission error, then the client application repeats the request. - Actions to avoid. Granting the Foundation4 database user superuser rights, where two grants suffice.
Startup
API server or worker containers restarting
-
Symptoms. After an installation, an upgrade or a restart, the
serverorworkercontainer restarts repeatedly and the pod showsCrashLoopBackOff. -
Diagnosis.
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c server --previouskubectl logs -n foundation4ai deploy/foundation4ai-api-server-worker -c worker --previousA license failure produces a warning with the system ID, followed by the error that stops the process (timestamps omitted):
WARN foundation4ai_lib::connection: Failed to validate license for system id: <system ID>Error: "Failed to initialize server: License check failed: Invalid license"The worker reports the same failures without the prefix
Failed to initialize server:. The text after the prefix names the cause:Log text Cause License check failed: Invalid license,License check failed: Expired licenseThe license does not match the system ID in the warning, or has expired Failed to connect to Foundation4.ai's services: ORM error:PostgreSQL connection or migration Failed to connect to Foundation4.ai's services: The database was migrated by a newer Foundation4 releaseThe image is older than the database schema, for example after helm rollbackFailed to connect to Foundation4.ai's services: Redis error:orInvalid parameter: Invalid Redis URLRedis-compatible cache Failed to connect to Foundation4.ai's services: Nats error:NATS JetStream Failed to connect to Foundation4.ai's services: Decryption error: Error decoding secret keyApplication secret format Failed to initialize with the master key:(API server),Failed to initialize Foundation4.ai:(worker)Master key Failed to parse config:Configuration file or value Failed to bind to addressListen address or port of the API server Failed to bind the metrics address,app.metrics_port must differ from app.portMetrics port of the API server Lines with
Waiting for the database, cache or queue; retrying in <delay>before the error show that PostgreSQL, the cache or NATS JetStream did not answer. -
Cause. At startup, the API server and each worker connect to PostgreSQL, apply pending migrations, check the license, connect to the cache and NATS JetStream, then authenticate with the master key. While PostgreSQL, the cache or NATS JetStream does not answer, the processes try again, with a delay that doubles up to 10 seconds, for about four minutes, and the startup probe gives the API server up to about 5 minutes before Kubernetes restarts the container. A license failure, any other failure and the end of the retries stop the process, and Kubernetes restarts the container. A master key that is expired, inactive or missing stops the workers; the API server keeps running, logs an error that names
admin update-admin-api-key --clear-expirationand refuses requests made with the master key. A master key secret that does not match the stored key stops both. -
Resolution. Correct the component named in the log: request a license for the system ID in the warning, as described in Licensing; restore PostgreSQL, the cache or NATS JetStream; restore the original application secret; restore an expired or inactive master key with
admin update-admin-api-key, as described in Command-line interface; or correct the configuration. A value infoundation4ai.secrets.env, including a new master key secret, changes through the change procedure in Secrets and keys, whose upgrade of the application release lets the master key Job store the new values. An image older than the database is replaced by the current version, as described in Backup, restore and upgrades. -
Actions to avoid. Generating a new application secret to clear a decryption error, which leaves the stored credentials of LLMs and embedding models unreadable. Changing the master key values in
foundation4ai.secrets.envwithout an upgrade of the application release. Deleting the bundled PostgreSQL volume claim, which creates an empty database with a new system ID.
Access
Login returning HTTP 401
-
Symptoms.
POST /loginreturns HTTP 401, and the dashboard sign-in fails. -
Diagnosis.
curl -sS -X POST "$FOUNDATION4_URL/login" \-H "x-api-key: $FOUNDATION4_API_KEY" \-H "x-api-key-secret: $FOUNDATION4_API_SECRET"kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \./foundation4ai database check-connectionThe
messageof the error body names the cause, as listed in Errors: a missing header, a key identifier that is not a UUID, orInvalid API Key or Secret. The connection check printsOk.when the configuration of the API server reaches PostgreSQL, andError: "Failed connecting to database: <reason>"otherwise. -
Cause.
Invalid API Key or Secretcovers an unknown key, a wrong secret, and an inactive or expired key. A database or cache failure while the API server checks the key returns HTTP 503Unable to validate the API key; try again laterinstead, and the API server logsCould not validate the API key. A master key that is expired or inactive returns HTTP 401 while the API server runs, and the API server log shows an error at startup that namesadmin update-admin-api-key --clear-expiration. A license problem does not produce HTTP 401: an invalid license stops the API server at startup, and an expired license returns HTTP 403Invalid license: Expiredon the operations that check the license. -
Resolution. Send the key identifier in
x-api-keyand the secret inx-api-key-secret, as described in Authenticate. The master key identifier and secret are the values ofFOUNDATION4AI_APP_MASTER_KEYandFOUNDATION4AI_APP_MASTER_SECRETinfoundation4ai.secrets.env. Restore an expired or inactive master key withadmin update-admin-api-key, as described in Command-line interface. -
Actions to avoid. Replacing the master key values in
foundation4ai.secrets.envwithout an upgrade of the application release: the master key Job stores new values only at an upgrade, and a restart before the upgrade stops the workers and can stop the API server.
Ingress configured but unreachable
-
Symptoms. Requests to the host name time out, or the ingress controller returns HTTP 404 or 503.
-
Diagnosis.
kubectl get ingressclasskubectl describe ingress foundation4ai -n foundation4aikubectl get endpoints foundation4ai-api-server foundation4ai-dashboard -n foundation4aikubectl get httproute foundation4ai -n foundation4ai -o yamlkubectl describe ingressshows the class, the host and the backends of the Ingress. An endpoints entry without addresses means that no ready pod serves the Service. The status of the HTTPRoute lists each parent with anAcceptedcondition;Falsewith a reason such asNoMatchingParentmeans that the parent reference matches no Gateway listener. -
Cause.
ingress.classNamenames a class that no controller serves. The request host differs fromingress.hosts, whose chart default ischart-example.local. No API server pod is ready. The defaulthttpRoute.parentRefs(Gatewaygateway, listenerhttp) match no Gateway. A network policy blocks the controller. -
Resolution. Set
classNameto an IngressClass from the list, set the host names iningress.hostsorhttpRoute.hostnamesand send requests to those names, and sethttpRoute.parentRefsto the Gateway and the listener. Allow the controller namespace in the network policies described in Security hardening. Check the API withGET /healthzorPOST /login; the Ingress and the HTTPRoute send/dashboardto the dashboard and every other path to the API. -
Actions to avoid. Changing the API server Service to
LoadBalancerorNodePortto bypass the controller, which also bypasses the TLS of the ingress.
Dashboard sign-in failure through a port forward
- Symptoms. A port forward to the dashboard Service shows the dashboard page, but the sign-in fails with a correct key and the message
Login failed: no API server answers at this address. Open the dashboard through the address (ingress or proxy) that also serves the API. - Diagnosis. The network panel of the browser's developer tools shows the sign-in request
POST /loginsent to the forwarded local address, where the dashboard server answers instead of the API server. - Cause. The dashboard is served under
/dashboard/and calls the API on the origin of the dashboard page, so the dashboard works only where one host name serves both/dashboardand the API. The chart's Ingress and HTTPRoute provide that host name; a port forward to one Service does not. - Resolution. Open the dashboard through the Ingress or the HTTPRoute host name, or through a local reverse proxy in front of port forwards to both Services, as described in API and dashboard access.
- Actions to avoid. Exposing the dashboard or the API server through a
LoadBalancerorNodePortService to replace the port forward, which bypasses the TLS of the ingress.
Ingestion
Documents remaining in pending or failed status
-
Symptoms. The pending count does not fall, documents remain
pendinglong after submission, or documents becomefailed. -
Diagnosis.
kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=500 --prefix \| grep -E "Error processing documents|Failed processing document|Document processing failed|Giving up|Undecodable"kubectl get pods -n foundation4ai -l app.kubernetes.io/name=api-server-workerkubectl logs -n foundation4ai deploy/foundation4ai-api-server-worker -c grpc --tail=100NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" \consumer info DOCUMENTS document-processorA line with
Failed processing documentnames one document, the pipeline and the reason, and a line withDocument processing failedgives the details of an internal error of one document. A line withError processing documents for pipelinenames a failure of a whole pipeline group, such as a database error; the documents of the group then store the reasoninternal error; see the worker logsfor an internal error. A line withGiving up on a document after repeated processing failuresmarks a document that becamefailed. A line withUndecodable queue message; terminating itmarks a job that no worker can read; the queue drops that job, and the document stayspending. The gRPC service log shows the errors of Python text splitters and embedding providers, and the consumer information shows the jobs that wait and the jobs delivered more than once. -
Cause. Processing fails for the document, for the text splitter or embedding model of the pipeline, or for the license, or no worker runs. A document that fails does not hold back the other documents of its batch. The worker retries a failing document after 10 seconds, then with a delay that doubles up to 10 minutes, and sets the status to
failedafter 12 attempts, about an hour after the first. The queue removes a job 24 hours after submission, so a document that waits that long, for example while no worker runs, stayspendingand is not processed again. -
Resolution. Remove the cause shown in the log: restore the gRPC service or the model server, install a valid license as described in the next entry, or start the workers. Documents that are still within their attempts are then processed without action. The client application submits again the
faileddocuments and the documents that arependingmore than 24 hours after creation, as described in Resubmission. -
Actions to avoid. Submitting documents again while the cause persists, which adds versions that fail in the same way. Purging or recreating the
DOCUMENTSstream, which removes the jobs of every queued document and leaves those documentspending.
Processing stopped after license expiry
-
Symptoms. The workers log
The license is no longer valid: the worker stops; queued documents wait for a renewed licenseand stop withError: "License check failed: Expired license", the restarted worker containers stop at startup with the same error, and the pending count stops falling. Operations that check the license return HTTP 403Invalid license: Expired, and a restarted API server container stops at startup withLicense check failed: Expired license. -
Diagnosis.
kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=200 --prefix \| grep -c "Expired license"LICENSE=$(grep '^FOUNDATION4AI_APP_LICENSE=' foundation4ai.secrets.env | cut -d= -f2-)kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \./foundation4ai license details "$LICENSE"A count above zero and the output
Error: "Expired"confirm the cause. Thelicense detailscommand needs a running API server pod. -
Cause. The workers check the license before each batch, and each worker stops when the license has expired. The documents that the workers have not taken stay in the queue. The processes read the license at startup, so a renewed license takes effect only after a restart, and a process that restarts with an expired license stops at startup. The queue removes a job 24 hours after submission, so a queued document is processed only when a renewed license is deployed within 24 hours of its submission.
-
Resolution. Obtain a renewed license, as described in Licensing, and apply the license with the change procedure in Secrets and keys, whose upgrade of the application release restarts the API server and the workers. The workers then process the queued documents. Then submit again the
faileddocuments and the documents that are stillpendingmore than 24 hours after creation, as described in Resubmission. -
Actions to avoid. Upgrading a release for other changes before the renewal, which stops at the license check. Purging the
DOCUMENTSstream.
Workers timing out on NATS JetStream
-
Symptoms. Worker containers log
Waiting for the database, cache or queue; retrying in <delay>for about four minutes, then restart withError: "Failed to connect to Foundation4.ai's services: Nats error: <reason>", where the reason reports a JetStream timeout or that JetStream is unavailable. The API server may report the same error afterFailed to initialize server:. A running API server returns submitted documents with the statusfailedand logsFailed queueing document for processing. -
Diagnosis.
kubectl get pods -n foundation4ai -l app.kubernetes.io/component=natskubectl logs -n foundation4ai foundation4ai-core-nats-0 -c nats --tail=50NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" account infokubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" stream info DOCUMENTSThe 3 NATS pods are expected
Runningwith every container ready.nats account infoshows whether JetStream is available to the account and how much storage is used, andnats stream infoshows the server that holds the stream. A NATS log that reports no JetStream leader, or storage limits reached, points to the NATS cluster. -
Cause. At startup, each worker and API server opens or creates the
documentsobject store and theDOCUMENTSstream. JetStream answers only while a majority of the 3 NATS servers runs, and refuses new data when the JetStream volume is full, at 10 GiB per server by default. A network policy that blocks port 4222 or 6222 has the same effect. -
Resolution. Restore the NATS pods, free or enlarge the JetStream storage described in Deployment overview and requirements, and allow ports 4222 and 6222 in the network policies. Then restart the workers:
kubectl rollout restart deployment/foundation4ai-api-server-worker -n foundation4aikubectl rollout status deployment/foundation4ai-api-server-worker -n foundation4aiExpected result:
successfully rolled out, and the worker log shows no further NATS error. -
Actions to avoid. Deleting the NATS JetStream volume claims or the
DOCUMENTSstream, which removes the queued jobs. Reducing NATS to one replica on an existing installation, which can remove the server that holds the stream.
Worker errors while reading the queue
-
Symptoms. The worker log shows
Error retrieving messages from message queue,Error receiving message from message queue,Undecodable queue message; terminating it,Processing result does not match any queued messageorProcessing a batch panicked, while the worker containers keep running. -
Diagnosis.
kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=500 --prefix \| grep -E "message queue|Undecodable|does not match any queued|panicked"A retrieval error that repeats every 2 seconds points to NATS JetStream. Each
Undecodable queue messageline also counts one job infoundation4ai_document_queue_failed. -
Cause. The worker logs queue errors and goes on. After a failed read from the queue, the worker waits 2 seconds and reads again. A job message that cannot be decoded is removed from the queue, because no worker can process the job, and the document of that job stays
pending. A batch that stops on an internal error is not acknowledged, and the queue delivers its jobs again after the acknowledgement timeout. -
Resolution. After repeated retrieval errors, check NATS JetStream as in the previous entry; the workers resume reading without a restart. When decoding errors or batch errors repeat, collect the worker logs and the output of
nats stream info DOCUMENTSandnats consumer info DOCUMENTS document-processor, and contact the Foundation4 provider. The client application submits again the documents that staypending, as described in Resubmission. -
Actions to avoid. Purging the
DOCUMENTSstream, which removes every queued job. Scaling the workers to 0 for more than 24 hours during the investigation, after which the queue has removed every waiting job.
Load
Workers restarted for exceeding memory
-
Symptoms. Worker pods restart during ingestion, and
kubectl describe podshowsLast State: TerminatedwithReason: OOMKilledand exit code 137, or the pod status isEvicted. -
Diagnosis.
kubectl get pods -n foundation4ai -l app.kubernetes.io/name=api-server-worker \-o jsonpath='{range .items[*]}{.metadata.name}{": "}{range .status.containerStatuses[*]}{.name}={.lastState.terminated.reason}{" "}{end}{"\n"}{end}'kubectl top pods -n foundation4ai --containersworker=OOMKilledorgrpc=OOMKillednames the container that exceeded the memory limit.kubectl top, which needs the metrics server, shows the memory of each container during a batch. -
Cause. Each worker takes up to 100 jobs per batch, and the memory of a batch grows with the size of the documents and with the embedding model. FastEmbed models run in the worker process, and Python embedding providers and text splitters run in the gRPC service. The charts set no resources.
api-server.worker.resources,api-server.grpc.resources,api-server.server.resourcesandapi-server.jobs.resourcesset the resources of each container, andapi-server.resourcesapplies to each container without a value of its own. A worker that stops during a batch leaves the jobs unacknowledged, the queue delivers the jobs again after the acknowledgement timeout, and the same batch can exceed the limit again. -
Resolution. Set requests and a memory limit for the container named in the diagnosis, then upgrade the application release, which restarts the pods with the new values. The values below are an example, not a sizing recommendation;
api-server.grpc.resourcestakes the same form for the gRPC service container, and Scaling and performance describes how to size the containers:api-server:worker:resources:requests:cpu: "1"memory: 2Gilimits:memory: 4Gi -
Actions to avoid. Removing the memory limit on shared nodes, which lets one batch take memory from other pods. Adding worker replicas to reduce memory use, which does not change the size of a batch.
Low ingestion throughput
-
Symptoms. The pending count grows during steady submission, and the message count of the
DOCUMENTSstream rises. -
Diagnosis. Read the stream state, then forward the metrics port of a worker in a separate terminal and read the worker metrics:
NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" stream info DOCUMENTSkubectl port-forward -n foundation4ai deploy/foundation4ai-api-server-worker 9090:9090curl -s http://localhost:9090/metrics | grep foundation4ai_document_queueThe counters
foundation4ai_document_queue_processed,_success,_skip,_removed,_retriesand_failedcount processing attempts and documents._processed_latencyis a histogram of the seconds per pipeline group, whose_sumdivided by_countgives the average time of a group. The gauges_text_splitter_latencyand_embedding_latencyhold the seconds that the last pipeline group spent in the text splitter and the embedding model. Two readings a few minutes apart give the documents per second of that worker. A rising_retriesor_failedcounter means that the backlog comes from failures, as described in the entry on pending and failed documents. The port forward reaches one worker pod, and Monitoring and logging describes the metrics. -
Cause. Each worker processes one batch at a time, and the pipeline groups of a batch one after another. Throughput is limited by the number of workers, by the CPU available to the worker and gRPC service containers, and by the time the embedding model needs per fragment.
-
Resolution. Add worker replicas with
api-server.replicaCount.worker, set CPU requests for the containers, or use an embedding model that runs on a dedicated model server, as described in Scaling and performance. -
Actions to avoid. Adding API server replicas, which do not process documents. Submitting more documents than the workers process in 24 hours, because the queue removes jobs after 24 hours.
HTTP 500 responses under load
-
Symptoms. Under load, requests return HTTP 500
Internal server errorafter about 10 seconds, and the API server log showsInternal server errorlines withkind="database"and an error that ends withFailed to acquire connection from pool: Connection pool timed out. A request whose key check waits for a connection returns HTTP 503Unable to validate the API key; try again laterinstead, withCould not validate the API keyin the log. -
Diagnosis.
psql "<administrator-connection-url>" -c "SHOW max_connections;"psql "<administrator-connection-url>" -c \"SELECT usename, state, count(*) FROM pg_stat_activity WHERE datname = '<database>' GROUP BY 1, 2;"kubectl get deployments,hpa -n foundation4aiThe connection counts per state show whether the API server pools are full, and the replica counts give the number of pools.
-
Cause. Each API server process holds at most
database.pool_sizeconnections, 3 in the chart defaults, and a request waits up to 10 seconds for a free connection. Bursts or long searches exhaust the pool. Each additional API server replica adds a pool, so replicas, including those that an autoscaler adds, can reachmax_connectionsof PostgreSQL, which then refuses new connections. Each worker holds 1 connection. -
Resolution. Raise the pool size in a
configsfile, keep the total belowmax_connectionsas described in Database, then upgrade the application release, which restarts the API server and the workers with the changed configuration:api-server:configs:f4ai-10-database-pool.yaml: |database:pool_size: 10helm upgrade foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mkubectl rollout status deployment/foundation4ai-api-server -n foundation4aiExpected result: Helm reports
STATUS: deployed, thendeployment "foundation4ai-api-server" successfully rolled out. -
Actions to avoid. Raising the API server replica count or the autoscaler maximum without a connection budget, which moves the failure to PostgreSQL.
Upgrades and changes
Pods using a previous value after a change
-
Symptoms. After a renewed license, a new password, a new URL or a change of
api-server.configs, the API server and the workers behave as before. -
Diagnosis.
kubectl get pods -n foundation4ai -l app.kubernetes.io/instance=foundation4aihelm history foundation4ai -n foundation4aiThe
AGEof the API server and worker pods is older than the change. -
Cause. Foundation4 reads configuration once at startup. The Deployments carry checksums of the configuration, of the
api-server.secretsentries and of the mounted keys offoundation4ai-secretsandfoundation4ai-core, so an upgrade of the application release that changes one of these values restarts the pods. A Secret changed without a later upgrade of the application release, such as a Secret applied after the last upgrade or a core release upgraded alone, does not restart the pods. An application release rendered without cluster access, such as through a GitOps tool, keeps the checksums of the Secrets constant. -
Resolution. Run the change procedure in Secrets and keys, which ends with the upgrade of the application release, or restart the pods:
kubectl rollout restart -n foundation4ai \deployment/foundation4ai-api-server deployment/foundation4ai-api-server-workerExpected result: both Deployments report
restarted, and the new pods show the changed behavior. -
Actions to avoid. Editing the Secrets
foundation4ai-coreorfoundation4ai-api-serverdirectly, which the next upgrade of the core or application release replaces.