Skip to main content

Uninstall and cleanup

This page describes how to remove a Foundation4 installation, which resources remain after each step, and how to clean up after an interrupted installation. Operators who retire an installation, reinstall Foundation4 or recover from a failed installation use this page.

Removal order​

An installation is removed in the reverse order of installation:

  1. Application release. foundation4ai first, so that the API server and the workers stop before the services they use.
  2. Core release. foundation4ai-core, which removes NATS JetStream, Valkey, Prometheus and the bundled PostgreSQL.
  3. Remaining resources. The namespace, or the resources that remain in the namespace.
  4. External resources. The external database and the local files, when the installation is retired.

A production installation is backed up before removal, as described in Backup, restore and upgrades, unless the data is to be destroyed.

Resources after uninstall​

helm uninstall removes the resources that a release manages. Resources that the releases create as Helm hooks, resources that the operator creates, and the volumes of the NATS JetStream StatefulSet remain.

ResourceRemains after helm uninstallRemoved by namespace deletion
Deployments, Services, StatefulSets, Ingress or HTTPRoute of both releasesNoNot applicable
Prometheus volume claim foundation4ai-core-prometheus-serverNoNot applicable
Prometheus ClusterRole and ClusterRoleBinding foundation4ai-core-prometheus-serverNoNot applicable; cluster-scoped
Jobs foundation4ai-api-server-db-migration, foundation4ai-api-server-license-check and foundation4ai-api-server-create-admin-api-key, and the Job podsYesYes
Secrets foundation4ai-core, foundation4ai-core-redis and foundation4ai-api-serverYesYes
ConfigMap foundation4ai-api-serverYesYes
Secret foundation4ai-secrets and image pull secretsYesYes
NATS JetStream volume claims foundation4ai-core-nats-js-foundation4ai-core-nats-0, -1 and -2YesYes
Bundled PostgreSQL dataNo; the temporary pod volume is deleted with the podNot applicable
External database: schemas public and embeddings and every stored objectYesNo

The remaining Secrets hold every credential of the installation, including the master key secret and the application secret. The remaining NATS JetStream volumes hold queued document text for up to 24 hours after submission.

Complete removal​

The procedure removes every resource of an installation from the cluster. The commands match the removal of Install for evaluation.

  1. Uninstall the application release:

    helm uninstall foundation4ai -n foundation4ai

    Expected result: release "foundation4ai" uninstalled.

  2. Uninstall the core release:

    helm uninstall foundation4ai-core -n foundation4ai

    Expected result: release "foundation4ai-core" uninstalled.

  3. List the resources that remain:

    kubectl get jobs,secrets,configmaps,pvc -n foundation4ai

    Expected result: the 3 Jobs, the Secrets foundation4ai-secrets, foundation4ai-core, foundation4ai-core-redis and foundation4ai-api-server, any image pull secrets, the ConfigMap foundation4ai-api-server and the 3 NATS JetStream volume claims. Kubernetes also lists the ConfigMap kube-root-ca.crt, which every namespace contains.

  4. Delete the namespace:

    kubectl delete namespace foundation4ai

    Expected result: namespace "foundation4ai" deleted. The namespace deletion also removes the volume claims, the Secrets and the completed Jobs.

  5. Confirm that no cluster-scoped resources remain:

    kubectl get clusterrole,clusterrolebinding | grep foundation4ai

    Expected result: no output.

When the namespace is deleted without the two uninstall steps, the ClusterRole and the ClusterRoleBinding foundation4ai-core-prometheus-server remain. The following command removes both:

kubectl delete clusterrole,clusterrolebinding foundation4ai-core-prometheus-server

Expected result: clusterrole.rbac.authorization.k8s.io "foundation4ai-core-prometheus-server" deleted and the matching line for the ClusterRoleBinding.

Removal within a shared namespace​

When the namespace holds other workloads and stays in place, the remaining resources are deleted by name after the two uninstall steps:

kubectl delete job -n foundation4ai foundation4ai-api-server-db-migration \
foundation4ai-api-server-license-check foundation4ai-api-server-create-admin-api-key
kubectl delete secret -n foundation4ai foundation4ai-api-server foundation4ai-core \
foundation4ai-core-redis foundation4ai-secrets
kubectl delete configmap -n foundation4ai foundation4ai-api-server
kubectl delete pvc -n foundation4ai foundation4ai-core-nats-js-foundation4ai-core-nats-0 \
foundation4ai-core-nats-js-foundation4ai-core-nats-1 foundation4ai-core-nats-js-foundation4ai-core-nats-2

Expected result: one deleted line for each resource. kubectl delete secret reports NotFound for foundation4ai-core-redis when the bundled Valkey was disabled. Image pull secrets created for Foundation4 are deleted by name as well.

External resources​

An installation with an external database keeps every pipeline, document, API key and the migration history in that database after the uninstall. A database administrator removes the database and the user when the installation is retired:

psql "$ADMIN_DATABASE_URL" -c 'DROP DATABASE foundation4ai'
psql "$ADMIN_DATABASE_URL" -c 'DROP ROLE foundation4ai'

Expected result: DROP DATABASE, then DROP ROLE. PostgreSQL refuses to drop a database with open connections, so the Foundation4 pods are removed first.

The following local files hold credentials and are deleted, or moved to the organization's secret store, when the installation is retired: foundation4ai.secrets.env, any backup of the Secret, and database dumps with the secrets that belong to each dump.

Reinstallation​

A reinstallation against the same external database keeps every stored object and the system ID, so the existing license remains valid. The reinstallation uses the application secret and the master key of the previous installation: another application secret makes the stored LLM API keys unreadable, and another master key secret stops the API server at startup.

The following rules apply to a reinstallation:

  • NATS JetStream volumes. Volumes kept from the previous installation hold jobs for documents of the previous installation. The volumes are kept for a reinstallation against the same database, and deleted when the database is new.
  • Remaining Jobs. Helm replaces the Jobs of the previous installation when the new installation runs the hooks, so the Jobs need no removal. A license check Job that is still waiting is deleted first, as described in Interrupted installation.
  • Evaluation profile. A reinstallation of the core release creates a new, empty bundled database with a new system ID, which needs a new license.

Interrupted installation​

An installation that stops at the Helm timeout, or a Helm command that is interrupted, leaves a release in the status failed, pending-install or pending-upgrade, and can leave a Job running. The following procedure brings the namespace back to a state from which the installation starts again.

  1. Show the status of both releases and the Jobs:

    helm list -n foundation4ai --all
    kubectl get jobs -n foundation4ai

    Expected result: the STATUS column of each release, and the completions of each Job.

  2. Delete a license check Job that is still running, which waits for 365 days after a failed check:

    kubectl delete job -n foundation4ai foundation4ai-api-server-license-check

    Expected result: job.batch "foundation4ai-api-server-license-check" deleted, or NotFound when the Job does not exist.

  3. Remove a release with the status failed or pending-install that never reached deployed:

    helm uninstall foundation4ai -n foundation4ai

    Expected result: release "foundation4ai" uninstalled. The same command with foundation4ai-core removes an interrupted core release.

  4. Return a release with the status pending-upgrade, or failed after an upgrade, to the last deployed revision:

    helm history foundation4ai -n foundation4ai
    helm rollback foundation4ai <last deployed revision> -n foundation4ai

    Expected result: Rollback was a success! Happy Helming!. When the interrupted upgrade applied new migrations, the rollback leaves the pods unable to start, as described in Backup, restore and upgrades; the upgrade is completed instead.

  5. Correct the cause of the interruption and repeat the installation step that failed.

    Expected result: Helm reports STATUS: deployed for the repeated step.

Troubleshooting​

Namespace remaining in terminating status​

  • Symptoms. kubectl delete namespace foundation4ai does not return, and kubectl get namespace foundation4ai shows the status Terminating.
  • Diagnosis. kubectl get namespace foundation4ai -o jsonpath='{.status.conditions}' names the resources that block the deletion. kubectl get pvc,pods -n foundation4ai shows the remaining volume claims and pods.
  • Cause. Resources with finalizers wait for a controller, for example volume claims that wait for the storage provisioner to release the volumes.
  • Resolution. Wait for the pods to stop and for the storage provisioner to delete the volumes. When the provisioner is unavailable, restore the provisioner first.
  • Actions to avoid. Removing finalizers by hand, which can leave volumes with document text in the storage backend.

Release reinstall blocked by another operation​

  • Symptoms. helm install or helm upgrade fails with a message that another operation is in progress.
  • Diagnosis. helm list -n foundation4ai --all shows the release in pending-install or pending-upgrade.
  • Cause. An earlier Helm command was interrupted before the command recorded the result.
  • Resolution. Follow the steps of Interrupted installation.
  • Actions to avoid. Deleting the Helm release Secrets in the namespace by hand, which removes the release history that a rollback needs.