Backup, restore and upgrades
This page describes what an operator backs up for a Foundation4 installation, how a backup is restored, and how Foundation4 is upgraded and rolled back. Operators and database administrators who plan recovery and maintenance use this page.
The commands on this page read the production values file foundation4ai.values.yaml, created in Install on Kubernetes, and read database connection URLs from shell variables: DATABASE_URL for the Foundation4 database and RESTORE_DATABASE_URL for a restore target. An evaluation installation uses evaluation.values.yaml and the kubectl exec forms of the database commands.
Backup set
The persistent state of an installation is the PostgreSQL database together with the secrets and settings that match the database. Every other component holds data that Foundation4 rebuilds or that expires.
| Item | Location | Backed up | Reason |
|---|---|---|---|
PostgreSQL database, schemas public and embeddings | PostgreSQL | Yes | Primary data store |
| Application secret | FOUNDATION4AI_APP_SECRET in the secrets file | Yes, with each database backup | Decrypts the LLM API keys and the embedding model credentials stored in the database |
| Master key identifier and secret | Secrets file | Yes, with each database backup | Keeps the master key of the backup; the master key Job replaces a stored master key secret that differs from the secrets file |
Secrets file and kustomization.yaml | Operator workstation | Yes, in the organization's secret store | Recreate foundation4ai-secrets |
| Values files | Operator workstation or version control | Yes | Recreate both releases |
| Chart version and image tags | Values files and Helm history | Recorded | Reinstall the same version |
| License | Secrets file | Recorded | Valid only for the system ID of the database |
| NATS JetStream volumes | Volume claims of the core release | No | Queued jobs and document text expire after 24 hours |
| Redis-compatible cache | Valkey memory | No | Rebuilt from the database |
| Prometheus data | Volume claim of the core release | Optional | Metrics history only |
Each database backup is paired with the secrets that were in effect when the backup was taken. The secrets are stored separately from the backup files, because the application secret protects the LLM API keys and the embedding model credentials inside the backup. Secrets and keys describes the application secret.
Database backup
A production database is backed up with the tools of the database service: physical backups, storage snapshots or point-in-time recovery, and logical dumps with pg_dump. The backup type decides the license after a restore, as Licensing describes:
- Physical backups and snapshots. A physical restore keeps the PostgreSQL system identifier and the catalog, and therefore keeps the system ID and the license.
- Logical dumps. A restore of a
pg_dumpfile into another cluster, or under another database name, produces a new system ID and needs a new license. A restore into a database of the same name on the same cluster keeps the system ID of a license of version 3. Licenses of versions 1 and 2 need a new license after every logical restore.
The following command writes a logical dump of a production database:
pg_dump --format=custom --no-comments \
--file="foundation4ai-$(date +%Y%m%d%H%M).dump" "$DATABASE_URL"
Expected result: the command prints nothing, exits with status 0, and the dump file exists. pg_restore --list <dump file> | grep "SCHEMA - embeddings" prints one line.
The following command writes a logical dump of the bundled database of an evaluation installation to the operator workstation:
kubectl exec -n foundation4ai foundation4ai-core-postgres-0 -c postgres -- \
pg_dump --format=custom --no-comments -U foundation4ai -d foundation4ai \
> foundation4ai-evaluation.dump
Expected result: the command prints nothing, and foundation4ai-evaluation.dump is larger than zero bytes.
pg_dump reads one consistent snapshot of the database. Documents that are pending when the dump starts remain pending after a restore, because the queued jobs are not part of the backup.
Logical restore
A logical restore replaces the database of an installation with a dump. The procedure applies to a restore into a new, empty database prepared as described in Database preparation. The secrets file holds the application secret and the master key of the installation that produced the dump.
-
Stop the API server and the workers:
kubectl scale -n foundation4ai deployment/foundation4ai-api-server \deployment/foundation4ai-api-server-worker --replicas=0Expected result:
deployment.apps/foundation4ai-api-server scaledanddeployment.apps/foundation4ai-api-server-worker scaled. -
Restore the dump and check the result:
pg_restore --no-owner --no-privileges --exit-on-error \--dbname="$RESTORE_DATABASE_URL" <dump file>psql "$RESTORE_DATABASE_URL" -c 'SELECT count(*) FROM pipelines'Expected result:
pg_restoreprints nothing, and the pipeline count equals the count of the source installation. -
Set
POSTGRES_URLinfoundation4ai.secrets.envto the restored database, apply the Secret and upgrade the core release:kubectl kustomize . | kubectl apply -n foundation4ai -f -helm upgrade foundation4ai-core ./charts/foundation4ai-core \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result:
secret/foundation4ai-secrets configured, and Helm reportsSTATUS: deployed. -
Clear the cache, which holds API key verification results and objects from before the restore:
kubectl rollout restart -n foundation4ai deployment/foundation4ai-core-rediskubectl rollout status -n foundation4ai deployment/foundation4ai-core-redisExpected result:
deployment "foundation4ai-core-redis" successfully rolled out. An external Redis-compatible cache is cleared with the tools of that service instead. -
Upgrade the application release, which checks the license against the restored database:
helm upgrade foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mkubectl logs -n foundation4ai job/foundation4ai-api-server-license-checkExpected result: when the restored database keeps the system ID, Helm reports
STATUS: deployed,kubectl logsreports that the Job is not found because Helm deleted the Job after the check succeeded, and the procedure continues with step 7. Otherwise Helm reports that the pre-upgrade hook failed, and the log showsSystemID: <system ID>andChecking for a valid license... Invalid. -
Obtain a license for the new system ID from the Foundation4 provider. Set the license in
foundation4ai.secrets.env, then apply the Secret and upgrade the application release:kubectl kustomize . | kubectl apply -n foundation4ai -f -helm upgrade foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result:
secret/foundation4ai-secrets configured, and Helm reportsSTATUS: deployed. -
Start the API server and the workers with the replica counts of the values file. The chart defaults are 1 and 3:
kubectl scale -n foundation4ai deployment/foundation4ai-api-server --replicas=1kubectl scale -n foundation4ai deployment/foundation4ai-api-server-worker --replicas=3kubectl rollout status -n foundation4ai deployment/foundation4ai-api-serverkubectl rollout status -n foundation4ai deployment/foundation4ai-api-server-workerExpected result: both Deployments report
successfully rolled out.
After the restore, the client application submits again the documents that remain pending or are failed, as described in Ingest documents reliably.
A restore into a new cluster installs both releases as described in Install on Kubernetes, with POSTGRES_URL pointing to the restored database and the secrets of the source installation.
Evaluation restore
The bundled database of the evaluation profile keeps the data in the volume claim postgres-data-foundation4ai-core-postgres-0, so a restore deletes the claim and restores the dump into a new, empty database:
-
Remove the application release, the bundled PostgreSQL StatefulSet and the volume claim:
helm uninstall foundation4ai -n foundation4aikubectl delete statefulset -n foundation4ai foundation4ai-core-postgreskubectl delete pvc -n foundation4ai postgres-data-foundation4ai-core-postgres-0 --ignore-not-foundExpected result:
release "foundation4ai" uninstalled,statefulset.apps "foundation4ai-core-postgres" deletedandpersistentvolumeclaim "postgres-data-foundation4ai-core-postgres-0" deleted. -
Recreate the bundled PostgreSQL with an upgrade of the core release:
helm upgrade foundation4ai-core ./charts/foundation4ai-core \-n foundation4ai -f evaluation.values.yaml --wait --timeout 10mkubectl wait -n foundation4ai --for=condition=Ready \pod/foundation4ai-core-postgres-0 --timeout=5mExpected result: Helm reports
STATUS: deployed, thenpod/foundation4ai-core-postgres-0 condition met. -
Restore the dump into the new database:
kubectl exec -i -n foundation4ai foundation4ai-core-postgres-0 -c postgres -- \pg_restore --no-owner --no-privileges --exit-on-error \-U foundation4ai -d foundation4ai < foundation4ai-evaluation.dumpExpected result: the command prints nothing and exits with status 0.
-
Install the application release and the license as in steps 4 to 6 of Install for evaluation. The restored database belongs to a new cluster and has a new system ID.
Expected result: in step 4, the license check log shows a system ID that differs from the system ID of the previous license, followed by
Invalid; after step 6, Helm reportsSTATUS: deployed.
The same procedure moves an evaluation installation whose bundled database has no volume claim, from an earlier chart version, to the current chart: the dump is taken with the earlier version, and steps 1 to 4 run with the current charts.
PostgreSQL major version upgrade
A major PostgreSQL version upgrade creates a new database cluster. Foundation4 moves to the new version with a logical dump and restore:
-
Prepare the database on the new PostgreSQL version as described in Database preparation, with the four extensions built for that version.
Expected result: the extension list of step 2 of the preparation names
pg_tokenizer,vchord,vchord_bm25andvector. -
Stop client writes and take a logical dump with the
pg_dumpof the new PostgreSQL version, as described in Database backup.Expected result: the dump file exists, and
pg_restore --list <dump file> | grep "SCHEMA - embeddings"prints one line. -
Restore the dump into the new database with the procedure of Logical restore.
Expected result: both Deployments report
successfully rolled outin step 7 of the logical restore.
The new cluster has a new PostgreSQL system identifier, so the installation has a new system ID after the restore and needs a new license, which step 6 of the logical restore installs. The maintenance window therefore includes the time to obtain the license.
Upgrade behavior
An upgrade installs new chart versions or image tags with helm upgrade, first for the core release and then for the application release. The application upgrade runs in the following order:
- Helm writes the ConfigMap and the Secret of the application release.
- The Job
foundation4ai-api-server-db-migrationapplies the new migrations. - The Job
foundation4ai-api-server-license-checkchecks the license. - The Job
foundation4ai-api-server-create-admin-api-keymakes the stored master key match the secrets file. - Helm updates the Deployments, and Kubernetes replaces the pods with a rolling update.
Helm deletes each Job after the Job succeeds. A failed Job remains until the next install or upgrade of the application release.
The following consequences apply:
- Old pods during migration. The pods of the previous version keep serving while the Jobs run, against a database that the migration is changing. The upgrade therefore runs in a maintenance window in which client applications do not write.
- Migration duration. A migration that rewrites fragment tables takes time in proportion to the number of fragments. The Helm timeout covers the migration, so large databases use a longer timeout than 10 minutes.
- Forward only. Migrations cannot be reverted, and the previous version does not start against a migrated database. Rollback describes the consequences.
- Extensions. The migrations need the extensions
vector,vchord,pg_tokenizerandvchord_bm25in the Foundation4 database. A database without them stops the migration Job, as Database describes. - Duplicate fragment rows. The migration that moves fragment text into separate tables keeps one row of each group of duplicate fragment rows, so duplicate rows do not stop the upgrade.
- Evaluation profile. The bundled database keeps the data in a volume claim across core release upgrades. A bundled database without a volume claim, from an earlier chart version, moves to the current chart with a dump and the procedure of Evaluation restore.
- Configuration changes. An upgrade that changes image tags, the configuration files or the values of the mounted Secrets restarts the pods, as described in Secrets and keys.
The Release notes list the changes of each version.
Pre-upgrade checks
-
Record the release history and the values of both releases:
helm history foundation4ai -n foundation4aihelm history foundation4ai-core -n foundation4aihelm get values foundation4ai -n foundation4ai > foundation4ai.previous-values.yamlExpected result: the last revision of each release has the status
deployed, and the values file is written. -
Back up the database and record the secrets, as described in Database backup.
Expected result: a dump file or a database service backup taken after client writes stopped.
-
List the applied migrations:
psql "$DATABASE_URL" -c 'SELECT version FROM schema_migrations ORDER BY version'Expected result: one row for each applied migration, starting with
m20250120_030022_initial_migration. -
Check that the database has the extensions that the migrations need:
psql "$DATABASE_URL" -c 'SELECT extname, extversion FROM pg_extension ORDER BY extname'Expected result: a list that names
pg_tokenizer,vchord,vchord_bm25andvector. A missing extension is created by a superuser, as described in Database preparation, before the upgrade. An evaluation installation runs the query withkubectl exec -n foundation4ai foundation4ai-core-postgres-0 -c postgres -- psql -U foundation4ai -d foundation4ai -c 'SELECT extname, extversion FROM pg_extension ORDER BY extname'.
Upgrade procedure
-
Set the image tags of the new release in
global.serverImageTag,global.grpcImageTagandglobal.dashboardImageTag, and any new values, infoundation4ai.values.yaml. The charts have no default image tags, so a render without the tags fails.Expected result:
grep ImageTag foundation4ai.values.yamlshows the new tags. -
Upgrade the core release:
helm upgrade foundation4ai-core ./charts/foundation4ai-core \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result: Helm reports
STATUS: deployed, andkubectl get pods -n foundation4aishows the NATS, Valkey and Prometheus podsRunning. -
Upgrade the application release, with a timeout that covers the migration:
helm upgrade foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 30mExpected result: Helm reports
STATUS: deployed. -
Verify the migration and the pods:
psql "$DATABASE_URL" -c 'SELECT version FROM schema_migrations ORDER BY version'kubectl get pods -n foundation4aiExpected result: the migration list contains the migrations of the new version, and the API server and worker pods show
2/2containers ready.POST /loginwith the master key returns status 201, as described in API and dashboard access. -
List the API keys whose pipeline grants name no classification. Searches follow the classification allow-list of the key, so such a key finds nothing until the key is granted the classifications that the key needs:
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \./foundation4ai admin keys-without-classificationsExpected result:
Every active API key that can read or search pipelines names classifications., or one line for each grant to correct, as described in Command-line interface.
An upgrade never uses helm uninstall followed by helm install. An uninstall keeps the database but loses the release history that a rollback needs.
Rollback
helm rollback restores the manifests of an earlier revision, such as the previous image tags. The charts define no rollback hooks, so a rollback runs no Job and reverts no migration. The safe rollback path depends on how far the upgrade progressed:
| Situation | Rollback path |
|---|---|
| The upgrade stopped before the migration Job applied a new migration, for example at an image pull error | helm rollback foundation4ai <revision> -n foundation4ai, or correct the cause and upgrade again |
| The migration Job failed | The failed migration left no changes. Correct the cause and upgrade again. When the same upgrade applied earlier migrations, the running pods keep serving, but a restarted pod of the previous version does not start. |
| The license check or the master key Job failed | The new migrations are applied. Correct the cause and upgrade again. The running pods keep serving, but a restarted pod of the previous version does not start. |
| The upgrade completed | Restore the pre-upgrade backup, then run helm rollback. A rollback without the restore leaves the API server and the workers unable to start. |
The migration list from the pre-upgrade checks shows which migrations the upgrade applied. A logical restore into another cluster or under another database name needs a new license, as described in Logical restore.
Troubleshooting
Upgrade failing on missing extensions
- Symptoms. The application upgrade stops at a failed pre-upgrade hook, and
kubectl logs -n foundation4ai job/foundation4ai-api-server-db-migrationends withThe database needs the vector, vchord, pg_tokenizer and vchord_bm25 extensions, which only a superuser can create. - Diagnosis. The extension check of the pre-upgrade checks names the extensions of the database.
- Cause. The database lacks an extension that a new migration creates, and the Foundation4 user cannot create the extension.
- Resolution. The failed migration left no changes. Create the extensions as described in Database, then upgrade again. The pods of the previous version keep running in the meantime.
- Actions to avoid. Restarting the API server or the workers of the previous version, which do not start when the upgrade applied earlier migrations.
Pods failing after a rollback with a missing migration
- Symptoms. After
helm rollback, theserverandworkercontainers restart repeatedly. - Diagnosis.
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c server --previousshows a message that containsthis migration has been applied but its file is missing. Current releases begin the message withThe database was migrated by a newer Foundation4 release. - Cause. The database schema belongs to the newer version. Migrations cannot be reverted, and the previous version refuses to start against migrations that the previous version does not know.
- Resolution. Return to the newer version with
helm rollback foundation4ai <newer revision> -n foundation4ai, or restore the pre-upgrade backup and then roll back. A logical restore into another cluster or under another database name needs a new license. - Actions to avoid. Deleting rows from
schema_migrations. The next upgrade then applies the migrations again to a schema that already contains the changes.
Restored installation stopped at the license check
- Symptoms. After a restore, the application upgrade fails at the license check hook, and the API server does not start.
- Diagnosis.
kubectl logs -n foundation4ai job/foundation4ai-api-server-license-checkshows a system ID that differs from the system ID of the license. - Cause. The restored database has a new system ID: the dump was restored into another cluster or under another database name, or the license has version 1 or 2.
- Resolution. Complete step 6 of Logical restore with a license for the new system ID.
- Actions to avoid. Restoring the dump again into another new database, which produces another system ID.