Secrets and keys

This page lists the Kubernetes Secrets of a Foundation4 installation, the format of every key, and the steps that bring a changed value into the running pods. The page also describes what the application secret and the master key protect. Operators who install, back up or change credentials use this page, and security reviewers use the Secret inventory.

The commands on this page read the production values file foundation4ai.values.yaml, created in Install on Kubernetes. An evaluation installation uses evaluation.values.yaml instead, as in Install for evaluation.

Secret chain

The operator creates one Secret. The two Helm releases copy values from that Secret into Secrets of their own each time a release is installed or upgraded:

1. kubectl kustomize . | kubectl apply -f - builds the Secret foundation4ai-secrets from the file foundation4ai.secrets.env, as kustomization.yaml defines.
2. The core release foundation4ai-core reads foundation4ai-secrets and writes the Secret foundation4ai-core, with the connection URLs and the four application values, and the Secret foundation4ai-core-redis, with the Valkey password.
3. The application release foundation4ai reads foundation4ai-core and writes the Secret foundation4ai-api-server, which holds the configuration files that the API server, the workers and the installation Jobs load at startup.

A changed value in foundation4ai.secrets.env therefore reaches the pods only after the Secret is applied, both releases are upgraded in this order, and the Deployments are restarted. Changes and restarts gives the procedure.

The releases write foundation4ai-core, foundation4ai-core-redis and foundation4ai-api-server as Helm hooks: Helm replaces the three Secrets before each install and upgrade, and helm uninstall leaves the Secrets in place. Uninstall and cleanup lists the resources that remain after an uninstall.

Secret inventory

Secret	Written by	Keys	Read by
foundation4ai-secrets	Operator, from foundation4ai.secrets.env	The keys of the secrets file	Core release templates; the bundled PostgreSQL pod (database name, user and passwords)
foundation4ai-core	Core release	FOUNDATION4AI_DATABASE_URL, FOUNDATION4AI_REDIS_URL, FOUNDATION4AI_NATS_URL, FOUNDATION4AI_PROMETHEUS_URL, FOUNDATION4AI_APP_LICENSE, FOUNDATION4AI_APP_SECRET, FOUNDATION4AI_APP_MASTER_KEY, FOUNDATION4AI_APP_MASTER_SECRET	Application release templates
foundation4ai-core-redis	Core release, when the bundled Valkey is enabled	default: the Valkey password	Bundled Valkey
foundation4ai-api-server	Application release	f4ai-zz-database-credentials.yaml, f4ai-zz-nats-credentials.yaml, f4ai-zz-redis-credentials.yaml, f4ai-zz-prometheus-credentials.yaml, f4ai-zz-app-credentials.yaml, and one key for each api-server.secrets entry	API server, worker and installation Job pods, as configuration files in /app/config
Image pull secrets	Operator	Registry credentials	Pods of the application release, through global.imagePullSecrets, and pods of the core release, through the keys that Charts, images and installation bundle lists

The following rules apply to the inventory:

• Two naming schemes. The keys with single underscores, such as FOUNDATION4AI_APP_MASTER_KEY, are read only by the chart templates. The Foundation4 process reads configuration files and environment variables with the prefix FOUNDATION4AI__ and double underscores. Configuration reference describes the configuration layers.
• Fixed names. The application release reads the Secret foundation4ai-core by name, so the core release is named foundation4ai-core. The core values global.secrets.*Key rename the keys that the core release reads; the pages of this section use the default names.
• Additional configuration. api-server.secrets adds configuration files to foundation4ai-api-server for settings that are confidential. Non-confidential settings, such as a database certificate authority, go in api-server.configs.
• Prometheus URL. The Foundation4 process does not read a Prometheus URL, so FOUNDATION4AI_PROMETHEUS_URL has no effect.

Charts, images and installation bundle describes image pull secrets.

Secrets file keys

The template foundation4ai.secrets.env.template contains the following keys. Two further keys are read by the core release but are absent from the template.

Key	Read when	Value
POSTGRES_DATABASE	Bundled PostgreSQL (postgres.enabled: true)	Database name, foundation4ai
POSTGRES_USER	Bundled PostgreSQL	Database user, foundation4ai
POSTGRES_SUPERUSER_PASSWORD	Bundled PostgreSQL	Hexadecimal password
POSTGRES_PASSWORD	Bundled PostgreSQL	Hexadecimal password
POSTGRES_URL	External PostgreSQL (postgres.enabled: false, the default)	Connection URL, as described in Database
REDIS_PASSWORD	Bundled Valkey (redis.enabled: true, the default)	Hexadecimal password
REDIS_URL	External Redis-compatible cache (redis.enabled: false)	redis://<user>:<password>@<host>:<port>
FOUNDATION4AI_APP_SECRET	Always	Application secret: 32 random bytes in URL-safe base64 encoding
FOUNDATION4AI_APP_MASTER_KEY	Always	Master key identifier: a universally unique identifier (UUID)
FOUNDATION4AI_APP_MASTER_SECRET	Always	Master key secret: a hexadecimal value
FOUNDATION4AI_APP_LICENSE	Always	The license from the Foundation4 provider, or the placeholder pending before the license exists
NATS_URL (not in the template)	External NATS JetStream (nats.enabled: false)	nats://<host>:4222
PROMETHEUS_URL (not in the template)	Bundled Prometheus disabled (prometheus.enabled: false)	Any URL; the value has no effect

The template contains sample passwords. Every password in foundation4ai.secrets.env is replaced with a generated value before the first installation.

Value generation

The following commands generate the values. Run each command once for each key that needs a value:

openssl rand -hex 24
uuidgen
openssl rand -base64 32 | tr '+/' '-_'

Expected result: openssl rand -hex 24 prints 48 hexadecimal characters, for a password or the master key secret. uuidgen prints a UUID such as 3f0c9b6e-2d4a-4c1e-9b7f-5a8d2e6c1f04, for the master key identifier. The last command prints 44 characters ending in =, for the application secret.

The formats follow from how the charts and Foundation4 read the values:

• Hexadecimal passwords. The core release inserts passwords into connection URLs without URL encoding. A password that contains @, /, #, ? or % breaks the URL, and a malformed cache URL stops the process at startup.
• Unquoted values. The application release writes the license, the application secret and both master key values into a YAML configuration file without quotes. A value that YAML reads as a number, a boolean or a null changes type. Generated hexadecimal, UUID and base64 values avoid this problem.
• Application secret. Foundation4 decodes the application secret as a Fernet key, which is exactly 32 bytes in URL-safe base64 encoding. Any other value stops the API server and the workers with Error decoding secret key.
• Master key identifier. The identifier must be a UUID. Any other value fails configuration parsing in every installation Job and pod with Failed to parse config.
• License placeholder. The license key holds a non-empty placeholder, such as pending, until the license exists. An empty value may fail configuration parsing instead of reporting a missing license.

Application secret

The application secret encrypts the data in the following table with Fernet, an authenticated symmetric encryption format.

Data	Location	Effect of a changed or lost application secret
API keys of LLMs	PostgreSQL	Stored keys cannot be decrypted. Requests that call such an LLM fail with HTTP 400, with details.error set to API key is required for OpenAI provider, until the API key of each LLM is set again.
API key verification results	Redis-compatible cache, 15 minutes	Entries that cannot be decrypted are discarded and computed again. No action is needed.
Agent execution traces	Redis-compatible cache, 60 minutes	Traces written before the change cannot be read.

The application secret does not protect the following data:

• Fragment text. Fragment text is encrypted with ChaCha20-Poly1305, using a key for each classification that is stored in the same database. Security at a glance describes the storage.
• API key secrets. Foundation4 stores an Argon2 hash of each API key secret, which needs no key.

A database restore needs the application secret that was in effect when the backup was taken, because the LLM API keys in the backup were encrypted with that secret. The application secret is therefore backed up together with each database backup and stored separately from the backup files. Backup, restore and upgrades describes the backup set. LLMs describes how LLM API keys are set again.

Master key

The master key is the administration key of the installation. The installation Job foundation4ai-api-server-create-admin-api-key creates the master key, named Master Admin Key, from FOUNDATION4AI_APP_MASTER_KEY and FOUNDATION4AI_APP_MASTER_SECRET at the first installation. Later runs of the Job leave the stored secret of an existing master key unchanged.

At every startup, the API server and each worker authenticate with the master key before serving requests or processing documents. The master key must exist, be active and unexpired, and the secret must match the stored hash. Otherwise the process exits with one of the following errors:

• API server. Failed to initialize server: Failed to initialize Foundation4aiCore with master key.
• Worker. A message that begins with Failed to initialize Foundation4.ai:.

The master key identifier and secret are fixed after the first installation. The operator records both values in the organization's secret store. Access control describes the permissions of the master key, and Authenticate reads the values back from the Secret.

Changes and restarts

Foundation4 reads configuration once, at process startup, and the Deployments carry no checksum of the Secrets. A changed Secret therefore takes effect only when the pods restart.

Change	Steps
License	The procedure below; Licensing adds a check of the new license first
Application secret	The procedure below, then set the API key of each LLM again
External PostgreSQL password or URL	Change the password in PostgreSQL, then the procedure below. Database connections fail from the password change until the restart completes.
External cache URL	The procedure below
Bundled Valkey password	The procedure below, with a restart of the Deployment foundation4ai-core-redis after step 2
Bundled PostgreSQL passwords	Not supported. The bundled chart sets the passwords only when the database is initialized.
Master key identifier or secret	Not supported after the first installation
api-server.configs or api-server.secrets entries	Steps 3 and 4 of the procedure below

The procedure applies a changed value from foundation4ai.secrets.env:

1. Edit foundation4ai.secrets.env and apply the Secret:

   kubectl kustomize . | kubectl apply -f -

   Expected result: secret/foundation4ai-secrets configured.

2. Upgrade the core release, which copies the values into foundation4ai-core:

   helm upgrade foundation4ai-core ./charts/foundation4ai-core \
     -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

   Expected result: Helm reports STATUS: deployed.

3. Upgrade the application release, which copies the values into foundation4ai-api-server:

   helm upgrade foundation4ai ./charts/foundation4ai \
     -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
   kubectl get jobs -n foundation4ai

   Expected result: Helm reports STATUS: deployed, and the Jobs foundation4ai-api-server-db-migration, foundation4ai-api-server-license-check and foundation4ai-api-server-create-admin-api-key show 1/1 completions.

4. Restart the API server and the workers:

   kubectl rollout restart -n foundation4ai \
     deployment/foundation4ai-api-server deployment/foundation4ai-api-server-worker
   kubectl rollout status -n foundation4ai deployment/foundation4ai-api-server
   kubectl rollout status -n foundation4ai deployment/foundation4ai-api-server-worker

   Expected result: both Deployments report restarted, then successfully rolled out.

Credential handling

Credentials can leak through operator actions, so the following rules apply:

• Secrets file. The deployment files exclude *.env files from version control. The operator keeps foundation4ai.secrets.env in the organization's secret store, not in a shared directory or a repository.
• Command output. kubectl get secret -o yaml, helm get hooks and helm get all print credentials in base64 encoding, which is an encoding and not encryption. Output of these commands is not pasted into tickets, chat messages or support requests.
• Scripts. Installation scripts do not trace commands with set -x while commands carry credentials, and do not echo the secrets file.
• Support requests. A support request to the Foundation4 provider carries the system ID, Job and pod logs and Helm status output, never the secrets file, a password, the application secret or the master key secret.
• Access. Read access to Secrets in the foundation4ai namespace is limited to the operators of the installation. Security hardening describes access restrictions.

Rendering and namespace limits

Both charts read Secrets from the cluster while Helm renders the templates, through the Helm lookup function. The following limits follow:

• Offline rendering. helm template, and tools that render charts with helm template, such as Argo CD, receive no data from lookup. The render then fails, or produces Secrets without credentials. Both releases are therefore installed and upgraded with helm install or helm upgrade against the cluster.
• Namespace. kustomization.yaml sets namespace: foundation4ai for foundation4ai-secrets, and each release reads Secrets from the release namespace. Both releases therefore run in foundation4ai, as Install on Kubernetes describes.

Troubleshooting

Troubleshooting describes two further problems with Secrets: pods using a previous value after a change and credentials missing after installation through a GitOps tool.

Installation Jobs failing with a configuration error

• Symptoms. The Job foundation4ai-api-server-db-migration fails, and Helm reports that the pre-install or pre-upgrade hook failed.
• Diagnosis. kubectl logs -n foundation4ai job/foundation4ai-api-server-db-migration shows Error: "Failed to parse config: <reason>". kubectl get secret foundation4ai-secrets -n foundation4ai -o jsonpath='{.data.FOUNDATION4AI_APP_MASTER_KEY}' | base64 -d shows whether the master key identifier is a UUID.
• Cause. A value in foundation4ai.secrets.env cannot be parsed: the master key identifier is not a UUID, the license is empty, or a value reads as a YAML number, boolean or null.
• Resolution. Correct the value, apply the Secret, upgrade the core release, and repeat the application install or upgrade.
• Actions to avoid. Editing the Secret foundation4ai-api-server directly. The next upgrade replaces the Secret with values from foundation4ai-core.

API server and workers failing with a secret key error

• Symptoms. The server and worker containers restart repeatedly after an installation or a change of the application secret.
• Diagnosis. kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c server --previous shows Failed to create database connection: Decryption error: Error decoding secret key.
• Cause. FOUNDATION4AI_APP_SECRET is not 32 bytes in URL-safe base64 encoding, for example because the value was generated with a different length or copied incompletely.
• Resolution. Restore the previous application secret from the secret store and apply the change procedure. For a first installation, generate a new value with openssl rand -base64 32 | tr '+/' '-_'.
• Actions to avoid. Generating a new application secret for an installation that holds LLM API keys, which makes the stored keys unreadable.