Secrets and keys
This page lists the Kubernetes Secrets of a Foundation4 installation, the format of every key, and the steps that bring a changed value into the running pods. The page also describes what the application secret and the master key protect. Operators who install, back up or change credentials use this page, and security reviewers use the Secret inventory.
The commands on this page read the production values file foundation4ai.values.yaml, created in Install on Kubernetes. An evaluation installation uses evaluation.values.yaml instead, as in Install for evaluation.
Secret chain
The operator creates one Secret. The core release builds the connection URLs from that Secret, and the pods of the application release read the credentials as files:
kubectl kustomize . | kubectl apply -n foundation4ai -f -builds the Secretfoundation4ai-secretsfrom the filefoundation4ai.secrets.env, askustomization.yamldefines.- The core release
foundation4ai-corereadsfoundation4ai-secretsand writes the Secretfoundation4ai-core, with the four connection URLs, the Secretfoundation4ai-core-redis, with the Valkey password, and the Secretfoundation4ai-core-nats-auth, with the NATS JetStream client password. - The application release
foundation4aimounts the license, the application secret and the master key values fromfoundation4ai-secrets, and the connection URLs fromfoundation4ai-core, as files under/app/config/secretsin the API server, worker and installation Job pods. The application release copies no credential into a Secret of the application release.
A changed value in foundation4ai.secrets.env therefore reaches the pods after the Secret is applied and the releases are upgraded: the core release for a connection value, then the application release. The upgrade of the application release restarts the API server and the workers when a mounted value has changed. Changes and restarts gives the procedure.
The releases write foundation4ai-core, foundation4ai-core-redis and foundation4ai-api-server as Helm hooks: Helm replaces the three Secrets before each install and upgrade, and helm uninstall leaves the Secrets in place. foundation4ai-core-nats-auth is a regular resource of the core release, which an upgrade updates in place and helm uninstall removes. Uninstall and cleanup lists the resources that remain after an uninstall.
Secret inventory
| Secret | Written by | Keys | Read by |
|---|---|---|---|
foundation4ai-secrets | Operator, from foundation4ai.secrets.env | The keys of the secrets file | Core release templates; the bundled PostgreSQL pod (database name, user and passwords); API server, worker and installation Job pods (the four FOUNDATION4AI_APP_* keys, as files) |
foundation4ai-core | Core release | FOUNDATION4AI_DATABASE_URL, FOUNDATION4AI_REDIS_URL, FOUNDATION4AI_NATS_URL, FOUNDATION4AI_PROMETHEUS_URL | API server, worker and installation Job pods, as files |
foundation4ai-core-redis | Core release, when the bundled Valkey is enabled | default: the Valkey password | Bundled Valkey |
foundation4ai-core-nats-auth | Core release, when the bundled NATS JetStream is enabled | auth.conf: the client user and password | Bundled NATS JetStream |
foundation4ai-api-server | Application release | One key for each api-server.secrets entry | API server, worker and installation Job pods, as configuration files in /app/config |
| Image pull secrets | Operator | Registry credentials | Pods of the application release, through global.imagePullSecrets, and pods of the core release, through the keys that Charts, images and installation bundle lists |
The pods of the application release read the following files under /app/config/secrets. The name of each file is the configuration setting that the file sets.
| File | Secret | Key |
|---|---|---|
app.license | foundation4ai-secrets | FOUNDATION4AI_APP_LICENSE |
app.secret | foundation4ai-secrets | FOUNDATION4AI_APP_SECRET |
app.master.key | foundation4ai-secrets | FOUNDATION4AI_APP_MASTER_KEY |
app.master.secret | foundation4ai-secrets | FOUNDATION4AI_APP_MASTER_SECRET |
database.url | foundation4ai-core | FOUNDATION4AI_DATABASE_URL |
redis.url | foundation4ai-core | FOUNDATION4AI_REDIS_URL |
nats.url | foundation4ai-core | FOUNDATION4AI_NATS_URL |
prometheus.url | foundation4ai-core | FOUNDATION4AI_PROMETHEUS_URL |
The following rules apply to the inventory:
- Two naming schemes. The keys with single underscores, such as
FOUNDATION4AI_APP_MASTER_KEY, are read only by the charts. The Foundation4 process reads configuration files, the files under/app/config/secretsas plain strings, and environment variables with the prefixFOUNDATION4AI__and double underscores. Configuration reference describes the configuration layers. - Fixed names. The application release reads the operator Secret named in
global.secrets.existingSecretand the core Secret named inapi-server.secret, by defaultfoundation4ai-secretsandfoundation4ai-core. The application release reads the fourFOUNDATION4AI_APP_*keys under these names. The core valuesglobal.secrets.*Keyrename the keys that the core release reads; the pages of this section use the default names. - Required keys. The pods of the application release mount the four
FOUNDATION4AI_APP_*keys, so the four keys are present infoundation4ai-secretseven before the license exists. - Additional configuration.
api-server.secretsadds configuration files tofoundation4ai-api-serverfor settings that are confidential. Non-confidential settings, such as a database certificate authority, go inapi-server.configs. - Prometheus URL. The API server queries Prometheus at this URL for the
/statisticsendpoint that Monitoring and logging describes. With the bundled Prometheus, the core release writes the URL of the bundled server. Otherwise the core release copiesPROMETHEUS_URL, and an empty value selectshttp://prometheus:9090.
Charts, images and installation bundle describes image pull secrets.
Secrets file keys
The template foundation4ai.secrets.env.template contains the following keys. PROMETHEUS_URL is commented out in the template.
| Key | Read when | Value |
|---|---|---|
POSTGRES_DATABASE | Bundled PostgreSQL (postgres.enabled: true) | Database name, foundation4ai |
POSTGRES_USER | Bundled PostgreSQL | Database user, foundation4ai |
POSTGRES_SUPERUSER_PASSWORD | Bundled PostgreSQL | Hexadecimal password |
POSTGRES_PASSWORD | Bundled PostgreSQL | Hexadecimal password |
POSTGRES_URL | External PostgreSQL (postgres.enabled: false, the default) | Connection URL, as described in Database |
REDIS_PASSWORD | Bundled Valkey (redis.enabled: true, the default) | Hexadecimal password |
REDIS_URL | External Redis-compatible cache (redis.enabled: false) | redis://<user>:<password>@<host>:<port>, or rediss:// for a cache that requires TLS |
NATS_PASSWORD | Bundled NATS JetStream (nats.enabled: true, the default) | Hexadecimal password of the client user foundation4ai |
NATS_URL | External NATS JetStream (nats.enabled: false) | nats://<host>:4222 |
PROMETHEUS_URL | Bundled Prometheus disabled (prometheus.enabled: false) | Prometheus base URL; optional |
FOUNDATION4AI_APP_SECRET | Always | Application secret: 32 random bytes in URL-safe base64 encoding |
FOUNDATION4AI_APP_MASTER_KEY | Always | Master key identifier: a universally unique identifier (UUID) |
FOUNDATION4AI_APP_MASTER_SECRET | Always | Master key secret: a hexadecimal value |
FOUNDATION4AI_APP_LICENSE | Always | The license from the Foundation4 provider, or a placeholder such as pending before the license exists |
The template contains sample passwords. Every password in foundation4ai.secrets.env is replaced with a generated value before the first installation.
Value generation
The following commands generate the values. Run each command once for each key that needs a value:
openssl rand -hex 24
uuidgen
openssl rand -base64 32 | tr '+/' '-_'
Expected result: openssl rand -hex 24 prints 48 hexadecimal characters, for a password or the master key secret. uuidgen prints a UUID such as 3f0c9b6e-2d4a-4c1e-9b7f-5a8d2e6c1f04, for the master key identifier. The last command prints 44 characters ending in =, for the application secret.
The formats follow from how the charts and Foundation4 read the values:
- Passwords. A password may contain any character, because the core release percent-encodes the passwords of the bundled services in the URLs that the core release builds. A URL in the secrets file, such as
POSTGRES_URL, is copied unchanged, so a password in such a URL is percent-encoded by the operator. A malformed cache URL stops the process at startup. Hexadecimal passwords need no encoding. - Application secret. Foundation4 decodes the application secret as a Fernet key, which is exactly 32 bytes in URL-safe base64 encoding. Any other value stops the API server and the workers with
Error decoding secret key. - Master key identifier. The identifier must be a UUID. Any other value fails configuration parsing in every installation Job and pod with
Failed to parse config. - License. An empty license value is reported as a missing license, and any other value that is not a license, such as
pending, as an invalid license. Licensing describes the license checks.
Application secret
The application secret encrypts the data in the following table with Fernet, an authenticated symmetric encryption format.
| Data | Location | Effect of a changed or lost application secret |
|---|---|---|
| API keys of LLMs | PostgreSQL | Stored keys cannot be decrypted, and Foundation4 calls such an LLM without an API key. A model server that requires a key refuses the call: agent executions and LLM queries return HTTP 502 with Upstream LLM error, and the OpenAI-compatible pass-through returns the response of the model server. The API key of each LLM is then set again. |
Credential parameters of embedding models, such as api_key | PostgreSQL | Stored credentials cannot be decrypted. Searches and document processing that use such an embedding model fail until the previous application secret is restored, because the parameters of an embedding model cannot be changed. |
| API key verification results | Redis-compatible cache, 15 minutes | Entries that cannot be decrypted are discarded and computed again. No action is needed. |
| Agent execution traces | Redis-compatible cache, 60 minutes | Traces written before the change cannot be read. |
The application secret does not protect the following data:
- Fragment text. Fragment text is encrypted with ChaCha20-Poly1305, using a key for each classification that is stored in the same database. Security at a glance describes the storage.
- API key secrets. Foundation4 stores an Argon2 hash of each API key secret, which needs no key.
A database restore needs the application secret that was in effect when the backup was taken, because the LLM API keys and the embedding model credentials in the backup were encrypted with that secret. The application secret is therefore backed up together with each database backup and stored separately from the backup files. Backup, restore and upgrades describes the backup set. LLMs describes how LLM API keys are set again.
Master key
The master key is the administration key of the installation. At every installation and upgrade, the installation Job foundation4ai-api-server-create-admin-api-key makes the stored master key match FOUNDATION4AI_APP_MASTER_KEY and FOUNDATION4AI_APP_MASTER_SECRET:
- New identifier. When no key has the identifier, the Job creates the master key, named
Master Admin Key. A previous master key with that name is renamedMaster Admin Key (replaced by <new identifier>)and keeps the administration permissions until an administrator deletes the previous key. - Changed secret. When the stored secret does not match, the Job replaces the stored secret. The previous secret stops authenticating at once.
- Active flag and expiry. The Job leaves the active flag and the expiry of an existing master key unchanged.
At startup, the API server and each worker initialize with the master key. A failure has the following effects:
- API server. A master key secret that does not match the stored key stops the API server with
Failed to initialize server: Failed to initialize with the master key: <reason>. A master key that is expired, inactive or missing does not stop the API server: the API server logsThe master key is expired, inactive or missing, so requests made with it are refused, followed by the command that restores the key, and serves requests made with other keys. - Worker. Any master key failure stops the worker with a message that begins with
Failed to initialize Foundation4.ai:, such asFailed to initialize Foundation4.ai: Invalid or missing key.
An expired or deactivated master key is restored inside the API server container, and the workers are then restarted:
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \
./foundation4ai admin update-admin-api-key --active true --clear-expiration
kubectl rollout restart -n foundation4ai deployment/foundation4ai-api-server-worker
kubectl rollout status -n foundation4ai deployment/foundation4ai-api-server-worker
Expected result: Master key updated successfully., then successfully rolled out for the workers. A deleted master key is created again by the master key Job at the next upgrade of the application release. Command-line interface lists the options of the command.
The operator records the master key identifier and secret in the organization's secret store. Access control describes the permissions and the rotation of the master key, and Authenticate reads the values back from the Secret.
Changes and restarts
Foundation4 reads configuration once, at process startup. The Deployments of the API server and the workers carry checksums of the configuration and of the mounted keys of foundation4ai-secrets and foundation4ai-core, so an upgrade of the application release that changes one of these values restarts the API server and the workers.
| Change | Steps |
|---|---|
| License | Steps 1 and 3 of the procedure below; Licensing adds a check of the new license first |
| Application secret | Steps 1 and 3 of the procedure below, then set the API key of each LLM again. Embedding models with credential parameters stop working, as Application secret describes. |
| Master key secret | Steps 1 and 3 of the procedure below. The previous secret stops authenticating when the master key Job runs. |
| Master key identifier | Steps 1 and 3 of the procedure below, then delete the previous master key once no client uses the previous key, as Master key describes |
| External PostgreSQL password or URL | Change the password in PostgreSQL, then the procedure below. Database connections fail from the password change until the restart completes. |
| External cache or NATS JetStream URL | The procedure below |
| Bundled NATS JetStream password | The procedure below. The bundled NATS JetStream applies the new password in step 2 without a restart. |
| Bundled Valkey password | The procedure below, with a restart of the Deployment foundation4ai-core-redis after step 2 |
| Bundled PostgreSQL passwords | Not supported. The bundled chart sets the passwords only when the database is initialized. |
api-server.configs or api-server.secrets entries | Step 3 of the procedure below |
The procedure applies a changed value from foundation4ai.secrets.env:
-
Edit
foundation4ai.secrets.envand apply the Secret:kubectl kustomize . | kubectl apply -n foundation4ai -f -Expected result:
secret/foundation4ai-secrets configured. -
Upgrade the core release, which writes the connection URLs into
foundation4ai-core:helm upgrade foundation4ai-core ./charts/foundation4ai-core \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result: Helm reports
STATUS: deployed. -
Upgrade the application release, which runs the installation Jobs and restarts the API server and the workers when a mounted value has changed:
helm upgrade foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mkubectl rollout status -n foundation4ai deployment/foundation4ai-api-serverkubectl rollout status -n foundation4ai deployment/foundation4ai-api-server-workerExpected result: Helm reports
STATUS: deployed, and both Deployments reportsuccessfully rolled out. Helm deletes each installation Job after the Job succeeds. -
Restart the API server and the workers by hand only when the application release was rendered without access to the cluster, as Rendering and namespace limits describes, or when a Secret changed without an upgrade:
kubectl rollout restart -n foundation4ai \deployment/foundation4ai-api-server deployment/foundation4ai-api-server-workerkubectl rollout status -n foundation4ai deployment/foundation4ai-api-serverkubectl rollout status -n foundation4ai deployment/foundation4ai-api-server-workerExpected result: both Deployments report
restarted, thensuccessfully rolled out.
Credential handling
Credentials can leak through operator actions, so the following rules apply:
- Secrets file. The deployment files exclude
*.envfiles from version control. The operator keepsfoundation4ai.secrets.envin the organization's secret store, not in a shared directory or a repository. - Command output.
kubectl get secret -o yaml,helm get hooksandhelm get allprint credentials in base64 encoding, which is an encoding and not encryption. Output of these commands is not pasted into tickets, chat messages or support requests. - Scripts. Installation scripts do not trace commands with
set -xwhile commands carry credentials, and do not echo the secrets file. - Support requests. A support request to the Foundation4 provider carries the system ID, Job and pod logs and Helm status output, never the secrets file, a password, the application secret or the master key secret.
- Access. Read access to Secrets in the
foundation4ainamespace is limited to the operators of the installation. Security hardening describes access restrictions.
Rendering and namespace limits
The two charts read the Secrets in different ways, and the following limits follow:
- Core release. The core release reads
foundation4ai-secretsthrough the Helmlookupfunction while Helm renders the templates, to build the connection URLs.helm template, client-side dry runs and tools that render charts offline, such as Argo CD, receive no data fromlookup, and the render fails withThe Secret foundation4ai-secrets was not found in namespace <namespace>. The core release is therefore installed and upgraded withhelm installorhelm upgradeagainst the cluster, or checked with--dry-run=server. - Application release. The application release mounts the Secrets in the pods, so the application release also renders with
helm template. The restart checksums then stay constant, and the API server and the workers are restarted by hand after a change of a mounted value, as in step 4 of the procedure. - Namespace.
kustomization.yamlsets no namespace, so the apply command names the namespace with-n. Each release reads Secrets from the release namespace, sofoundation4ai-secretsand both releases share one namespace,foundation4aion these pages, as Install on Kubernetes describes.
Troubleshooting
Troubleshooting describes two further problems with Secrets: pods using a previous value after a change and credentials missing after installation through a GitOps tool.
Installation Jobs failing with a configuration error
- Symptoms. The Job
foundation4ai-api-server-db-migrationfails, and Helm reports that the pre-install or pre-upgrade hook failed. - Diagnosis.
kubectl logs -n foundation4ai job/foundation4ai-api-server-db-migrationshowsError: "Failed to parse config: <reason>".kubectl get secret foundation4ai-secrets -n foundation4ai -o jsonpath='{.data.FOUNDATION4AI_APP_MASTER_KEY}' | base64 -dshows whether the master key identifier is a UUID. - Cause. A setting cannot be parsed: the master key identifier is not a UUID, or an entry of
api-server.configsorapi-server.secretsis not valid YAML. - Resolution. Correct the value, apply the Secret, and repeat the application install or upgrade. Helm replaces the failed Job when the Job runs again.
- Actions to avoid. Editing the Secrets
foundation4ai-coreorfoundation4ai-api-serverdirectly. The next upgrade replaces both Secrets.
API server and workers failing with a secret key error
- Symptoms. The
serverandworkercontainers restart repeatedly after an installation or a change of the application secret. - Diagnosis.
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c server --previousshowsFailed to connect to Foundation4.ai's services: Decryption error: Error decoding secret key. - Cause.
FOUNDATION4AI_APP_SECRETis not 32 bytes in URL-safe base64 encoding, for example because the value was generated with a different length or copied incompletely. - Resolution. Restore the previous application secret from the secret store, then apply steps 1 and 3 of the change procedure. For a first installation, generate a new value with
openssl rand -base64 32 | tr '+/' '-_'. - Actions to avoid. Generating a new application secret for an installation that holds LLM API keys or embedding model credentials, which makes the stored values unreadable.