Skip to main content

Secrets and keys

This page lists the Kubernetes Secrets of a Foundation4 installation, the format of every key, and the steps that bring a changed value into the running pods. The page also describes what the application secret and the master key protect. Operators who install, back up or change credentials use this page, and security reviewers use the Secret inventory.

The commands on this page read the production values file foundation4ai.values.yaml, created in Install on Kubernetes. An evaluation installation uses evaluation.values.yaml instead, as in Install for evaluation.

Secret chain​

The operator creates one Secret. The core release builds the connection URLs from that Secret, and the pods of the application release read the credentials as files:

  1. kubectl kustomize . | kubectl apply -n foundation4ai -f - builds the Secret foundation4ai-secrets from the file foundation4ai.secrets.env, as kustomization.yaml defines.
  2. The core release foundation4ai-core reads foundation4ai-secrets and writes the Secret foundation4ai-core, with the four connection URLs, the Secret foundation4ai-core-redis, with the Valkey password, and the Secret foundation4ai-core-nats-auth, with the NATS JetStream client password.
  3. The application release foundation4ai mounts the license, the application secret and the master key values from foundation4ai-secrets, and the connection URLs from foundation4ai-core, as files under /app/config/secrets in the API server, worker and installation Job pods. The application release copies no credential into a Secret of the application release.

A changed value in foundation4ai.secrets.env therefore reaches the pods after the Secret is applied and the releases are upgraded: the core release for a connection value, then the application release. The upgrade of the application release restarts the API server and the workers when a mounted value has changed. Changes and restarts gives the procedure.

The releases write foundation4ai-core, foundation4ai-core-redis and foundation4ai-api-server as Helm hooks: Helm replaces the three Secrets before each install and upgrade, and helm uninstall leaves the Secrets in place. foundation4ai-core-nats-auth is a regular resource of the core release, which an upgrade updates in place and helm uninstall removes. Uninstall and cleanup lists the resources that remain after an uninstall.

Secret inventory​

SecretWritten byKeysRead by
foundation4ai-secretsOperator, from foundation4ai.secrets.envThe keys of the secrets fileCore release templates; the bundled PostgreSQL pod (database name, user and passwords); API server, worker and installation Job pods (the four FOUNDATION4AI_APP_* keys, as files)
foundation4ai-coreCore releaseFOUNDATION4AI_DATABASE_URL, FOUNDATION4AI_REDIS_URL, FOUNDATION4AI_NATS_URL, FOUNDATION4AI_PROMETHEUS_URLAPI server, worker and installation Job pods, as files
foundation4ai-core-redisCore release, when the bundled Valkey is enableddefault: the Valkey passwordBundled Valkey
foundation4ai-core-nats-authCore release, when the bundled NATS JetStream is enabledauth.conf: the client user and passwordBundled NATS JetStream
foundation4ai-api-serverApplication releaseOne key for each api-server.secrets entryAPI server, worker and installation Job pods, as configuration files in /app/config
Image pull secretsOperatorRegistry credentialsPods of the application release, through global.imagePullSecrets, and pods of the core release, through the keys that Charts, images and installation bundle lists

The pods of the application release read the following files under /app/config/secrets. The name of each file is the configuration setting that the file sets.

FileSecretKey
app.licensefoundation4ai-secretsFOUNDATION4AI_APP_LICENSE
app.secretfoundation4ai-secretsFOUNDATION4AI_APP_SECRET
app.master.keyfoundation4ai-secretsFOUNDATION4AI_APP_MASTER_KEY
app.master.secretfoundation4ai-secretsFOUNDATION4AI_APP_MASTER_SECRET
database.urlfoundation4ai-coreFOUNDATION4AI_DATABASE_URL
redis.urlfoundation4ai-coreFOUNDATION4AI_REDIS_URL
nats.urlfoundation4ai-coreFOUNDATION4AI_NATS_URL
prometheus.urlfoundation4ai-coreFOUNDATION4AI_PROMETHEUS_URL

The following rules apply to the inventory:

  • Two naming schemes. The keys with single underscores, such as FOUNDATION4AI_APP_MASTER_KEY, are read only by the charts. The Foundation4 process reads configuration files, the files under /app/config/secrets as plain strings, and environment variables with the prefix FOUNDATION4AI__ and double underscores. Configuration reference describes the configuration layers.
  • Fixed names. The application release reads the operator Secret named in global.secrets.existingSecret and the core Secret named in api-server.secret, by default foundation4ai-secrets and foundation4ai-core. The application release reads the four FOUNDATION4AI_APP_* keys under these names. The core values global.secrets.*Key rename the keys that the core release reads; the pages of this section use the default names.
  • Required keys. The pods of the application release mount the four FOUNDATION4AI_APP_* keys, so the four keys are present in foundation4ai-secrets even before the license exists.
  • Additional configuration. api-server.secrets adds configuration files to foundation4ai-api-server for settings that are confidential. Non-confidential settings, such as a database certificate authority, go in api-server.configs.
  • Prometheus URL. The API server queries Prometheus at this URL for the /statistics endpoint that Monitoring and logging describes. With the bundled Prometheus, the core release writes the URL of the bundled server. Otherwise the core release copies PROMETHEUS_URL, and an empty value selects http://prometheus:9090.

Charts, images and installation bundle describes image pull secrets.

Secrets file keys​

The template foundation4ai.secrets.env.template contains the following keys. PROMETHEUS_URL is commented out in the template.

KeyRead whenValue
POSTGRES_DATABASEBundled PostgreSQL (postgres.enabled: true)Database name, foundation4ai
POSTGRES_USERBundled PostgreSQLDatabase user, foundation4ai
POSTGRES_SUPERUSER_PASSWORDBundled PostgreSQLHexadecimal password
POSTGRES_PASSWORDBundled PostgreSQLHexadecimal password
POSTGRES_URLExternal PostgreSQL (postgres.enabled: false, the default)Connection URL, as described in Database
REDIS_PASSWORDBundled Valkey (redis.enabled: true, the default)Hexadecimal password
REDIS_URLExternal Redis-compatible cache (redis.enabled: false)redis://<user>:<password>@<host>:<port>, or rediss:// for a cache that requires TLS
NATS_PASSWORDBundled NATS JetStream (nats.enabled: true, the default)Hexadecimal password of the client user foundation4ai
NATS_URLExternal NATS JetStream (nats.enabled: false)nats://<host>:4222
PROMETHEUS_URLBundled Prometheus disabled (prometheus.enabled: false)Prometheus base URL; optional
FOUNDATION4AI_APP_SECRETAlwaysApplication secret: 32 random bytes in URL-safe base64 encoding
FOUNDATION4AI_APP_MASTER_KEYAlwaysMaster key identifier: a universally unique identifier (UUID)
FOUNDATION4AI_APP_MASTER_SECRETAlwaysMaster key secret: a hexadecimal value
FOUNDATION4AI_APP_LICENSEAlwaysThe license from the Foundation4 provider, or a placeholder such as pending before the license exists

The template contains sample passwords. Every password in foundation4ai.secrets.env is replaced with a generated value before the first installation.

Value generation​

The following commands generate the values. Run each command once for each key that needs a value:

openssl rand -hex 24
uuidgen
openssl rand -base64 32 | tr '+/' '-_'

Expected result: openssl rand -hex 24 prints 48 hexadecimal characters, for a password or the master key secret. uuidgen prints a UUID such as 3f0c9b6e-2d4a-4c1e-9b7f-5a8d2e6c1f04, for the master key identifier. The last command prints 44 characters ending in =, for the application secret.

The formats follow from how the charts and Foundation4 read the values:

  • Passwords. A password may contain any character, because the core release percent-encodes the passwords of the bundled services in the URLs that the core release builds. A URL in the secrets file, such as POSTGRES_URL, is copied unchanged, so a password in such a URL is percent-encoded by the operator. A malformed cache URL stops the process at startup. Hexadecimal passwords need no encoding.
  • Application secret. Foundation4 decodes the application secret as a Fernet key, which is exactly 32 bytes in URL-safe base64 encoding. Any other value stops the API server and the workers with Error decoding secret key.
  • Master key identifier. The identifier must be a UUID. Any other value fails configuration parsing in every installation Job and pod with Failed to parse config.
  • License. An empty license value is reported as a missing license, and any other value that is not a license, such as pending, as an invalid license. Licensing describes the license checks.

Application secret​

The application secret encrypts the data in the following table with Fernet, an authenticated symmetric encryption format.

DataLocationEffect of a changed or lost application secret
API keys of LLMsPostgreSQLStored keys cannot be decrypted, and Foundation4 calls such an LLM without an API key. A model server that requires a key refuses the call: agent executions and LLM queries return HTTP 502 with Upstream LLM error, and the OpenAI-compatible pass-through returns the response of the model server. The API key of each LLM is then set again.
Credential parameters of embedding models, such as api_keyPostgreSQLStored credentials cannot be decrypted. Searches and document processing that use such an embedding model fail until the previous application secret is restored, because the parameters of an embedding model cannot be changed.
API key verification resultsRedis-compatible cache, 15 minutesEntries that cannot be decrypted are discarded and computed again. No action is needed.
Agent execution tracesRedis-compatible cache, 60 minutesTraces written before the change cannot be read.

The application secret does not protect the following data:

  • Fragment text. Fragment text is encrypted with ChaCha20-Poly1305, using a key for each classification that is stored in the same database. Security at a glance describes the storage.
  • API key secrets. Foundation4 stores an Argon2 hash of each API key secret, which needs no key.

A database restore needs the application secret that was in effect when the backup was taken, because the LLM API keys and the embedding model credentials in the backup were encrypted with that secret. The application secret is therefore backed up together with each database backup and stored separately from the backup files. Backup, restore and upgrades describes the backup set. LLMs describes how LLM API keys are set again.

Master key​

The master key is the administration key of the installation. At every installation and upgrade, the installation Job foundation4ai-api-server-create-admin-api-key makes the stored master key match FOUNDATION4AI_APP_MASTER_KEY and FOUNDATION4AI_APP_MASTER_SECRET:

  • New identifier. When no key has the identifier, the Job creates the master key, named Master Admin Key. A previous master key with that name is renamed Master Admin Key (replaced by <new identifier>) and keeps the administration permissions until an administrator deletes the previous key.
  • Changed secret. When the stored secret does not match, the Job replaces the stored secret. The previous secret stops authenticating at once.
  • Active flag and expiry. The Job leaves the active flag and the expiry of an existing master key unchanged.

At startup, the API server and each worker initialize with the master key. A failure has the following effects:

  • API server. A master key secret that does not match the stored key stops the API server with Failed to initialize server: Failed to initialize with the master key: <reason>. A master key that is expired, inactive or missing does not stop the API server: the API server logs The master key is expired, inactive or missing, so requests made with it are refused, followed by the command that restores the key, and serves requests made with other keys.
  • Worker. Any master key failure stops the worker with a message that begins with Failed to initialize Foundation4.ai:, such as Failed to initialize Foundation4.ai: Invalid or missing key.

An expired or deactivated master key is restored inside the API server container, and the workers are then restarted:

kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \
./foundation4ai admin update-admin-api-key --active true --clear-expiration
kubectl rollout restart -n foundation4ai deployment/foundation4ai-api-server-worker
kubectl rollout status -n foundation4ai deployment/foundation4ai-api-server-worker

Expected result: Master key updated successfully., then successfully rolled out for the workers. A deleted master key is created again by the master key Job at the next upgrade of the application release. Command-line interface lists the options of the command.

The operator records the master key identifier and secret in the organization's secret store. Access control describes the permissions and the rotation of the master key, and Authenticate reads the values back from the Secret.

Changes and restarts​

Foundation4 reads configuration once, at process startup. The Deployments of the API server and the workers carry checksums of the configuration and of the mounted keys of foundation4ai-secrets and foundation4ai-core, so an upgrade of the application release that changes one of these values restarts the API server and the workers.

ChangeSteps
LicenseSteps 1 and 3 of the procedure below; Licensing adds a check of the new license first
Application secretSteps 1 and 3 of the procedure below, then set the API key of each LLM again. Embedding models with credential parameters stop working, as Application secret describes.
Master key secretSteps 1 and 3 of the procedure below. The previous secret stops authenticating when the master key Job runs.
Master key identifierSteps 1 and 3 of the procedure below, then delete the previous master key once no client uses the previous key, as Master key describes
External PostgreSQL password or URLChange the password in PostgreSQL, then the procedure below. Database connections fail from the password change until the restart completes.
External cache or NATS JetStream URLThe procedure below
Bundled NATS JetStream passwordThe procedure below. The bundled NATS JetStream applies the new password in step 2 without a restart.
Bundled Valkey passwordThe procedure below, with a restart of the Deployment foundation4ai-core-redis after step 2
Bundled PostgreSQL passwordsNot supported. The bundled chart sets the passwords only when the database is initialized.
api-server.configs or api-server.secrets entriesStep 3 of the procedure below

The procedure applies a changed value from foundation4ai.secrets.env:

  1. Edit foundation4ai.secrets.env and apply the Secret:

    kubectl kustomize . | kubectl apply -n foundation4ai -f -

    Expected result: secret/foundation4ai-secrets configured.

  2. Upgrade the core release, which writes the connection URLs into foundation4ai-core:

    helm upgrade foundation4ai-core ./charts/foundation4ai-core \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

    Expected result: Helm reports STATUS: deployed.

  3. Upgrade the application release, which runs the installation Jobs and restarts the API server and the workers when a mounted value has changed:

    helm upgrade foundation4ai ./charts/foundation4ai \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
    kubectl rollout status -n foundation4ai deployment/foundation4ai-api-server
    kubectl rollout status -n foundation4ai deployment/foundation4ai-api-server-worker

    Expected result: Helm reports STATUS: deployed, and both Deployments report successfully rolled out. Helm deletes each installation Job after the Job succeeds.

  4. Restart the API server and the workers by hand only when the application release was rendered without access to the cluster, as Rendering and namespace limits describes, or when a Secret changed without an upgrade:

    kubectl rollout restart -n foundation4ai \
    deployment/foundation4ai-api-server deployment/foundation4ai-api-server-worker
    kubectl rollout status -n foundation4ai deployment/foundation4ai-api-server
    kubectl rollout status -n foundation4ai deployment/foundation4ai-api-server-worker

    Expected result: both Deployments report restarted, then successfully rolled out.

Credential handling​

Credentials can leak through operator actions, so the following rules apply:

  • Secrets file. The deployment files exclude *.env files from version control. The operator keeps foundation4ai.secrets.env in the organization's secret store, not in a shared directory or a repository.
  • Command output. kubectl get secret -o yaml, helm get hooks and helm get all print credentials in base64 encoding, which is an encoding and not encryption. Output of these commands is not pasted into tickets, chat messages or support requests.
  • Scripts. Installation scripts do not trace commands with set -x while commands carry credentials, and do not echo the secrets file.
  • Support requests. A support request to the Foundation4 provider carries the system ID, Job and pod logs and Helm status output, never the secrets file, a password, the application secret or the master key secret.
  • Access. Read access to Secrets in the foundation4ai namespace is limited to the operators of the installation. Security hardening describes access restrictions.

Rendering and namespace limits​

The two charts read the Secrets in different ways, and the following limits follow:

  • Core release. The core release reads foundation4ai-secrets through the Helm lookup function while Helm renders the templates, to build the connection URLs. helm template, client-side dry runs and tools that render charts offline, such as Argo CD, receive no data from lookup, and the render fails with The Secret foundation4ai-secrets was not found in namespace <namespace>. The core release is therefore installed and upgraded with helm install or helm upgrade against the cluster, or checked with --dry-run=server.
  • Application release. The application release mounts the Secrets in the pods, so the application release also renders with helm template. The restart checksums then stay constant, and the API server and the workers are restarted by hand after a change of a mounted value, as in step 4 of the procedure.
  • Namespace. kustomization.yaml sets no namespace, so the apply command names the namespace with -n. Each release reads Secrets from the release namespace, so foundation4ai-secrets and both releases share one namespace, foundation4ai on these pages, as Install on Kubernetes describes.

Troubleshooting​

Troubleshooting describes two further problems with Secrets: pods using a previous value after a change and credentials missing after installation through a GitOps tool.

Installation Jobs failing with a configuration error​

  • Symptoms. The Job foundation4ai-api-server-db-migration fails, and Helm reports that the pre-install or pre-upgrade hook failed.
  • Diagnosis. kubectl logs -n foundation4ai job/foundation4ai-api-server-db-migration shows Error: "Failed to parse config: <reason>". kubectl get secret foundation4ai-secrets -n foundation4ai -o jsonpath='{.data.FOUNDATION4AI_APP_MASTER_KEY}' | base64 -d shows whether the master key identifier is a UUID.
  • Cause. A setting cannot be parsed: the master key identifier is not a UUID, or an entry of api-server.configs or api-server.secrets is not valid YAML.
  • Resolution. Correct the value, apply the Secret, and repeat the application install or upgrade. Helm replaces the failed Job when the Job runs again.
  • Actions to avoid. Editing the Secrets foundation4ai-core or foundation4ai-api-server directly. The next upgrade replaces both Secrets.

API server and workers failing with a secret key error​

  • Symptoms. The server and worker containers restart repeatedly after an installation or a change of the application secret.
  • Diagnosis. kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c server --previous shows Failed to connect to Foundation4.ai's services: Decryption error: Error decoding secret key.
  • Cause. FOUNDATION4AI_APP_SECRET is not 32 bytes in URL-safe base64 encoding, for example because the value was generated with a different length or copied incompletely.
  • Resolution. Restore the previous application secret from the secret store, then apply steps 1 and 3 of the change procedure. For a first installation, generate a new value with openssl rand -base64 32 | tr '+/' '-_'.
  • Actions to avoid. Generating a new application secret for an installation that holds LLM API keys or embedding model credentials, which makes the stored values unreadable.