Install on Kubernetes
This page installs Foundation4 on Kubernetes in the production profile: an external PostgreSQL database, images from a registry, resource settings and access through an ingress. Operators follow the numbered procedure once per installation and use the later sections for updates and secret changes. Deployment overview and requirements describes the components and requirements; Install for evaluation installs the evaluation profile on a single node.
Command forms
The deployment files include a recipe file, .justfile, for the command runner just. Each recipe wraps one or more commands. The steps below show both forms, except step 8, which uses the recipe just check.
just create-secrets. Creates the namespace when the namespace does not exist, then runskubectl kustomize | kubectl apply -n foundation4ai -f -in the working directory and creates or updates the Secretfoundation4ai-secrets.just deploy-core. Runshelm upgrade --install foundation4ai-core ./charts/foundation4ai-core -n foundation4ai --create-namespace --timeout 10m -f foundation4ai.values.yaml.just deploy. Runs the same command for the releasefoundation4aiand the chart./charts/foundation4ai.just system-id. Prints the system ID from the log of the license check Job. After installation, the recipe starts a temporary podget-system-idfrom the pod template of the API server, which prints the system ID.just check. Starts test pods that connect to Valkey, NATS JetStream and Prometheus, one after the other.
The recipes behave as follows:
- Values file. The recipes always read
foundation4ai.values.yaml. - Waiting. The recipes pass
--timeoutwith the value of the environment variableHELM_TIMEOUT, 10 minutes by default, and no--wait. Helm waits for the hook Jobs up to the timeout and returns without waiting for the Deployments. - Cluster and namespace. The environment variable
CLUSTERselects a kubeconfig context, and the variableNAMESPACEselects the namespace,foundation4aiby default. The recipes read the variables from the environment or from an optional.envfile in the working directory. - Check output.
just checkprints a line that ends inokfor each check that passes and stops at the first check that fails. Each check waits up to 120 seconds for its test pod.
Hook order
Each release creates some objects as Helm hooks before the other objects, in order of hook weight. An installation that fails stops at the first Job that does not complete, and kubectl -n foundation4ai get jobs shows which Job.
| Release | Weight | Object | Log on success |
|---|---|---|---|
foundation4ai-core | -10 | Secret foundation4ai-core: connection URLs | Not applicable |
foundation4ai-core | -5 | Secret foundation4ai-core-redis: the Valkey password | Not applicable |
foundation4ai | -10 | ConfigMap and Secret foundation4ai-api-server: configuration files | Not applicable |
foundation4ai | -9 | Job foundation4ai-api-server-db-migration: database migration | Migrated to latest schema. |
foundation4ai | -8 | Job foundation4ai-api-server-license-check: license check | Checking for a valid license... OK |
foundation4ai | -7 | Job foundation4ai-api-server-create-admin-api-key: master key | Master key created successfully. on the first installation, Master key already matches the configuration. on an upgrade |
The hooks run on every install and every upgrade. The Deployments, Services and access objects follow after the last hook completes. Helm deletes the three Jobs once all of them complete, so the Job logs can be read only while the Jobs run. When a Job fails, the Jobs stay, with their logs, until the next install or upgrade.
The license check prints the system ID and then the result: OK, Missing, Invalid, Expired, Invalid version, Exceeded limits, or a note that the system ID for the license version could not be computed. On any result other than OK, the Job fails at once without a retry, and Helm stops the installation with an error. Troubleshooting describes the installation failures, and Licensing describes the license.
Installation
Run every command from the directory that holds the deployment files.
1. Namespace
The pull secret of step 2 and the Secret of step 5 are created in the namespace foundation4ai. Create the namespace first:
kubectl create namespace foundation4ai
Recipe form: none; just create-secrets in step 5 also creates the namespace, but runs after step 2.
Expected result: namespace/foundation4ai created.
2. Image pull secret
This step applies when the registry requires credentials. Create the pull secret from a Docker configuration file that holds the registry credentials:
kubectl -n foundation4ai create secret generic <pull secret> \
--type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=<docker config file>
Expected result: secret/<pull secret> created. Charts, images and installation bundle describes mirrored core images and expiring registry tokens.
3. Values file
The recipes read foundation4ai.values.yaml. The deployment files include a sample file with that name, which points the model and package images at the provider's registry and names the provider's pull secret. Keep the sample under another name:
mv foundation4ai.values.yaml foundation4ai.values.sample.yaml
Create foundation4ai.values.yaml with the following content, replacing each value in angle brackets:
global:
serverImageTag: "<api server image tag>"
grpcImageTag: "<grpc service image tag>"
dashboardImageTag: "<dashboard image tag>"
imagePullSecrets:
- <pull secret>
postgres:
enabled: false
api-server:
image:
repository: <registry>/foundation4ai-api
repositoryGrpc: <registry>/foundation4ai-grpc
resources:
requests:
cpu: "<cpu request>"
memory: "<memory request>"
limits:
memory: "<memory limit>"
dashboard:
image:
repository: <registry>/foundation4ai-dashboard
ingress:
enabled: true
className: <ingress class>
hosts:
- host: <host name>
tls:
- secretName: <tls secret>
hosts:
- <host name>
The values file sets the following:
- Image tags. The three tags are required. The charts provide no default tag.
- External database.
postgres.enabled: falseuses the external database named in the secrets file. - Resources.
api-server.resourcesapplies to the API server, worker and gRPC service containers and to the installation Jobs. The valuesapi-server.server.resources,api-server.worker.resources,api-server.grpc.resourcesandapi-server.jobs.resourcesset a container's own requests and limits instead. Scaling and performance describes resource settings for the other components. - Pull credentials. Remove
imagePullSecretswhen the registry needs no credentials. The plain secret name also serves the Valkey pod of the core release, which reads the same value. - Access. The ingress routes
/dashboardto the dashboard and every other path to the API server. API and dashboard access describes the ingress, the Transport Layer Security (TLS) Secret and the HTTPRoute alternative. Theingressblock can also be added in a later upgrade.
Confirm that no placeholder remains:
grep -n '<' foundation4ai.values.yaml
Expected result: no output.
4. Secrets file
Copy the template and generate the values. openssl rand -hex 24 produces a password or secret, uuidgen produces the master key identifier and the openssl rand -base64 32 command produces the application secret:
cp foundation4ai.secrets.env.template foundation4ai.secrets.env
openssl rand -hex 24
uuidgen
openssl rand -base64 32 | tr '+/' '-_'
Edit foundation4ai.secrets.env and set the following keys:
| Key | Value |
|---|---|
POSTGRES_URL | The connection URL of the external database, postgres://<user>:<password>@<host>:5432/<database>, with every character of the user and the password other than letters, digits, ., _, ~ and - percent-encoded |
POSTGRES_DATABASE, POSTGRES_USER | Unchanged; the production profile does not use these keys |
POSTGRES_SUPERUSER_PASSWORD, POSTGRES_PASSWORD | New openssl rand -hex 24 values, so that no template password remains; the production profile does not use these keys |
REDIS_PASSWORD | A new openssl rand -hex 24 value |
REDIS_URL | Empty; the core release builds the Valkey URL |
NATS_PASSWORD | A new openssl rand -hex 24 value; NATS JetStream accepts only clients that sign in with this password |
NATS_URL | Empty; the core release builds the NATS JetStream URL |
FOUNDATION4AI_APP_SECRET | The openssl rand -base64 32 value |
FOUNDATION4AI_APP_MASTER_KEY | The uuidgen value |
FOUNDATION4AI_APP_MASTER_SECRET | A new openssl rand -hex 24 value |
FOUNDATION4AI_APP_LICENSE | pending until step 10 |
Record the master key identifier, the master secret and the application secret in the organization's secret store. The stored API keys of LLMs cannot be decrypted without the application secret. Secrets and keys describes each secret.
Expected result: the file contains a value for every key except REDIS_URL and NATS_URL.
5. Secret
kubectl kustomize . | kubectl apply -n foundation4ai -f -
Recipe form: just create-secrets.
Expected result: secret/foundation4ai-secrets created.
6. Pre-installation checks
Run the checks in Pre-installation checks, including the cleanup at the end of that page.
Expected result: every check shows the expected result, and the test objects are deleted.
7. Core release
helm upgrade --install foundation4ai-core ./charts/foundation4ai-core \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai get pods,pvc
Recipe form: just deploy-core, then kubectl -n foundation4ai get pods,pvc until every pod is ready.
Expected result: Helm reports STATUS: deployed, and the following pods are Running. 4 volume claims are Bound. No PostgreSQL pod runs in the production profile.
| Pod | Ready |
|---|---|
foundation4ai-core-nats-0, foundation4ai-core-nats-1, foundation4ai-core-nats-2 | 2/2 |
foundation4ai-core-nats-box-<suffix> | 1/1 |
foundation4ai-core-redis-<suffix> | 1/1 |
foundation4ai-core-prometheus-server-<suffix> | 2/2 |
8. Core release check
The check signs in to Valkey with the Valkey password and to NATS JetStream with the NATS JetStream password, and queries the readiness endpoint of Prometheus:
just check
Expected result:
Checking Redis connectivity... ok
Checking NATS connectivity... ok
Checking Prometheus connectivity... ok
A check that fails stops the recipe after up to 120 seconds with a kubectl wait error, and the test pod stays for inspection. The recipe pulls the public images redis:alpine, natsio/nats-box:latest and curlimages/curl. Where the cluster cannot pull them, the Running status of the pods from step 7 and the stream check of step 12 serve as the check.
9. System ID
The first application install runs the database migration and then stops at the license check, because the license is still pending. This failure is expected.
helm upgrade --install foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --timeout 10m
kubectl -n foundation4ai get jobs
kubectl -n foundation4ai logs job/foundation4ai-api-server-license-check
Recipe form: just deploy, then just system-id, which prints only the system ID.
Expected result: Helm reports that the pre-install hook failed with BackoffLimitExceeded. The migration Job shows 1/1 completions and the license check Job 0/1. The license check log contains the system ID:
SystemID: <system ID>
Checking for a valid license... Invalid
Error: "License check failed: Invalid"
When the first line reads SystemID: unavailable with a reason, Foundation4 cannot compute the system ID from the database, for example because the database user cannot run pg_control_system(). Licensing describes the system ID.
10. License
Send the system ID to the Foundation4 provider and receive the license. Replace pending with the license in foundation4ai.secrets.env, then update the Secret:
kubectl kustomize . | kubectl apply -n foundation4ai -f -
Recipe form: just create-secrets.
Expected result: secret/foundation4ai-secrets configured.
The application pods read the license from the Secret foundation4ai-secrets, so the core release needs no upgrade. The failed license check Job stays until the installation of step 11 replaces the Job.
11. Application release
Remove the failed release and install the application release again:
helm uninstall foundation4ai -n foundation4ai
helm upgrade --install foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai get jobs,pods
Recipe form: the helm uninstall command, then just deploy, then kubectl -n foundation4ai rollout status deployment/foundation4ai-api-server-worker to wait for the workers.
Expected result: release "foundation4ai" uninstalled, then Helm reports STATUS: deployed, which confirms that the migration, the license check and the master key Jobs completed. Helm then deletes the Jobs, so no Job is listed. The pod foundation4ai-api-server-<suffix> and the 3 pods foundation4ai-api-server-worker-<suffix> show 2/2 containers ready, and the pod foundation4ai-dashboard-<suffix> shows 1/1.
12. Installation check
Confirm that the API server or a worker created the document queue in NATS JetStream:
kubectl -n foundation4ai exec deploy/foundation4ai-core-nats-box -- nats --user foundation4ai \
--password "$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)" \
stream info DOCUMENTS
Expected result: the stream information for DOCUMENTS, with work-queue retention, a maximum age of 1 day and 3 replicas.
Then sign in with the master key as described in API and dashboard access. Expected result: POST /login returns status 201 and the name Master Admin Key.
13. First API keys
The master key serves only for administration and does not belong in the configuration of a client application. Create one key for each client application, with permissions on the individual objects that the application uses, as described in Manage API keys and permissions. The requests run with the master key in the shell variables, as in step 12.
Foundation4 returns the secret of a new key only in the response to POST /api-keys and stores only a hash of the secret. Record each key identifier and secret in the organization's secret store before handing the key to the team that runs the client application.
Expected result: POST /login with each new key returns status 201 and the name of the key, as described in Key check.
Evaluation profile differences
The evaluation profile follows the same order with the values of Install for evaluation. The differences are the following:
| Setting | Evaluation | Production |
|---|---|---|
| Values file | evaluation.values.yaml; the recipes read only foundation4ai.values.yaml | foundation4ai.values.yaml |
postgres.enabled | true | false |
POSTGRES_URL | Empty; the core release builds the URL | The external database URL |
| Resources | None | api-server.resources and the settings in Scaling and performance |
| Workers | api-server.replicaCount.worker: 1 | The default of 3 or more |
| Core release pods | A PostgreSQL pod in addition to the production pods | The pods listed in step 7 |
| Access | Port forward, and an ingress for the dashboard | Ingress or HTTPRoute with TLS |
Updates
An update, such as new image tags or changed values, upgrades both releases in order:
helm upgrade foundation4ai-core ./charts/foundation4ai-core \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
helm upgrade foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
Recipe form: just deploy-core, then just deploy.
Expected result: Helm reports STATUS: deployed for both releases, which confirms that the three Jobs completed again.
The upgrade runs the hooks of the hook order table. The migration Job migrates the database while the existing pods continue to serve requests; the Deployments then roll to the new version. The master key Job leaves the master key unchanged while the master key identifier and the master secret stay the same. An update never uninstalls a release, because uninstalling removes the API server and workers and interrupts service. The migrations cannot be reversed, so a return to the previous version after a migration needs the database backup taken before the upgrade. Backup, restore and upgrades describes the backup before an upgrade and the recovery from a failed migration.
Secret and license changes
The API server and workers read the credentials and the connection URLs only at startup. After a change to the secrets file, such as a renewed license, update the Secret and both releases:
kubectl kustomize . | kubectl apply -n foundation4ai -f -
helm upgrade foundation4ai-core ./charts/foundation4ai-core \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
helm upgrade foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai rollout status deployment/foundation4ai-api-server-worker
Recipe form: just create-secrets, just deploy-core, just deploy, then the kubectl rollout status command.
Expected result: Helm reports STATUS: deployed for both releases, then deployment "foundation4ai-api-server-worker" successfully rolled out.
The upgrade of the application release restarts the API server and workers when the license, the application secret, the master key identifier, the master secret or a connection URL changed. Where the charts are rendered offline, for example by a GitOps tool, the pods do not restart on their own, and kubectl -n foundation4ai rollout restart deployment/foundation4ai-api-server deployment/foundation4ai-api-server-worker restarts them after the upgrade.
The master key Job of the upgrade applies a changed master key identifier or master secret. A changed application secret requires the API key of each LLM to be set again. Secrets and keys lists the secrets that can change.
Namespace and release names
The deployment files fix the names of an installation:
- Namespace. The commands name the namespace, and the recipes use the namespace in the
NAMESPACEvariable. Both releases and the Secretfoundation4ai-secretsare in the same namespace. This documentation uses the namespacefoundation4ai. - Core release. The application release reads the Secret
foundation4ai-coreby default, and the core values name the Secretsfoundation4ai-core-redisandfoundation4ai-core-nats-authafter the release name. The core release is therefore namedfoundation4ai-core. Another name requires overrides of the three Secret names in the values. - Application release. This documentation uses the release name
foundation4ai. The names of the Jobs, Deployments and Services, such asfoundation4ai-api-server, derive from the release name.