Skip to main content

Install on Kubernetes

This page installs Foundation4 on Kubernetes in the production profile: an external PostgreSQL database, images from a registry, resource settings and access through an ingress. Operators follow the numbered procedure once per installation and use the later sections for updates and secret changes. Deployment overview and requirements describes the components and requirements; Install for evaluation installs the evaluation profile on a single node.

Command forms​

The deployment files include a recipe file, .justfile, for the command runner just. Each recipe wraps one or more commands. The steps below show both forms, except step 8, which uses the recipe just check.

  • just create-secrets. Creates the namespace when the namespace does not exist, then runs kubectl kustomize | kubectl apply -n foundation4ai -f - in the working directory and creates or updates the Secret foundation4ai-secrets.
  • just deploy-core. Runs helm upgrade --install foundation4ai-core ./charts/foundation4ai-core -n foundation4ai --create-namespace --timeout 10m -f foundation4ai.values.yaml.
  • just deploy. Runs the same command for the release foundation4ai and the chart ./charts/foundation4ai.
  • just system-id. Prints the system ID from the log of the license check Job. After installation, the recipe starts a temporary pod get-system-id from the pod template of the API server, which prints the system ID.
  • just check. Starts test pods that connect to Valkey, NATS JetStream and Prometheus, one after the other.

The recipes behave as follows:

  • Values file. The recipes always read foundation4ai.values.yaml.
  • Waiting. The recipes pass --timeout with the value of the environment variable HELM_TIMEOUT, 10 minutes by default, and no --wait. Helm waits for the hook Jobs up to the timeout and returns without waiting for the Deployments.
  • Cluster and namespace. The environment variable CLUSTER selects a kubeconfig context, and the variable NAMESPACE selects the namespace, foundation4ai by default. The recipes read the variables from the environment or from an optional .env file in the working directory.
  • Check output. just check prints a line that ends in ok for each check that passes and stops at the first check that fails. Each check waits up to 120 seconds for its test pod.

Hook order​

Each release creates some objects as Helm hooks before the other objects, in order of hook weight. An installation that fails stops at the first Job that does not complete, and kubectl -n foundation4ai get jobs shows which Job.

ReleaseWeightObjectLog on success
foundation4ai-core-10Secret foundation4ai-core: connection URLsNot applicable
foundation4ai-core-5Secret foundation4ai-core-redis: the Valkey passwordNot applicable
foundation4ai-10ConfigMap and Secret foundation4ai-api-server: configuration filesNot applicable
foundation4ai-9Job foundation4ai-api-server-db-migration: database migrationMigrated to latest schema.
foundation4ai-8Job foundation4ai-api-server-license-check: license checkChecking for a valid license... OK
foundation4ai-7Job foundation4ai-api-server-create-admin-api-key: master keyMaster key created successfully. on the first installation, Master key already matches the configuration. on an upgrade

The hooks run on every install and every upgrade. The Deployments, Services and access objects follow after the last hook completes. Helm deletes the three Jobs once all of them complete, so the Job logs can be read only while the Jobs run. When a Job fails, the Jobs stay, with their logs, until the next install or upgrade.

The license check prints the system ID and then the result: OK, Missing, Invalid, Expired, Invalid version, Exceeded limits, or a note that the system ID for the license version could not be computed. On any result other than OK, the Job fails at once without a retry, and Helm stops the installation with an error. Troubleshooting describes the installation failures, and Licensing describes the license.

Installation​

Run every command from the directory that holds the deployment files.

1. Namespace​

The pull secret of step 2 and the Secret of step 5 are created in the namespace foundation4ai. Create the namespace first:

kubectl create namespace foundation4ai

Recipe form: none; just create-secrets in step 5 also creates the namespace, but runs after step 2.

Expected result: namespace/foundation4ai created.

2. Image pull secret​

This step applies when the registry requires credentials. Create the pull secret from a Docker configuration file that holds the registry credentials:

kubectl -n foundation4ai create secret generic <pull secret> \
--type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=<docker config file>

Expected result: secret/<pull secret> created. Charts, images and installation bundle describes mirrored core images and expiring registry tokens.

3. Values file​

The recipes read foundation4ai.values.yaml. The deployment files include a sample file with that name, which points the model and package images at the provider's registry and names the provider's pull secret. Keep the sample under another name:

mv foundation4ai.values.yaml foundation4ai.values.sample.yaml

Create foundation4ai.values.yaml with the following content, replacing each value in angle brackets:

global:
serverImageTag: "<api server image tag>"
grpcImageTag: "<grpc service image tag>"
dashboardImageTag: "<dashboard image tag>"
imagePullSecrets:
- <pull secret>

postgres:
enabled: false

api-server:
image:
repository: <registry>/foundation4ai-api
repositoryGrpc: <registry>/foundation4ai-grpc
resources:
requests:
cpu: "<cpu request>"
memory: "<memory request>"
limits:
memory: "<memory limit>"

dashboard:
image:
repository: <registry>/foundation4ai-dashboard

ingress:
enabled: true
className: <ingress class>
hosts:
- host: <host name>
tls:
- secretName: <tls secret>
hosts:
- <host name>

The values file sets the following:

  • Image tags. The three tags are required. The charts provide no default tag.
  • External database. postgres.enabled: false uses the external database named in the secrets file.
  • Resources. api-server.resources applies to the API server, worker and gRPC service containers and to the installation Jobs. The values api-server.server.resources, api-server.worker.resources, api-server.grpc.resources and api-server.jobs.resources set a container's own requests and limits instead. Scaling and performance describes resource settings for the other components.
  • Pull credentials. Remove imagePullSecrets when the registry needs no credentials. The plain secret name also serves the Valkey pod of the core release, which reads the same value.
  • Access. The ingress routes /dashboard to the dashboard and every other path to the API server. API and dashboard access describes the ingress, the Transport Layer Security (TLS) Secret and the HTTPRoute alternative. The ingress block can also be added in a later upgrade.

Confirm that no placeholder remains:

grep -n '<' foundation4ai.values.yaml

Expected result: no output.

4. Secrets file​

Copy the template and generate the values. openssl rand -hex 24 produces a password or secret, uuidgen produces the master key identifier and the openssl rand -base64 32 command produces the application secret:

cp foundation4ai.secrets.env.template foundation4ai.secrets.env
openssl rand -hex 24
uuidgen
openssl rand -base64 32 | tr '+/' '-_'

Edit foundation4ai.secrets.env and set the following keys:

KeyValue
POSTGRES_URLThe connection URL of the external database, postgres://<user>:<password>@<host>:5432/<database>, with every character of the user and the password other than letters, digits, ., _, ~ and - percent-encoded
POSTGRES_DATABASE, POSTGRES_USERUnchanged; the production profile does not use these keys
POSTGRES_SUPERUSER_PASSWORD, POSTGRES_PASSWORDNew openssl rand -hex 24 values, so that no template password remains; the production profile does not use these keys
REDIS_PASSWORDA new openssl rand -hex 24 value
REDIS_URLEmpty; the core release builds the Valkey URL
NATS_PASSWORDA new openssl rand -hex 24 value; NATS JetStream accepts only clients that sign in with this password
NATS_URLEmpty; the core release builds the NATS JetStream URL
FOUNDATION4AI_APP_SECRETThe openssl rand -base64 32 value
FOUNDATION4AI_APP_MASTER_KEYThe uuidgen value
FOUNDATION4AI_APP_MASTER_SECRETA new openssl rand -hex 24 value
FOUNDATION4AI_APP_LICENSEpending until step 10

Record the master key identifier, the master secret and the application secret in the organization's secret store. The stored API keys of LLMs cannot be decrypted without the application secret. Secrets and keys describes each secret.

Expected result: the file contains a value for every key except REDIS_URL and NATS_URL.

5. Secret​

kubectl kustomize . | kubectl apply -n foundation4ai -f -

Recipe form: just create-secrets.

Expected result: secret/foundation4ai-secrets created.

6. Pre-installation checks​

Run the checks in Pre-installation checks, including the cleanup at the end of that page.

Expected result: every check shows the expected result, and the test objects are deleted.

7. Core release​

helm upgrade --install foundation4ai-core ./charts/foundation4ai-core \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai get pods,pvc

Recipe form: just deploy-core, then kubectl -n foundation4ai get pods,pvc until every pod is ready.

Expected result: Helm reports STATUS: deployed, and the following pods are Running. 4 volume claims are Bound. No PostgreSQL pod runs in the production profile.

PodReady
foundation4ai-core-nats-0, foundation4ai-core-nats-1, foundation4ai-core-nats-22/2
foundation4ai-core-nats-box-<suffix>1/1
foundation4ai-core-redis-<suffix>1/1
foundation4ai-core-prometheus-server-<suffix>2/2

8. Core release check​

The check signs in to Valkey with the Valkey password and to NATS JetStream with the NATS JetStream password, and queries the readiness endpoint of Prometheus:

just check

Expected result:

Checking Redis connectivity... ok
Checking NATS connectivity... ok
Checking Prometheus connectivity... ok

A check that fails stops the recipe after up to 120 seconds with a kubectl wait error, and the test pod stays for inspection. The recipe pulls the public images redis:alpine, natsio/nats-box:latest and curlimages/curl. Where the cluster cannot pull them, the Running status of the pods from step 7 and the stream check of step 12 serve as the check.

9. System ID​

The first application install runs the database migration and then stops at the license check, because the license is still pending. This failure is expected.

helm upgrade --install foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --timeout 10m
kubectl -n foundation4ai get jobs
kubectl -n foundation4ai logs job/foundation4ai-api-server-license-check

Recipe form: just deploy, then just system-id, which prints only the system ID.

Expected result: Helm reports that the pre-install hook failed with BackoffLimitExceeded. The migration Job shows 1/1 completions and the license check Job 0/1. The license check log contains the system ID:

SystemID: <system ID>
Checking for a valid license... Invalid
Error: "License check failed: Invalid"

When the first line reads SystemID: unavailable with a reason, Foundation4 cannot compute the system ID from the database, for example because the database user cannot run pg_control_system(). Licensing describes the system ID.

10. License​

Send the system ID to the Foundation4 provider and receive the license. Replace pending with the license in foundation4ai.secrets.env, then update the Secret:

kubectl kustomize . | kubectl apply -n foundation4ai -f -

Recipe form: just create-secrets.

Expected result: secret/foundation4ai-secrets configured.

The application pods read the license from the Secret foundation4ai-secrets, so the core release needs no upgrade. The failed license check Job stays until the installation of step 11 replaces the Job.

11. Application release​

Remove the failed release and install the application release again:

helm uninstall foundation4ai -n foundation4ai
helm upgrade --install foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai get jobs,pods

Recipe form: the helm uninstall command, then just deploy, then kubectl -n foundation4ai rollout status deployment/foundation4ai-api-server-worker to wait for the workers.

Expected result: release "foundation4ai" uninstalled, then Helm reports STATUS: deployed, which confirms that the migration, the license check and the master key Jobs completed. Helm then deletes the Jobs, so no Job is listed. The pod foundation4ai-api-server-<suffix> and the 3 pods foundation4ai-api-server-worker-<suffix> show 2/2 containers ready, and the pod foundation4ai-dashboard-<suffix> shows 1/1.

12. Installation check​

Confirm that the API server or a worker created the document queue in NATS JetStream:

kubectl -n foundation4ai exec deploy/foundation4ai-core-nats-box -- nats --user foundation4ai \
--password "$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)" \
stream info DOCUMENTS

Expected result: the stream information for DOCUMENTS, with work-queue retention, a maximum age of 1 day and 3 replicas.

Then sign in with the master key as described in API and dashboard access. Expected result: POST /login returns status 201 and the name Master Admin Key.

13. First API keys​

The master key serves only for administration and does not belong in the configuration of a client application. Create one key for each client application, with permissions on the individual objects that the application uses, as described in Manage API keys and permissions. The requests run with the master key in the shell variables, as in step 12.

Foundation4 returns the secret of a new key only in the response to POST /api-keys and stores only a hash of the secret. Record each key identifier and secret in the organization's secret store before handing the key to the team that runs the client application.

Expected result: POST /login with each new key returns status 201 and the name of the key, as described in Key check.

Evaluation profile differences​

The evaluation profile follows the same order with the values of Install for evaluation. The differences are the following:

SettingEvaluationProduction
Values fileevaluation.values.yaml; the recipes read only foundation4ai.values.yamlfoundation4ai.values.yaml
postgres.enabledtruefalse
POSTGRES_URLEmpty; the core release builds the URLThe external database URL
ResourcesNoneapi-server.resources and the settings in Scaling and performance
Workersapi-server.replicaCount.worker: 1The default of 3 or more
Core release podsA PostgreSQL pod in addition to the production podsThe pods listed in step 7
AccessPort forward, and an ingress for the dashboardIngress or HTTPRoute with TLS

Updates​

An update, such as new image tags or changed values, upgrades both releases in order:

helm upgrade foundation4ai-core ./charts/foundation4ai-core \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
helm upgrade foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

Recipe form: just deploy-core, then just deploy.

Expected result: Helm reports STATUS: deployed for both releases, which confirms that the three Jobs completed again.

The upgrade runs the hooks of the hook order table. The migration Job migrates the database while the existing pods continue to serve requests; the Deployments then roll to the new version. The master key Job leaves the master key unchanged while the master key identifier and the master secret stay the same. An update never uninstalls a release, because uninstalling removes the API server and workers and interrupts service. The migrations cannot be reversed, so a return to the previous version after a migration needs the database backup taken before the upgrade. Backup, restore and upgrades describes the backup before an upgrade and the recovery from a failed migration.

Secret and license changes​

The API server and workers read the credentials and the connection URLs only at startup. After a change to the secrets file, such as a renewed license, update the Secret and both releases:

kubectl kustomize . | kubectl apply -n foundation4ai -f -
helm upgrade foundation4ai-core ./charts/foundation4ai-core \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
helm upgrade foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai rollout status deployment/foundation4ai-api-server-worker

Recipe form: just create-secrets, just deploy-core, just deploy, then the kubectl rollout status command.

Expected result: Helm reports STATUS: deployed for both releases, then deployment "foundation4ai-api-server-worker" successfully rolled out.

The upgrade of the application release restarts the API server and workers when the license, the application secret, the master key identifier, the master secret or a connection URL changed. Where the charts are rendered offline, for example by a GitOps tool, the pods do not restart on their own, and kubectl -n foundation4ai rollout restart deployment/foundation4ai-api-server deployment/foundation4ai-api-server-worker restarts them after the upgrade.

The master key Job of the upgrade applies a changed master key identifier or master secret. A changed application secret requires the API key of each LLM to be set again. Secrets and keys lists the secrets that can change.

Namespace and release names​

The deployment files fix the names of an installation:

  • Namespace. The commands name the namespace, and the recipes use the namespace in the NAMESPACE variable. Both releases and the Secret foundation4ai-secrets are in the same namespace. This documentation uses the namespace foundation4ai.
  • Core release. The application release reads the Secret foundation4ai-core by default, and the core values name the Secrets foundation4ai-core-redis and foundation4ai-core-nats-auth after the release name. The core release is therefore named foundation4ai-core. Another name requires overrides of the three Secret names in the values.
  • Application release. This documentation uses the release name foundation4ai. The names of the Jobs, Deployments and Services, such as foundation4ai-api-server, derive from the release name.