Install on Kubernetes

This page installs Foundation4 on Kubernetes in the production profile: an external PostgreSQL database, images from a registry, resource settings and access through an ingress. Operators follow the numbered procedure once per installation and use the later sections for updates and secret changes. Deployment overview and requirements describes the components and requirements; Install for evaluation installs the evaluation profile on a single node.

Command forms

The deployment files include a recipe file, .justfile, for the command runner just. Each recipe wraps one or two commands, and every step below shows both forms.

• just create-secrets. Runs kubectl kustomize | kubectl apply -f - in the working directory and creates or updates the Secret foundation4ai-secrets.
• just deploy-core. Runs helm upgrade --install foundation4ai-core ./charts/foundation4ai-core -n foundation4ai --create-namespace -f foundation4ai.values.yaml.
• just deploy. Runs the same command for the release foundation4ai and the chart ./charts/foundation4ai.
• just check. Starts test pods that connect to Valkey, NATS JetStream and Prometheus.

The recipes behave as follows:

• Values file. The recipes always read foundation4ai.values.yaml.
• Waiting. The recipes pass no --wait and no --timeout. Helm waits for the hook Jobs up to the default timeout of 5 minutes and returns without waiting for the Deployments.
• Cluster and namespace. The environment variable CLUSTER selects a kubeconfig context; the variable NAMESPACE keeps the default foundation4ai, because the Secret is always created in foundation4ai.
• Check output. just check prints ok after each check whatever the result of the check. The output of each check shows the actual result.

Hook order

Each release creates some objects as Helm hooks before the other objects, in order of hook weight. An installation that stalls stops at the first Job that does not complete, and kubectl -n foundation4ai get jobs shows which Job.

Release	Weight	Object	Log on success
foundation4ai-core	-10	Secret foundation4ai-core: connection URLs, license, application secret and master key	Not applicable
foundation4ai-core	-5	Secret foundation4ai-core-redis: the Valkey password	Not applicable
foundation4ai	-10	ConfigMap and Secret foundation4ai-api-server: configuration files	Not applicable
foundation4ai	-9	Job foundation4ai-api-server-db-migration: database migration	Migrated to latest schema.
foundation4ai	-8	Job foundation4ai-api-server-license-check: license check	Checking for a valid license... OK
foundation4ai	-7	Job foundation4ai-api-server-create-admin-api-key: master key	Admin API key created successfully.

The hooks run on every install and every upgrade. The Deployments, Services and access objects follow after the last hook completes.

The license check prints the system ID and then the result: OK, Missing, Invalid or Expired. On any result other than OK, the Job waits instead of failing, so Helm stops the installation at the timeout. Troubleshooting describes each result, and Licensing describes the license.

Installation

Run every command from the directory that holds the deployment files.

1. Namespace

The Secret of step 5 is always created in the namespace foundation4ai, and just create-secrets does not create the namespace. Create the namespace first in both command forms:

kubectl create namespace foundation4ai

Expected result: namespace/foundation4ai created.

2. Image pull secret

This step applies when the registry requires credentials. Create the pull secret from a Docker configuration file that holds the registry credentials:

kubectl -n foundation4ai create secret generic <pull secret> \
  --type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=<docker config file>

Expected result: secret/<pull secret> created. Charts, images and installation bundle describes mirrored core images and expiring registry tokens.

3. Values file

The recipes read foundation4ai.values.yaml. The deployment files include a sample file with that name, which points at the provider's registry and enables debug logging. Keep the sample under another name:

mv foundation4ai.values.yaml foundation4ai.values.sample.yaml

Create foundation4ai.values.yaml with the following content, replacing each value in angle brackets:

global:
  serverImageTag: "<api server image tag>"
  grpcImageTag: "<grpc service image tag>"
  dashboardImageTag: "<dashboard image tag>"
  imagePullSecrets:
  - name: <pull secret>

postgres:
  enabled: false

api-server:
  image:
    repository: <registry>/foundation4ai-api
    repositoryGrpc: <registry>/foundation4ai-grpc
  resources:
    requests:
      cpu: "<cpu request>"
      memory: "<memory request>"
    limits:
      memory: "<memory limit>"

dashboard:
  image:
    repository: <registry>/foundation4ai-dashboard

ingress:
  enabled: true
  className: <ingress class>
  hosts:
  - host: <host name>
  tls:
  - secretName: <tls secret>
    hosts:
    - <host name>

The values file sets the following:

• Image tags. The three tags are required. The charts provide no default tag.
• External database. postgres.enabled: false uses the external database named in the secrets file.
• Resources. api-server.resources applies to the API server, worker and both gRPC service containers, so the values fit the largest of the four. Scaling and performance describes resource settings for the other components.
• Pull credentials. Remove imagePullSecrets when the registry needs no credentials.
• Access. The ingress routes /dashboard to the dashboard and every other path to the API server. API and dashboard access describes the ingress, the Transport Layer Security (TLS) Secret and the HTTPRoute alternative. The ingress block can also be added in a later upgrade.

Confirm that no placeholder remains:

grep -n '<' foundation4ai.values.yaml

Expected result: no output.

4. Secrets file

Copy the template and generate the values. openssl rand -hex 24 produces a password or secret, uuidgen produces the master key identifier and the openssl rand -base64 32 command produces the application secret:

cp foundation4ai.secrets.env.template foundation4ai.secrets.env
openssl rand -hex 24
uuidgen
openssl rand -base64 32 | tr '+/' '-_'

Edit foundation4ai.secrets.env and set the following keys:

Key	Value
POSTGRES_URL	The connection URL of the external database, postgres://<user>:<password>@<host>:5432/<database>, with a password of letters and digits only
POSTGRES_DATABASE, POSTGRES_USER	Unchanged; the production profile does not use these keys
POSTGRES_SUPERUSER_PASSWORD, POSTGRES_PASSWORD	New openssl rand -hex 24 values, so that no template password remains; the production profile does not use these keys
REDIS_PASSWORD	A new openssl rand -hex 24 value
REDIS_URL	Empty; the core release builds the Valkey URL
FOUNDATION4AI_APP_SECRET	The openssl rand -base64 32 value
FOUNDATION4AI_APP_MASTER_KEY	The uuidgen value
FOUNDATION4AI_APP_MASTER_SECRET	A new openssl rand -hex 24 value
FOUNDATION4AI_APP_LICENSE	pending until step 10

Record the master key identifier, the master secret and the application secret in the organization's secret store. The master secret cannot be changed after installation, and the stored API keys of LLMs cannot be decrypted without the application secret. Secrets and keys describes each secret.

Expected result: the file contains a value for every key except REDIS_URL.

5. Secret

kubectl kustomize . | kubectl apply -f -

Recipe form: just create-secrets.

Expected result: secret/foundation4ai-secrets created.

6. Pre-installation checks

Run the checks in Pre-installation checks, including the cleanup at the end of that page.

Expected result: every check shows the expected result, and the test objects are deleted.

7. Core release

helm upgrade --install foundation4ai-core ./charts/foundation4ai-core \
  -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai get pods,pvc

Recipe form: just deploy-core, then kubectl -n foundation4ai get pods,pvc until every pod is ready.

Expected result: Helm reports STATUS: deployed, and the following pods are Running. 4 volume claims are Bound. No PostgreSQL pod runs in the production profile.

Pod	Ready
foundation4ai-core-nats-0, foundation4ai-core-nats-1, foundation4ai-core-nats-2	2/2
foundation4ai-core-nats-box-<suffix>	1/1
foundation4ai-core-redis-<suffix>	1/1
foundation4ai-core-prometheus-server-<suffix>	2/2

8. Core release check

helm test foundation4ai-core -n foundation4ai

Recipe form: just check, reading the output of each check rather than the ok marks. The recipe pulls public images.

Expected result: the test pods foundation4ai-core-nats-test-request-reply and foundation4ai-core-redis-test-auth-existing report Succeeded. The Valkey test pulls a public image and fails where the cluster cannot reach Docker Hub; in that case, the Running status of the Valkey pod from step 7 is the check. A Prometheus server pod that restarts repeatedly is described in Troubleshooting.

9. System ID

The first application install runs the database migration and then stops at the license check, because the license is still pending. This failure is expected.

helm upgrade --install foundation4ai ./charts/foundation4ai \
  -n foundation4ai -f foundation4ai.values.yaml --timeout 3m
kubectl -n foundation4ai get jobs
kubectl -n foundation4ai logs job/foundation4ai-api-server-license-check

Recipe form: just deploy, which waits 5 minutes. The Job log can be read from a second terminal as soon as the license check Job starts.

Expected result: Helm reports that the pre-install hook failed. The migration Job shows 1/1 completions and the license check Job 0/1. The license check log contains the system ID:

SystemID: <system ID>
Checking for a valid license... Invalid

10. License

Send the system ID to the Foundation4 provider and receive the license. Replace pending with the license in foundation4ai.secrets.env. Then update the Secret, upgrade the core release so that the core release copies the license into the Secret foundation4ai-core, and delete the waiting license check Job:

kubectl kustomize . | kubectl apply -f -
helm upgrade foundation4ai-core ./charts/foundation4ai-core \
  -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai delete job foundation4ai-api-server-license-check

Recipe form: just create-secrets, just deploy-core, then the kubectl delete job command.

Expected result: secret/foundation4ai-secrets configured, Helm reports STATUS: deployed for foundation4ai-core, and job.batch "foundation4ai-api-server-license-check" deleted.

11. Application release

Remove the failed release and install the application release again:

helm uninstall foundation4ai -n foundation4ai
helm upgrade --install foundation4ai ./charts/foundation4ai \
  -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai get jobs,pods
kubectl -n foundation4ai logs job/foundation4ai-api-server-license-check
kubectl -n foundation4ai logs job/foundation4ai-api-server-create-admin-api-key

Recipe form: the helm uninstall command, then just deploy, then kubectl -n foundation4ai rollout status deployment/foundation4ai-api-server-worker to wait for the workers.

Expected result: release "foundation4ai" uninstalled, then Helm reports STATUS: deployed. The three Jobs show 1/1 completions. The pod foundation4ai-api-server-<suffix> and the 3 pods foundation4ai-api-server-worker-<suffix> show 2/2 containers ready, and the pod foundation4ai-dashboard-<suffix> shows 1/1. The license check log ends with Checking for a valid license... OK, and the master key log shows Admin API key created successfully.

12. Installation check

Confirm that the API server or a worker created the document queue in NATS JetStream:

kubectl -n foundation4ai exec deploy/foundation4ai-core-nats-box -- nats stream info DOCUMENTS

Expected result: the stream information for DOCUMENTS, with work-queue retention and a maximum age of 1 day.

Then sign in with the master key as described in API and dashboard access. Expected result: POST /login returns status 201 and the name Master Admin Key.

13. First API keys

The master key serves only for administration and does not belong in the configuration of a client application. Create one key for each client application, with permissions on the individual objects that the application uses, as described in Manage API keys and permissions. The requests run with the master key in the shell variables, as in step 12.

Foundation4 returns the secret of a new key only in the response to POST /api-keys and stores only a hash of the secret. Record each key identifier and secret in the organization's secret store before handing the key to the team that runs the client application.

Expected result: POST /login with each new key returns status 201 and the name of the key, as described in Key check.

Evaluation profile differences

The evaluation profile follows the same order with the values of Install for evaluation. The differences are the following:

Setting	Evaluation	Production
Values file	evaluation.values.yaml; the recipes read only foundation4ai.values.yaml	foundation4ai.values.yaml
postgres.enabled	true	false
POSTGRES_URL	Empty; the core release builds the URL	The external database URL
Resources	None	api-server.resources and the settings in Scaling and performance
Workers	api-server.replicaCount.worker: 1	The default of 3 or more
Core release pods	A PostgreSQL pod in addition to the production pods	The pods listed in step 7
Access	Port forward, and an ingress for the dashboard	Ingress or HTTPRoute with TLS

Updates

An update, such as new image tags or changed values, upgrades both releases in order:

helm upgrade foundation4ai-core ./charts/foundation4ai-core \
  -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
helm upgrade foundation4ai ./charts/foundation4ai \
  -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

Recipe form: just deploy-core, then just deploy.

Expected result: Helm reports STATUS: deployed for both releases, and the three Jobs complete again.

The upgrade runs the hooks of the hook order table. The migration Job migrates the database while the existing pods continue to serve requests; the Deployments then roll to the new version. The master key Job leaves an existing master key unchanged. An update never uninstalls a release, because uninstalling removes the API server and workers and interrupts service. Backup, restore and upgrades describes the backup before an upgrade and the recovery from a failed migration.

Secret and license changes

The hook Secrets change on every upgrade, but the Deployments restart only when the pod specification changes. After a change to the secrets file, such as a renewed license, update the Secret and both releases, then restart the API server and workers:

kubectl kustomize . | kubectl apply -f -
helm upgrade foundation4ai-core ./charts/foundation4ai-core \
  -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
helm upgrade foundation4ai ./charts/foundation4ai \
  -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl -n foundation4ai rollout restart deployment/foundation4ai-api-server deployment/foundation4ai-api-server-worker
kubectl -n foundation4ai rollout status deployment/foundation4ai-api-server-worker

Recipe form: just create-secrets, just deploy-core, just deploy, then the two kubectl rollout commands.

Expected result: deployment.apps/foundation4ai-api-server restarted, deployment.apps/foundation4ai-api-server-worker restarted, then deployment "foundation4ai-api-server-worker" successfully rolled out.

The master key never changes after installation, and a changed application secret requires the API key of each LLM to be set again. Secrets and keys lists the secrets that can change.

Namespace and release names

The deployment files fix the names of an installation:

• Namespace. The kustomization file creates the Secret in foundation4ai, whatever namespace the commands or the NAMESPACE variable name. Foundation4 is therefore installed in foundation4ai, one installation per cluster.
• Core release. The application release reads the Secret foundation4ai-core by name, so the core release is always named foundation4ai-core.
• Application release. This documentation uses the release name foundation4ai. The names of the Jobs, Deployments and Services, such as foundation4ai-api-server, derive from the release name.