Skip to main content

Security hardening

This page describes the hardening measures that the charts and the code support in a production deployment: TLS on client and database connections, network policies, pod security settings, availability protection, least-privilege API keys, secret storage, logging and access to model servers. Operators and security reviewers use this page when they prepare a production installation and when they review one. Security at a glance summarizes how Foundation4 protects stored content.

Hardening measures​

MeasureConfigured inChart default
TLS for client connectionsingress.tls, or the Gateway listenerNo TLS
Restriction of paths without an API keyIngress controller or GatewayNone
TLS to PostgreSQLPOSTGRES_URL and database.ca_certBundled PostgreSQL: sslmode=disable
TLS to NATS JetStreamnats.config.nats.tls and nats.config.cluster.tls of the core release, api-server.natsCaSecretNo TLS
Network policiesManifests applied by the operatorNone
Pod security contextapi-server and dashboard valuesAPI server, workers and installation Jobs: user 10001, no privilege escalation, no capabilities. Dashboard: empty
Service account tokensserviceAccount.automount values and the default service accountMounted
Disruption budgets and priority classespodDisruptionBudget and priorityClassName valuesNATS JetStream budget only
API keysFoundation4 APIMaster key only
Log levelapp.log_levelwarn

Client TLS​

The API server and the dashboard serve plain HTTP inside the cluster. The Service foundation4ai-api-server listens on port 80 and forwards to port 8000, and the Service foundation4ai-dashboard listens on port 80 and forwards to port 3000. TLS for client applications and browsers terminates at the ingress controller or at the Gateway.

Ingress​

  1. Create the TLS Secret from the certificate and key issued for the host name:

    kubectl create secret tls foundation4ai-tls -n foundation4ai --cert=tls.crt --key=tls.key

    Expected result: secret/foundation4ai-tls created. A cluster with cert-manager can instead set an issuer annotation, such as cert-manager.io/cluster-issuer: <issuer-name>, in ingress.annotations, and cert-manager creates the Secret.

  2. Set the host name and the TLS Secret in foundation4ai.values.yaml, then upgrade the application release:

    ingress:
    enabled: true
    className: <ingress-class>
    hosts:
    - host: foundation4.example.com
    tls:
    - secretName: foundation4ai-tls
    hosts:
    - foundation4.example.com
    helm upgrade foundation4ai ./charts/foundation4ai \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
    kubectl get ingress -n foundation4ai

    Expected result: Helm reports STATUS: deployed, and the Ingress foundation4ai lists the host name with ports 80, 443.

  3. Check the certificate and an authenticated request through the host name:

    openssl s_client -connect foundation4.example.com:443 -servername foundation4.example.com </dev/null 2>/dev/null \
    | openssl x509 -noout -subject -enddate
    curl -sS -o /dev/null -w '%{http_code}\n' -X POST https://foundation4.example.com/login \
    -H "x-api-key: $FOUNDATION4_API_KEY" \
    -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

    Expected result: the certificate subject names the host, notAfter shows the expiry date, and the request prints 201.

Gateway​

With httpRoute.enabled set to true, the chart attaches an HTTPRoute to the Gateway listener named in httpRoute.parentRefs. The chart default names the listener http of a Gateway called gateway, so a deployment with TLS names an HTTPS listener instead. The Gateway, the listener and the certificate Secret are created outside the charts:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: foundation4ai
namespace: foundation4ai
spec:
gatewayClassName: <gateway-class>
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: foundation4.example.com
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: foundation4ai-tls
httpRoute:
enabled: true
parentRefs:
- name: foundation4ai
sectionName: https
hostnames:
- foundation4.example.com

A Gateway in another namespace adds namespace to the parent reference and allows routes from the foundation4ai namespace in the listener's allowedRoutes. Through the chart's HTTPRoute, as through the Ingress, /dashboard reaches the dashboard and every other path, including GET /, reaches the API server. API and dashboard access describes both routes.

Paths without an API key​

The API server answers the root route /, the health check /healthz and the API documentation routes without an API key, as API conventions lists. The Ingress and the HTTPRoute of the chart forward every path outside /dashboard to the API server, so these routes are reachable wherever the API is exposed. The following rules apply to a production deployment:

  • Monitoring routes. The API server serves /metrics on the metrics port 9090, which no Service, Ingress or HTTPRoute of the charts exposes, so the metrics are reachable only inside the cluster. /statistics and /license require an admin API key. The dashboard calls none of these routes.
  • Collection and probes. The bundled Prometheus server scrapes the pods inside the cluster, and the kubelet sends the probes to the pods directly, so a restriction at the ingress affects neither metric collection nor the probes.
  • Health check and API documentation routes. The health check and the documentation routes are restricted at the ingress controller or the Gateway where the security policy of the deployment requires a restriction, for example with a second Ingress for the same host that lists the paths and the source address rule of the ingress controller.

PostgreSQL TLS​

Foundation4 connects to an external PostgreSQL database with the URL in POSTGRES_URL of foundation4ai.secrets.env. The sslmode parameter of the URL sets encryption and certificate checks. The configuration key database.ca_cert supplies the certificate authority (CA) certificate in PEM format. The API server, the workers and the installation Jobs read both settings.

sslmodeEncryptionCertificate check
disableNoneNone
prefer, the default when the parameter is absentWhen the server offers TLSNone
requireAlwaysWith database.ca_cert, certificate signed by the CA, as verify-ca; without database.ca_cert, none
verify-caAlwaysCertificate signed by the CA
verify-fullAlwaysCertificate signed by the CA and issued for the host name in the URL

A production deployment uses verify-full:

POSTGRES_URL=postgres://<user>:<password>@<database-host>:5432/<database>?sslmode=verify-full

The CA certificate goes in a configuration file under api-server.configs. The file name f4ai-10-database-tls.yaml sorts after the chart defaults and before the credential files that the chart generates, as the load order in Configuration reference describes:

api-server:
configs:
f4ai-10-database-tls.yaml: |
database:
ca_cert: |
-----BEGIN CERTIFICATE-----
<CA certificate lines>
-----END CERTIFICATE-----

The change takes effect through the change procedure in Secrets and keys, which ends with a restart of the API server and the workers. A database session with administrative rights then confirms that every Foundation4 connection uses TLS:

psql "<administrator-connection-url>" -c \
"SELECT a.usename, s.ssl, s.version FROM pg_stat_ssl s JOIN pg_stat_activity a USING (pid) WHERE a.datname = '<database>';"

Expected result: every row shows ssl as t and a TLS version such as TLSv1.3. The database server can also reject unencrypted connections, with hostssl entries in pg_hba.conf or the equivalent setting of a managed PostgreSQL service. The bundled PostgreSQL of the evaluation profile connects with sslmode=disable and is not a production database, as described in Database.

Connections inside the namespace​

The core release builds redis:// and nats:// URLs, so by default the API server and the workers connect to the Redis-compatible cache and to NATS JetStream without TLS. The following settings apply to these connections:

  • NATS JetStream authentication. NATS JetStream accepts only clients that authenticate with the user foundation4ai and the NATS_PASSWORD value of the operator Secret.
  • NATS JetStream client TLS. The core values nats.config.nats.tls.enabled: true and nats.config.nats.tls.secretName name a Secret with tls.crt, tls.key and ca.crt, whose certificate covers foundation4ai-core-nats. The application value api-server.natsCaSecret names the Secret that holds the CA certificate, under the key api-server.natsCaKey, ca.crt by default. The core release then builds a tls:// URL, and the API server and the workers verify the certificate of NATS JetStream against that CA.
  • NATS JetStream route TLS. The core values nats.config.cluster.tls.enabled: true, nats.config.cluster.tls.secretName and nats.config.cluster.tls.merge with ca_file: /etc/nats-certs/cluster/ca.crt and verify: true turn on TLS between the three NATS servers. The certificate of the Secret covers *.foundation4ai-core-nats-headless and allows server and client authentication. Turning route TLS on or off needs a restart of all NATS servers together, which interrupts the queue for a few seconds.
  • External cache. The REDIS_URL of an external cache can use rediss://, and the API server and the workers then verify the certificate of the cache against the system CA store of the image. No setting supplies a private CA for the cache. The bundled Valkey is reached with redis://.
  • gRPC service. The gRPC service runs as a sidecar container, a second container in the same Kubernetes pod, next to each API server and worker process, and is reached on localhost.

The network policies in the next section limit these connections to the pods that use them.

Network policies​

The charts ship no network policies for the Foundation4 pods. The Valkey and PostgreSQL charts render a policy only when redis.networkPolicy or postgres.networkPolicy is set, and the Prometheus chart only when prometheus.networkPolicy.enabled is true. None of these values is set by default, and the NATS chart has no policy template. The policy set below allows the connections that the charts and the code use and denies all other traffic to and from the pods of the namespace. The policies name container ports, not Service ports.

SourceDestinationPortPurpose
Ingress controller or GatewayAPI server pods8000API requests
Ingress controller or GatewayDashboard pods3000Dashboard
API server, workers, installation JobsPostgreSQL5432Primary data store
API server, workers, master key JobValkey6379Cache
API server, workersNATS JetStream4222Queue and object store
NATS JetStreamNATS JetStream6222Routes between the 3 NATS servers
NATS utility pod foundation4ai-core-nats-boxNATS JetStream4222Operator commands
Prometheus serverAPI server pods, worker pods9090Metrics
Prometheus serverKubernetes API443 or 6443Pod discovery
API server, workersModel serversServer portEmbedding and LLM requests
Every podCluster DNS53Name resolution

The installation Job pods carry the label app.kubernetes.io/name: api-server-job, not the labels of the API server, so the policy foundation4ai-jobs-egress names the Jobs.

Policy set​

The policy set uses the following placeholders:

PlaceholderValue
<ingress-namespace>Namespace of the ingress controller or the Gateway data plane, such as ingress-nginx, or kube-system for the Traefik controller of k3s
<model-server-cidr>Address range of the embedding and LLM servers, with the server port in place of 443 where the servers listen on another port
<kubernetes-api-address>Address of the Kubernetes API from step 2 of the procedure, with the port in place of 6443 where the API listens on another port
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-app-egress
spec:
podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [api-server, api-server-worker]
- key: app.kubernetes.io/instance
operator: In
values: [foundation4ai]
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: postgres
app.kubernetes.io/instance: foundation4ai-core
ports:
- protocol: TCP
port: 5432
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: redis
app.kubernetes.io/instance: foundation4ai-core
ports:
- protocol: TCP
port: 6379
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
ports:
- protocol: TCP
port: 4222
- to:
- ipBlock:
cidr: <model-server-cidr>
ports:
- protocol: TCP
port: 443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-jobs-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-server-job
app.kubernetes.io/instance: foundation4ai
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: postgres
app.kubernetes.io/instance: foundation4ai-core
ports:
- protocol: TCP
port: 5432
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: redis
app.kubernetes.io/instance: foundation4ai-core
ports:
- protocol: TCP
port: 6379
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-api-server-ingress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-server
app.kubernetes.io/instance: foundation4ai
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: <ingress-namespace>
ports:
- protocol: TCP
port: 8000
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/component: server
ports:
- protocol: TCP
port: 9090
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-worker-ingress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-server-worker
app.kubernetes.io/instance: foundation4ai
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/component: server
ports:
- protocol: TCP
port: 9090
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-dashboard-ingress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: dashboard
app.kubernetes.io/instance: foundation4ai
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: <ingress-namespace>
ports:
- protocol: TCP
port: 3000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-core-nats
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [api-server, api-server-worker]
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats-box
ports:
- protocol: TCP
port: 4222
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
ports:
- protocol: TCP
port: 6222
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
ports:
- protocol: TCP
port: 6222
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-core-nats-box-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats-box
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
ports:
- protocol: TCP
port: 4222
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-core-data-ingress
spec:
podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [redis, postgres]
- key: app.kubernetes.io/instance
operator: In
values: [foundation4ai-core]
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [api-server, api-server-worker, api-server-job]
ports:
- protocol: TCP
port: 6379
- protocol: TCP
port: 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-core-prometheus-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/component: server
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [api-server, api-server-worker]
ports:
- protocol: TCP
port: 9090
- to:
- ipBlock:
cidr: <kubernetes-api-address>/32
ports:
- protocol: TCP
port: 6443

The policies apply as follows:

  1. Show the labels that the policies select:

    kubectl get pods -n foundation4ai \
    -L app.kubernetes.io/name,app.kubernetes.io/instance,app.kubernetes.io/component

    Expected result: the API server, worker and dashboard pods carry the names api-server, api-server-worker and dashboard with the instance foundation4ai, and a running or failed installation Job pod carries the name api-server-job. The NATS pods carry the name nats with the components nats and nats-box, and the Valkey, Prometheus server and, in the evaluation profile, PostgreSQL pods carry the names redis, prometheus and postgres, all with the instance foundation4ai-core.

  2. Read the address and port of the Kubernetes API, which the Prometheus server contacts for pod discovery:

    kubectl get endpoints kubernetes -n default \
    -o jsonpath='{.subsets[0].addresses[0].ip}{" "}{.subsets[0].ports[0].port}{"\n"}'

    Expected result: one address and one port, such as 10.0.0.1 6443. A control plane with several addresses needs one ipBlock entry per address.

  3. Save the policy set as foundation4ai-network-policies.yaml, replace the placeholders and apply the file:

    kubectl apply -n foundation4ai -f foundation4ai-network-policies.yaml

    Expected result: one networkpolicy.networking.k8s.io/<name> created line for each of the 11 policies.

  4. Check the pods, NATS JetStream and the API:

    kubectl get pods -n foundation4ai
    NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
    kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \
    nats --user foundation4ai --password "$NATS_PASSWORD" server check connection

    Expected result: every pod stays Running with an unchanged ready count, and the NATS check reports OK. The request of step 3 of the ingress procedure prints 201, and a document submitted as in Ingest documents reliably reaches the status success.

The following points apply to the policy set:

  • External PostgreSQL. The production profile replaces the postgres destination of foundation4ai-app-egress and foundation4ai-jobs-egress with an ipBlock entry for the database address range on port 5432.
  • Installation Jobs. The migration Job and the license check Job connect to PostgreSQL only, and the master key Job to PostgreSQL and the Redis-compatible cache. No installation Job connects to NATS JetStream or to a model server.
  • Health probes. The kubelet runs the HTTP and gRPC probes from the node. Most network plugins let node traffic reach local pods regardless of policy. A plugin that applies policies to node traffic needs an ingress rule for the node addresses, and the ready counts in step 4 show whether the probes pass.
  • Port forwarding. kubectl port-forward connects inside the network namespace of the pod, so the port forwards in API and dashboard access are expected to work under the policies (inferred).
  • Diagnostic pods. A temporary pod started in the namespace, such as a PostgreSQL client, matches no allow rule and reaches only DNS. An operator adds a policy for such a pod or runs the check from outside the namespace.
  • Internet access. The set allows no connection to the internet. FastEmbed models whose files are not in the image cannot download, as described in Models, packages and air-gapped installs.

Pod security​

The api-server values set a security context for the API server, worker and installation Job pods. The pods run as the user and group 10001 with runAsNonRoot, fsGroup 10001 and the RuntimeDefault seccomp profile, and every container runs without privilege escalation and with all capabilities dropped. These defaults meet the restricted Pod Security Standard. The root file system stays writable, because the gRPC service writes its Python environment into the container at start. An API server or gRPC service image that runs as user 0 does not start with these defaults.

api-server.securityContext applies to every container of these pods, and api-server.server.securityContext, api-server.worker.securityContext, api-server.grpc.securityContext and api-server.jobs.securityContext replace the shared value for one kind of container.

The dashboard values set no security context: podSecurityContext and securityContext are empty in the dashboard values, and the dashboard image runs as user 1001. The core charts set security contexts for Valkey, which runs as user 1000 with a read-only root file system and no capabilities, and for the bundled PostgreSQL and the Prometheus server, which run as users 999 and 65534 with runAsNonRoot. The NATS chart sets no security context.

The following values give the dashboard the restrictions of the API server pods:

dashboard:
podSecurityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL

The Pod Security Admission labels of the namespace enforce a Pod Security Standard. The restricted standard rejects the NATS JetStream pods of the core release, which carry no security context, and the dashboard pods without the values above, so a namespace that holds both releases enforces baseline and reports restricted violations as warnings:

  1. Preview the effect on the running pods:

    kubectl label --dry-run=server --overwrite namespace foundation4ai \
    pod-security.kubernetes.io/enforce=baseline

    Expected result: namespace/foundation4ai labeled (server dry run), with no warning about existing pods.

  2. Apply the labels:

    kubectl label --overwrite namespace foundation4ai \
    pod-security.kubernetes.io/enforce=baseline pod-security.kubernetes.io/warn=restricted

    Expected result: namespace/foundation4ai labeled. Later Helm upgrades print restricted warnings for the pods that do not meet the standard and still succeed.

Service account tokens​

The Foundation4 processes make no calls to the Kubernetes API, so the Foundation4 pods need no service account token. The API server and dashboard charts create service accounts that mount the token by default, and the worker pods use the service account of the API server chart. The installation Jobs and the NATS JetStream pods use the default service account of the namespace. The following values and patch stop the token mounts:

api-server:
serviceAccount:
automount: false
dashboard:
serviceAccount:
automount: false
kubectl patch serviceaccount default -n foundation4ai -p '{"automountServiceAccountToken": false}'

Expected result: serviceaccount/default patched. The setting applies to pods created after the change. After the next restart, kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ls /var/run/secrets/kubernetes.io/serviceaccount reports that the directory does not exist. The Prometheus server keeps the token of the Prometheus service account, which pod discovery needs.

Images​

  • Tags. The charts provide no default image tags: the values file names the tags in global.serverImageTag, global.grpcImageTag and global.dashboardImageTag, or in the image values of each chart, and the rendering of the chart fails without them, as listed in Values reference. A production deployment uses tags that the registry never reassigns.
  • Digests. The charts join the repository and the tag with a colon, so a tag of the form <tag>@sha256:<digest> also pins the image digest (inferred).
  • Registry. Production clusters pull from an internal registry or mirror, as described in Charts, images and installation bundle. global.foundation4aiImageRegistry, such as registry.example.com/foundation4ai, replaces the registry host of the API server, gRPC service and dashboard images.

Availability protection​

The charts create no PodDisruptionBudget and no priority class for the API server, the workers or the dashboard by default. The NATS chart creates the budget foundation4ai-core-nats, which allows one of the three NATS servers to be unavailable at a time. The Prometheus budget is disabled by default, and Valkey runs as one replica.

The podDisruptionBudget values of the application release create budgets that keep API server, worker and dashboard capacity during node drains and cluster upgrades. Each value is a budget specification without a selector, and the chart adds the selector:

api-server:
podDisruptionBudget:
server:
maxUnavailable: 1
worker:
maxUnavailable: 1
dashboard:
podDisruptionBudget:
maxUnavailable: 1
helm upgrade foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl get pdb -n foundation4ai

Expected result: Helm reports STATUS: deployed, and foundation4ai-api-server, foundation4ai-api-server-worker, foundation4ai-dashboard and foundation4ai-core-nats are listed, each with ALLOWED DISRUPTIONS of 1 when every pod is ready.

  • Job pods. The installation Job pods carry the label app.kubernetes.io/name: api-server-job, so the API server budget covers only the API server pods.
  • Replica count. A budget keeps the API available only with 2 or more API server replicas. With api-server.replicaCount.server at the default of 1, a drain stops the API server until the pod is scheduled again.
  • Queue replicas. Foundation4 creates the DOCUMENTS stream and the stream of the documents object store with the number of copies in global.natsReplicas, 3 by default, so the queue and the stored document text remain available while one of the three NATS servers is unavailable. At startup, the API server and the workers change the replica count of an existing stream to the configured value and log Changing the replica count of a NATS stream at INFO. The value does not exceed the number of NATS servers, so a single external NATS server needs global.natsReplicas: 1. nats stream info DOCUMENTS, described in Operator command reference, shows the replica count.
  • Worker shutdown. On the termination signal, a worker completes the batch in progress and exits. A batch that is still running at the end of the termination grace period, 30 seconds by default, returns to the queue and is processed again.

A priority class protects the Foundation4 pods from preemption by less important workloads when nodes run short of resources. The following class is saved as foundation4ai-priority-class.yaml:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: foundation4ai
value: 100000
globalDefault: false
description: Foundation4 services
kubectl apply -f foundation4ai-priority-class.yaml

Expected result: priorityclass.scheduling.k8s.io/foundation4ai created.

The core charts accept the class in the values of the core release. The NATS chart takes the class through a pod template merge, which has not been tested:

redis:
priorityClassName: foundation4ai
postgres:
priorityClassName: foundation4ai
prometheus:
server:
priorityClassName: foundation4ai
nats:
podTemplate:
merge:
spec:
priorityClassName: foundation4ai

The application release takes the class in api-server.priorityClassName, for the API server and worker pods, and in dashboard.priorityClassName:

api-server:
priorityClassName: foundation4ai
dashboard:
priorityClassName: foundation4ai

After the upgrade of the application release, the following command shows the class of each Deployment:

kubectl get deployments -n foundation4ai \
-o custom-columns=NAME:.metadata.name,PRIORITY:.spec.template.spec.priorityClassName

Expected result: foundation4ai in the PRIORITY column of each Foundation4 Deployment.

API keys​

Foundation4 authenticates every request with an API key and checks the permissions of the key on each object. A production deployment applies the following rules, which Manage API keys and permissions describes step by step:

  • One key per application. Each client application or service holds a separate key, so that one key can be replaced or deleted without affecting other applications.
  • Object permissions. Each key holds permissions on the individual objects that the application uses, at the lowest level that each operation needs, as listed in Access control.
  • Master key. The master key serves only for administration and does not belong in client application configuration.
  • Expiration and rotation. Each key carries an expiration date and is replaced on a schedule, as described in the key rotation section of the guide.
  • Review. The operator reviews the key list and the permissions of each key on a schedule.
  • Dashboard sign-in. The dashboard keeps the API key and the secret in the session storage of one browser tab, so a new tab or a new browser session asks for the key again, and closing the tab removes the key from the browser.

Secret storage​

The Kubernetes Secrets in the foundation4ai namespace and the file foundation4ai.secrets.env hold the database and cache passwords, the application secret, the master key and the license. Secrets and keys lists each Secret and the handling rules for command output and support requests. The following controls apply to the storage:

  • Read access. Role-based access control (RBAC) grants get, list and watch on Secrets, and create on pods/exec, in the namespace only to the operators of the installation. kubectl auth can-i get secrets -n foundation4ai --as <user> and kubectl auth can-i create pods --subresource=exec -n foundation4ai --as <user> print no for every other user.
  • Encryption at rest. The cluster encrypts Secrets at rest, through the encryption configuration of the Kubernetes API server or the key management service of the platform.
  • Secrets file. foundation4ai.secrets.env is kept in the organization's secret store, and working copies are deleted after use.
  • Recovery copies. The secret store also holds the application secret, the master key identifier and secret, and the license outside the cluster, because a database restore needs the same application secret and master key, as described in Backup, restore and upgrades.
  • Helm values. Credentials stay out of values files and --set arguments, because Helm stores the values in the release history, and helm get values prints the values.

Logging​

The chart sets app.log_level to warn for the API server and the workers, so the processes log warnings and errors only. The sample values file supplied with the deployment files keeps warn and shows, as a comment, a configs file that sets debug for a diagnosis. The following command shows every log level that the values set:

helm get values foundation4ai -n foundation4ai | grep -n log_level

Expected result: no output, or only lines with warn.

Lower levels add, at info, a line for each refused request and each processed document and, at debug and trace, a line for every request and lines for database statements. Each lower level increases the log volume, so a lower level is set only for a limited diagnosis and removed afterwards. Configuration reference describes how a configs file sets the level. Logs are operational records, so access to the log store is limited to the operators of the installation.

Model servers​

The API server pods call LLM servers and embedding servers, and the worker pods call embedding servers, over HTTP. The following rules keep these connections inside the organization's network:

  • Internal endpoints. Model servers run inside the organization's network, and the endpoint of each embedding model and LLM names an internal address. The API server and the gRPC service accept private addresses with the default true of api-server.allowPrivateEndpoints, as described in LLMs. Providers and models lists the endpoint parameters.
  • Restricted listeners. Each model server accepts connections only from the cluster's pod or node addresses, and the egress rule of the policy set names only the model server addresses.
  • Encrypted endpoints. An endpoint on a network shared with other systems uses an https:// URL, because requests to model servers carry document text and prompts.