Security hardening
This page describes the hardening measures that the charts and the code support in a production deployment: TLS on client and database connections, network policies, pod security settings, availability protection, least-privilege API keys, secret storage, logging and access to model servers. Operators and security reviewers use this page when they prepare a production installation and when they review one. Security at a glance summarizes how Foundation4 protects stored content.
Hardening measures
| Measure | Configured in | Chart default |
|---|---|---|
| TLS for client connections | ingress.tls, or the Gateway listener | No TLS |
| Restriction of paths without an API key | Ingress controller or Gateway | None |
| TLS to PostgreSQL | POSTGRES_URL and database.ca_cert | Bundled PostgreSQL: sslmode=disable |
| TLS to NATS JetStream | nats.config.nats.tls and nats.config.cluster.tls of the core release, api-server.natsCaSecret | No TLS |
| Network policies | Manifests applied by the operator | None |
| Pod security context | api-server and dashboard values | API server, workers and installation Jobs: user 10001, no privilege escalation, no capabilities. Dashboard: empty |
| Service account tokens | serviceAccount.automount values and the default service account | Mounted |
| Disruption budgets and priority classes | podDisruptionBudget and priorityClassName values | NATS JetStream budget only |
| API keys | Foundation4 API | Master key only |
| Log level | app.log_level | warn |
Client TLS
The API server and the dashboard serve plain HTTP inside the cluster. The Service foundation4ai-api-server listens on port 80 and forwards to port 8000, and the Service foundation4ai-dashboard listens on port 80 and forwards to port 3000. TLS for client applications and browsers terminates at the ingress controller or at the Gateway.
Ingress
-
Create the TLS Secret from the certificate and key issued for the host name:
kubectl create secret tls foundation4ai-tls -n foundation4ai --cert=tls.crt --key=tls.keyExpected result:
secret/foundation4ai-tls created. A cluster with cert-manager can instead set an issuer annotation, such ascert-manager.io/cluster-issuer: <issuer-name>, iningress.annotations, and cert-manager creates the Secret. -
Set the host name and the TLS Secret in
foundation4ai.values.yaml, then upgrade the application release:ingress:enabled: trueclassName: <ingress-class>hosts:- host: foundation4.example.comtls:- secretName: foundation4ai-tlshosts:- foundation4.example.comhelm upgrade foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mkubectl get ingress -n foundation4aiExpected result: Helm reports
STATUS: deployed, and the Ingressfoundation4ailists the host name with ports80, 443. -
Check the certificate and an authenticated request through the host name:
openssl s_client -connect foundation4.example.com:443 -servername foundation4.example.com </dev/null 2>/dev/null \| openssl x509 -noout -subject -enddatecurl -sS -o /dev/null -w '%{http_code}\n' -X POST https://foundation4.example.com/login \-H "x-api-key: $FOUNDATION4_API_KEY" \-H "x-api-key-secret: $FOUNDATION4_API_SECRET"Expected result: the certificate subject names the host,
notAftershows the expiry date, and the request prints201.
Gateway
With httpRoute.enabled set to true, the chart attaches an HTTPRoute to the Gateway listener named in httpRoute.parentRefs. The chart default names the listener http of a Gateway called gateway, so a deployment with TLS names an HTTPS listener instead. The Gateway, the listener and the certificate Secret are created outside the charts:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: foundation4ai
namespace: foundation4ai
spec:
gatewayClassName: <gateway-class>
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: foundation4.example.com
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: foundation4ai-tls
httpRoute:
enabled: true
parentRefs:
- name: foundation4ai
sectionName: https
hostnames:
- foundation4.example.com
A Gateway in another namespace adds namespace to the parent reference and allows routes from the foundation4ai namespace in the listener's allowedRoutes. Through the chart's HTTPRoute, as through the Ingress, /dashboard reaches the dashboard and every other path, including GET /, reaches the API server. API and dashboard access describes both routes.
Paths without an API key
The API server answers the root route /, the health check /healthz and the API documentation routes without an API key, as API conventions lists. The Ingress and the HTTPRoute of the chart forward every path outside /dashboard to the API server, so these routes are reachable wherever the API is exposed. The following rules apply to a production deployment:
- Monitoring routes. The API server serves
/metricson the metrics port 9090, which no Service, Ingress or HTTPRoute of the charts exposes, so the metrics are reachable only inside the cluster./statisticsand/licenserequire an admin API key. The dashboard calls none of these routes. - Collection and probes. The bundled Prometheus server scrapes the pods inside the cluster, and the kubelet sends the probes to the pods directly, so a restriction at the ingress affects neither metric collection nor the probes.
- Health check and API documentation routes. The health check and the documentation routes are restricted at the ingress controller or the Gateway where the security policy of the deployment requires a restriction, for example with a second Ingress for the same host that lists the paths and the source address rule of the ingress controller.
PostgreSQL TLS
Foundation4 connects to an external PostgreSQL database with the URL in POSTGRES_URL of foundation4ai.secrets.env. The sslmode parameter of the URL sets encryption and certificate checks. The configuration key database.ca_cert supplies the certificate authority (CA) certificate in PEM format. The API server, the workers and the installation Jobs read both settings.
sslmode | Encryption | Certificate check |
|---|---|---|
disable | None | None |
prefer, the default when the parameter is absent | When the server offers TLS | None |
require | Always | With database.ca_cert, certificate signed by the CA, as verify-ca; without database.ca_cert, none |
verify-ca | Always | Certificate signed by the CA |
verify-full | Always | Certificate signed by the CA and issued for the host name in the URL |
A production deployment uses verify-full:
POSTGRES_URL=postgres://<user>:<password>@<database-host>:5432/<database>?sslmode=verify-full
The CA certificate goes in a configuration file under api-server.configs. The file name f4ai-10-database-tls.yaml sorts after the chart defaults and before the credential files that the chart generates, as the load order in Configuration reference describes:
api-server:
configs:
f4ai-10-database-tls.yaml: |
database:
ca_cert: |
-----BEGIN CERTIFICATE-----
<CA certificate lines>
-----END CERTIFICATE-----
The change takes effect through the change procedure in Secrets and keys, which ends with a restart of the API server and the workers. A database session with administrative rights then confirms that every Foundation4 connection uses TLS:
psql "<administrator-connection-url>" -c \
"SELECT a.usename, s.ssl, s.version FROM pg_stat_ssl s JOIN pg_stat_activity a USING (pid) WHERE a.datname = '<database>';"
Expected result: every row shows ssl as t and a TLS version such as TLSv1.3. The database server can also reject unencrypted connections, with hostssl entries in pg_hba.conf or the equivalent setting of a managed PostgreSQL service. The bundled PostgreSQL of the evaluation profile connects with sslmode=disable and is not a production database, as described in Database.
Connections inside the namespace
The core release builds redis:// and nats:// URLs, so by default the API server and the workers connect to the Redis-compatible cache and to NATS JetStream without TLS. The following settings apply to these connections:
- NATS JetStream authentication. NATS JetStream accepts only clients that authenticate with the user
foundation4aiand theNATS_PASSWORDvalue of the operator Secret. - NATS JetStream client TLS. The core values
nats.config.nats.tls.enabled: trueandnats.config.nats.tls.secretNamename a Secret withtls.crt,tls.keyandca.crt, whose certificate coversfoundation4ai-core-nats. The application valueapi-server.natsCaSecretnames the Secret that holds the CA certificate, under the keyapi-server.natsCaKey,ca.crtby default. The core release then builds atls://URL, and the API server and the workers verify the certificate of NATS JetStream against that CA. - NATS JetStream route TLS. The core values
nats.config.cluster.tls.enabled: true,nats.config.cluster.tls.secretNameandnats.config.cluster.tls.mergewithca_file: /etc/nats-certs/cluster/ca.crtandverify: trueturn on TLS between the three NATS servers. The certificate of the Secret covers*.foundation4ai-core-nats-headlessand allows server and client authentication. Turning route TLS on or off needs a restart of all NATS servers together, which interrupts the queue for a few seconds. - External cache. The
REDIS_URLof an external cache can userediss://, and the API server and the workers then verify the certificate of the cache against the system CA store of the image. No setting supplies a private CA for the cache. The bundled Valkey is reached withredis://. - gRPC service. The gRPC service runs as a sidecar container, a second container in the same Kubernetes pod, next to each API server and worker process, and is reached on
localhost.
The network policies in the next section limit these connections to the pods that use them.
Network policies
The charts ship no network policies for the Foundation4 pods. The Valkey and PostgreSQL charts render a policy only when redis.networkPolicy or postgres.networkPolicy is set, and the Prometheus chart only when prometheus.networkPolicy.enabled is true. None of these values is set by default, and the NATS chart has no policy template. The policy set below allows the connections that the charts and the code use and denies all other traffic to and from the pods of the namespace. The policies name container ports, not Service ports.
| Source | Destination | Port | Purpose |
|---|---|---|---|
| Ingress controller or Gateway | API server pods | 8000 | API requests |
| Ingress controller or Gateway | Dashboard pods | 3000 | Dashboard |
| API server, workers, installation Jobs | PostgreSQL | 5432 | Primary data store |
| API server, workers, master key Job | Valkey | 6379 | Cache |
| API server, workers | NATS JetStream | 4222 | Queue and object store |
| NATS JetStream | NATS JetStream | 6222 | Routes between the 3 NATS servers |
NATS utility pod foundation4ai-core-nats-box | NATS JetStream | 4222 | Operator commands |
| Prometheus server | API server pods, worker pods | 9090 | Metrics |
| Prometheus server | Kubernetes API | 443 or 6443 | Pod discovery |
| API server, workers | Model servers | Server port | Embedding and LLM requests |
| Every pod | Cluster DNS | 53 | Name resolution |
The installation Job pods carry the label app.kubernetes.io/name: api-server-job, not the labels of the API server, so the policy foundation4ai-jobs-egress names the Jobs.
Policy set
The policy set uses the following placeholders:
| Placeholder | Value |
|---|---|
<ingress-namespace> | Namespace of the ingress controller or the Gateway data plane, such as ingress-nginx, or kube-system for the Traefik controller of k3s |
<model-server-cidr> | Address range of the embedding and LLM servers, with the server port in place of 443 where the servers listen on another port |
<kubernetes-api-address> | Address of the Kubernetes API from step 2 of the procedure, with the port in place of 6443 where the API listens on another port |
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-app-egress
spec:
podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [api-server, api-server-worker]
- key: app.kubernetes.io/instance
operator: In
values: [foundation4ai]
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: postgres
app.kubernetes.io/instance: foundation4ai-core
ports:
- protocol: TCP
port: 5432
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: redis
app.kubernetes.io/instance: foundation4ai-core
ports:
- protocol: TCP
port: 6379
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
ports:
- protocol: TCP
port: 4222
- to:
- ipBlock:
cidr: <model-server-cidr>
ports:
- protocol: TCP
port: 443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-jobs-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-server-job
app.kubernetes.io/instance: foundation4ai
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: postgres
app.kubernetes.io/instance: foundation4ai-core
ports:
- protocol: TCP
port: 5432
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: redis
app.kubernetes.io/instance: foundation4ai-core
ports:
- protocol: TCP
port: 6379
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-api-server-ingress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-server
app.kubernetes.io/instance: foundation4ai
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: <ingress-namespace>
ports:
- protocol: TCP
port: 8000
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/component: server
ports:
- protocol: TCP
port: 9090
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-worker-ingress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-server-worker
app.kubernetes.io/instance: foundation4ai
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/component: server
ports:
- protocol: TCP
port: 9090
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-dashboard-ingress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: dashboard
app.kubernetes.io/instance: foundation4ai
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: <ingress-namespace>
ports:
- protocol: TCP
port: 3000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-core-nats
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [api-server, api-server-worker]
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats-box
ports:
- protocol: TCP
port: 4222
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
ports:
- protocol: TCP
port: 6222
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
ports:
- protocol: TCP
port: 6222
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-core-nats-box-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats-box
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: nats
app.kubernetes.io/component: nats
ports:
- protocol: TCP
port: 4222
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-core-data-ingress
spec:
podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [redis, postgres]
- key: app.kubernetes.io/instance
operator: In
values: [foundation4ai-core]
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [api-server, api-server-worker, api-server-job]
ports:
- protocol: TCP
port: 6379
- protocol: TCP
port: 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foundation4ai-core-prometheus-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/component: server
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: [api-server, api-server-worker]
ports:
- protocol: TCP
port: 9090
- to:
- ipBlock:
cidr: <kubernetes-api-address>/32
ports:
- protocol: TCP
port: 6443
The policies apply as follows:
-
Show the labels that the policies select:
kubectl get pods -n foundation4ai \-L app.kubernetes.io/name,app.kubernetes.io/instance,app.kubernetes.io/componentExpected result: the API server, worker and dashboard pods carry the names
api-server,api-server-workeranddashboardwith the instancefoundation4ai, and a running or failed installation Job pod carries the nameapi-server-job. The NATS pods carry the namenatswith the componentsnatsandnats-box, and the Valkey, Prometheus server and, in the evaluation profile, PostgreSQL pods carry the namesredis,prometheusandpostgres, all with the instancefoundation4ai-core. -
Read the address and port of the Kubernetes API, which the Prometheus server contacts for pod discovery:
kubectl get endpoints kubernetes -n default \-o jsonpath='{.subsets[0].addresses[0].ip}{" "}{.subsets[0].ports[0].port}{"\n"}'Expected result: one address and one port, such as
10.0.0.1 6443. A control plane with several addresses needs oneipBlockentry per address. -
Save the policy set as
foundation4ai-network-policies.yaml, replace the placeholders and apply the file:kubectl apply -n foundation4ai -f foundation4ai-network-policies.yamlExpected result: one
networkpolicy.networking.k8s.io/<name> createdline for each of the 11 policies. -
Check the pods, NATS JetStream and the API:
kubectl get pods -n foundation4aiNATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \nats --user foundation4ai --password "$NATS_PASSWORD" server check connectionExpected result: every pod stays
Runningwith an unchanged ready count, and the NATS check reportsOK. The request of step 3 of the ingress procedure prints201, and a document submitted as in Ingest documents reliably reaches the statussuccess.
The following points apply to the policy set:
- External PostgreSQL. The production profile replaces the
postgresdestination offoundation4ai-app-egressandfoundation4ai-jobs-egresswith anipBlockentry for the database address range on port 5432. - Installation Jobs. The migration Job and the license check Job connect to PostgreSQL only, and the master key Job to PostgreSQL and the Redis-compatible cache. No installation Job connects to NATS JetStream or to a model server.
- Health probes. The kubelet runs the HTTP and gRPC probes from the node. Most network plugins let node traffic reach local pods regardless of policy. A plugin that applies policies to node traffic needs an ingress rule for the node addresses, and the ready counts in step 4 show whether the probes pass.
- Port forwarding.
kubectl port-forwardconnects inside the network namespace of the pod, so the port forwards in API and dashboard access are expected to work under the policies (inferred). - Diagnostic pods. A temporary pod started in the namespace, such as a PostgreSQL client, matches no allow rule and reaches only DNS. An operator adds a policy for such a pod or runs the check from outside the namespace.
- Internet access. The set allows no connection to the internet. FastEmbed models whose files are not in the image cannot download, as described in Models, packages and air-gapped installs.
Pod security
The api-server values set a security context for the API server, worker and installation Job pods. The pods run as the user and group 10001 with runAsNonRoot, fsGroup 10001 and the RuntimeDefault seccomp profile, and every container runs without privilege escalation and with all capabilities dropped. These defaults meet the restricted Pod Security Standard. The root file system stays writable, because the gRPC service writes its Python environment into the container at start. An API server or gRPC service image that runs as user 0 does not start with these defaults.
api-server.securityContext applies to every container of these pods, and api-server.server.securityContext, api-server.worker.securityContext, api-server.grpc.securityContext and api-server.jobs.securityContext replace the shared value for one kind of container.
The dashboard values set no security context: podSecurityContext and securityContext are empty in the dashboard values, and the dashboard image runs as user 1001. The core charts set security contexts for Valkey, which runs as user 1000 with a read-only root file system and no capabilities, and for the bundled PostgreSQL and the Prometheus server, which run as users 999 and 65534 with runAsNonRoot. The NATS chart sets no security context.
The following values give the dashboard the restrictions of the API server pods:
dashboard:
podSecurityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
The Pod Security Admission labels of the namespace enforce a Pod Security Standard. The restricted standard rejects the NATS JetStream pods of the core release, which carry no security context, and the dashboard pods without the values above, so a namespace that holds both releases enforces baseline and reports restricted violations as warnings:
-
Preview the effect on the running pods:
kubectl label --dry-run=server --overwrite namespace foundation4ai \pod-security.kubernetes.io/enforce=baselineExpected result:
namespace/foundation4ai labeled (server dry run), with no warning about existing pods. -
Apply the labels:
kubectl label --overwrite namespace foundation4ai \pod-security.kubernetes.io/enforce=baseline pod-security.kubernetes.io/warn=restrictedExpected result:
namespace/foundation4ai labeled. Later Helm upgrades printrestrictedwarnings for the pods that do not meet the standard and still succeed.
Service account tokens
The Foundation4 processes make no calls to the Kubernetes API, so the Foundation4 pods need no service account token. The API server and dashboard charts create service accounts that mount the token by default, and the worker pods use the service account of the API server chart. The installation Jobs and the NATS JetStream pods use the default service account of the namespace. The following values and patch stop the token mounts:
api-server:
serviceAccount:
automount: false
dashboard:
serviceAccount:
automount: false
kubectl patch serviceaccount default -n foundation4ai -p '{"automountServiceAccountToken": false}'
Expected result: serviceaccount/default patched. The setting applies to pods created after the change. After the next restart, kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ls /var/run/secrets/kubernetes.io/serviceaccount reports that the directory does not exist. The Prometheus server keeps the token of the Prometheus service account, which pod discovery needs.
Images
- Tags. The charts provide no default image tags: the values file names the tags in
global.serverImageTag,global.grpcImageTagandglobal.dashboardImageTag, or in the image values of each chart, and the rendering of the chart fails without them, as listed in Values reference. A production deployment uses tags that the registry never reassigns. - Digests. The charts join the repository and the tag with a colon, so a tag of the form
<tag>@sha256:<digest>also pins the image digest (inferred). - Registry. Production clusters pull from an internal registry or mirror, as described in Charts, images and installation bundle.
global.foundation4aiImageRegistry, such asregistry.example.com/foundation4ai, replaces the registry host of the API server, gRPC service and dashboard images.
Availability protection
The charts create no PodDisruptionBudget and no priority class for the API server, the workers or the dashboard by default. The NATS chart creates the budget foundation4ai-core-nats, which allows one of the three NATS servers to be unavailable at a time. The Prometheus budget is disabled by default, and Valkey runs as one replica.
The podDisruptionBudget values of the application release create budgets that keep API server, worker and dashboard capacity during node drains and cluster upgrades. Each value is a budget specification without a selector, and the chart adds the selector:
api-server:
podDisruptionBudget:
server:
maxUnavailable: 1
worker:
maxUnavailable: 1
dashboard:
podDisruptionBudget:
maxUnavailable: 1
helm upgrade foundation4ai ./charts/foundation4ai \
-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
kubectl get pdb -n foundation4ai
Expected result: Helm reports STATUS: deployed, and foundation4ai-api-server, foundation4ai-api-server-worker, foundation4ai-dashboard and foundation4ai-core-nats are listed, each with ALLOWED DISRUPTIONS of 1 when every pod is ready.
- Job pods. The installation Job pods carry the label
app.kubernetes.io/name: api-server-job, so the API server budget covers only the API server pods. - Replica count. A budget keeps the API available only with 2 or more API server replicas. With
api-server.replicaCount.serverat the default of 1, a drain stops the API server until the pod is scheduled again. - Queue replicas. Foundation4 creates the
DOCUMENTSstream and the stream of thedocumentsobject store with the number of copies inglobal.natsReplicas, 3 by default, so the queue and the stored document text remain available while one of the three NATS servers is unavailable. At startup, the API server and the workers change the replica count of an existing stream to the configured value and logChanging the replica count of a NATS streamatINFO. The value does not exceed the number of NATS servers, so a single external NATS server needsglobal.natsReplicas: 1.nats stream info DOCUMENTS, described in Operator command reference, shows the replica count. - Worker shutdown. On the termination signal, a worker completes the batch in progress and exits. A batch that is still running at the end of the termination grace period, 30 seconds by default, returns to the queue and is processed again.
A priority class protects the Foundation4 pods from preemption by less important workloads when nodes run short of resources. The following class is saved as foundation4ai-priority-class.yaml:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: foundation4ai
value: 100000
globalDefault: false
description: Foundation4 services
kubectl apply -f foundation4ai-priority-class.yaml
Expected result: priorityclass.scheduling.k8s.io/foundation4ai created.
The core charts accept the class in the values of the core release. The NATS chart takes the class through a pod template merge, which has not been tested:
redis:
priorityClassName: foundation4ai
postgres:
priorityClassName: foundation4ai
prometheus:
server:
priorityClassName: foundation4ai
nats:
podTemplate:
merge:
spec:
priorityClassName: foundation4ai
The application release takes the class in api-server.priorityClassName, for the API server and worker pods, and in dashboard.priorityClassName:
api-server:
priorityClassName: foundation4ai
dashboard:
priorityClassName: foundation4ai
After the upgrade of the application release, the following command shows the class of each Deployment:
kubectl get deployments -n foundation4ai \
-o custom-columns=NAME:.metadata.name,PRIORITY:.spec.template.spec.priorityClassName
Expected result: foundation4ai in the PRIORITY column of each Foundation4 Deployment.
API keys
Foundation4 authenticates every request with an API key and checks the permissions of the key on each object. A production deployment applies the following rules, which Manage API keys and permissions describes step by step:
- One key per application. Each client application or service holds a separate key, so that one key can be replaced or deleted without affecting other applications.
- Object permissions. Each key holds permissions on the individual objects that the application uses, at the lowest level that each operation needs, as listed in Access control.
- Master key. The master key serves only for administration and does not belong in client application configuration.
- Expiration and rotation. Each key carries an expiration date and is replaced on a schedule, as described in the key rotation section of the guide.
- Review. The operator reviews the key list and the permissions of each key on a schedule.
- Dashboard sign-in. The dashboard keeps the API key and the secret in the session storage of one browser tab, so a new tab or a new browser session asks for the key again, and closing the tab removes the key from the browser.
Secret storage
The Kubernetes Secrets in the foundation4ai namespace and the file foundation4ai.secrets.env hold the database and cache passwords, the application secret, the master key and the license. Secrets and keys lists each Secret and the handling rules for command output and support requests. The following controls apply to the storage:
- Read access. Role-based access control (RBAC) grants
get,listandwatchon Secrets, andcreateonpods/exec, in the namespace only to the operators of the installation.kubectl auth can-i get secrets -n foundation4ai --as <user>andkubectl auth can-i create pods --subresource=exec -n foundation4ai --as <user>printnofor every other user. - Encryption at rest. The cluster encrypts Secrets at rest, through the encryption configuration of the Kubernetes API server or the key management service of the platform.
- Secrets file.
foundation4ai.secrets.envis kept in the organization's secret store, and working copies are deleted after use. - Recovery copies. The secret store also holds the application secret, the master key identifier and secret, and the license outside the cluster, because a database restore needs the same application secret and master key, as described in Backup, restore and upgrades.
- Helm values. Credentials stay out of values files and
--setarguments, because Helm stores the values in the release history, andhelm get valuesprints the values.
Logging
The chart sets app.log_level to warn for the API server and the workers, so the processes log warnings and errors only. The sample values file supplied with the deployment files keeps warn and shows, as a comment, a configs file that sets debug for a diagnosis. The following command shows every log level that the values set:
helm get values foundation4ai -n foundation4ai | grep -n log_level
Expected result: no output, or only lines with warn.
Lower levels add, at info, a line for each refused request and each processed document and, at debug and trace, a line for every request and lines for database statements. Each lower level increases the log volume, so a lower level is set only for a limited diagnosis and removed afterwards. Configuration reference describes how a configs file sets the level. Logs are operational records, so access to the log store is limited to the operators of the installation.
Model servers
The API server pods call LLM servers and embedding servers, and the worker pods call embedding servers, over HTTP. The following rules keep these connections inside the organization's network:
- Internal endpoints. Model servers run inside the organization's network, and the
endpointof each embedding model and LLM names an internal address. The API server and the gRPC service accept private addresses with the defaulttrueofapi-server.allowPrivateEndpoints, as described in LLMs. Providers and models lists the endpoint parameters. - Restricted listeners. Each model server accepts connections only from the cluster's pod or node addresses, and the egress rule of the policy set names only the model server addresses.
- Encrypted endpoints. An endpoint on a network shared with other systems uses an
https://URL, because requests to model servers carry document text and prompts.