Skip to main content

Security hardening

This page describes the hardening measures that the charts and the code support in a production deployment: TLS on client and database connections, network policies, pod security settings, availability protection, least-privilege API keys, secret storage, logging and access to model servers. Operators and security reviewers use this page when they prepare a production installation and when they review one. Security at a glance summarizes how Foundation4 protects stored content.

Hardening measures​

MeasureConfigured inChart default
TLS for client connectionsingress.tls, or the Gateway listenerNo TLS
Restriction of paths without an API keyIngress controller or GatewayNone
TLS to PostgreSQLPOSTGRES_URL and database.ca_certBundled PostgreSQL: sslmode=disable
Network policiesManifests applied by the operatorNone
Pod security contextapi-server and dashboard valuesEmpty
Service account tokensserviceAccount.automount values and the default service accountMounted
Disruption budgets and priority classesManifests and patches applied by the operatorNATS JetStream budget only
API keysFoundation4 APIMaster key only
Log levelapp.log_levelwarn

Client TLS​

The API server and the dashboard serve plain HTTP inside the cluster. The Service foundation4ai-api-server listens on port 80 and forwards to port 8000, and the Service foundation4ai-dashboard listens on port 80 and forwards to port 3000. TLS for client applications and browsers terminates at the ingress controller or at the Gateway.

Ingress​

  1. Create the TLS Secret from the certificate and key issued for the host name:

    kubectl create secret tls foundation4ai-tls -n foundation4ai --cert=tls.crt --key=tls.key

    Expected result: secret/foundation4ai-tls created. A cluster with cert-manager can instead set an issuer annotation, such as cert-manager.io/cluster-issuer: <issuer-name>, in ingress.annotations, and cert-manager creates the Secret.

  2. Set the host name and the TLS Secret in foundation4ai.values.yaml, then upgrade the application release:

    ingress:
      enabled: true
      className: <ingress-class>
      hosts:
      - host: foundation4.example.com
      tls:
      - secretName: foundation4ai-tls
        hosts:
        - foundation4.example.com
    helm upgrade foundation4ai ./charts/foundation4ai \
      -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
    kubectl get ingress -n foundation4ai

    Expected result: Helm reports STATUS: deployed, and the Ingress foundation4ai lists the host name with ports 80, 443.

  3. Check the certificate and an authenticated request through the host name:

    openssl s_client -connect foundation4.example.com:443 -servername foundation4.example.com </dev/null 2>/dev/null \
      | openssl x509 -noout -subject -enddate
    curl -sS -o /dev/null -w '%{http_code}\n' -X POST https://foundation4.example.com/login \
      -H "x-api-key: $FOUNDATION4_API_KEY" \
      -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

    Expected result: the certificate subject names the host, notAfter shows the expiry date, and the request prints 201.

Gateway​

With httpRoute.enabled set to true, the chart attaches an HTTPRoute to the Gateway listener named in httpRoute.parentRefs. The chart default names the listener http of a Gateway called gateway, so a deployment with TLS names an HTTPS listener instead. The Gateway, the listener and the certificate Secret are created outside the charts:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: foundation4ai
  namespace: foundation4ai
spec:
  gatewayClassName: <gateway-class>
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    hostname: foundation4.example.com
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: foundation4ai-tls
httpRoute:
  enabled: true
  parentRefs:
  - name: foundation4ai
    sectionName: https
  hostnames:
  - foundation4.example.com

A Gateway in another namespace adds namespace to the parent reference and allows routes from the foundation4ai namespace in the listener's allowedRoutes. Through the chart's HTTPRoute, GET / redirects to /dashboard, so checks of the API through a Gateway use POST /login, as in step 3 of the ingress procedure. API and dashboard access describes both routes.

Paths without an API key​

The API server answers the root route /, the health check /healthz, the monitoring routes /metrics and /statistics and the API documentation routes without an API key, as API conventions lists. The Ingress and the HTTPRoute of the chart forward every path outside /dashboard to the API server, so these routes are reachable wherever the API is exposed. The following rules apply to a production deployment:

  • Monitoring routes. /metrics and /statistics are restricted at the ingress controller or the Gateway to the networks of the monitoring system and the operators, for example with a second Ingress for the same host that lists the two paths and the source address rule of the ingress controller. The dashboard does not call either route.
  • Collection and probes. The bundled Prometheus server scrapes the pods inside the cluster, and the kubelet sends the probes to the pods directly, so a restriction at the ingress affects neither metric collection nor the probes.
  • API documentation routes. The documentation routes are restricted in the same way where the security policy of the deployment requires a restriction.

PostgreSQL TLS​

Foundation4 connects to an external PostgreSQL database with the URL in POSTGRES_URL of foundation4ai.secrets.env. The sslmode parameter of the URL sets encryption and certificate checks. The configuration key database.ca_cert supplies the certificate authority (CA) certificate in PEM format. The API server, the workers and the installation Jobs read both settings.

sslmodeEncryptionCertificate check
disableNoneNone
prefer, the default when the parameter is absentWhen the server offers TLSNone
requireAlwaysNone, even when database.ca_cert is set
verify-caAlwaysCertificate signed by the CA
verify-fullAlwaysCertificate signed by the CA and issued for the host name in the URL

A production deployment uses verify-full:

POSTGRES_URL=postgres://<user>:<password>@<database-host>:5432/<database>?sslmode=verify-full

The CA certificate goes in a configuration file under api-server.configs. The file name f4ai-10-database-tls.yaml sorts after the chart defaults and before the credential files that the chart generates, as the load order in Configuration reference describes:

api-server:
  configs:
    f4ai-10-database-tls.yaml: |
      database:
        ca_cert: |
          -----BEGIN CERTIFICATE-----
          <CA certificate lines>
          -----END CERTIFICATE-----

The change takes effect through the change procedure in Secrets and keys, which ends with a restart of the API server and the workers. A database session with administrative rights then confirms that every Foundation4 connection uses TLS:

psql "<administrator-connection-url>" -c \
  "SELECT a.usename, s.ssl, s.version FROM pg_stat_ssl s JOIN pg_stat_activity a USING (pid) WHERE a.datname = '<database>';"

Expected result: every row shows ssl as t and a TLS version such as TLSv1.3. The database server can also reject unencrypted connections, with hostssl entries in pg_hba.conf or the equivalent setting of a managed PostgreSQL service. The bundled PostgreSQL of the evaluation profile connects with sslmode=disable and is not a production database, as described in Database.

Connections inside the namespace​

The core release builds redis:// and nats:// URLs, so the API server and the workers connect to the Redis-compatible cache and to NATS JetStream without TLS. The Foundation4 build supports only redis:// URLs for the cache, so an external cache is also reached without TLS. The gRPC service runs as a sidecar container, a second container in the same Kubernetes pod, next to each API server and worker process, and is reached on localhost. The network policies in the next section limit these connections to the pods that use them.

Network policies​

The charts ship no network policies for the Foundation4 pods. The Valkey and PostgreSQL charts render a policy only when redis.networkPolicy or postgres.networkPolicy is set, and the Prometheus chart only when prometheus.networkPolicy.enabled is true. None of these values is set by default, and the NATS chart has no policy template. The policy set below allows the connections that the charts and the code use and denies all other traffic to and from the pods of the namespace. The policies name container ports, not Service ports.

SourceDestinationPortPurpose
Ingress controller or GatewayAPI server pods8000API requests
Ingress controller or GatewayDashboard pods3000Dashboard
API server, workers, installation JobsPostgreSQL5432Primary data store
API server, workers, installation JobsValkey6379Cache
API server, workers, installation JobsNATS JetStream4222Queue and object store
NATS JetStreamNATS JetStream6222Routes between the 3 NATS servers
NATS utility pod foundation4ai-core-nats-boxNATS JetStream4222Operator commands
Prometheus serverAPI server pods, worker pods8000, 9090Metrics
Prometheus serverKubernetes API443 or 6443Pod discovery
API server, workersModel serversServer portEmbedding and LLM requests
Every podCluster DNS53Name resolution

The installation Jobs carry the pod labels of the API server, so the API server policies also apply to the Jobs.

Policy set​

The policy set uses the following placeholders:

PlaceholderValue
<ingress-namespace>Namespace of the ingress controller or the Gateway data plane, such as ingress-nginx, or kube-system for the Traefik controller of k3s
<model-server-cidr>Address range of the embedding and LLM servers, with the server port in place of 443 where the servers listen on another port
<kubernetes-api-address>Address of the Kubernetes API from step 2 of the procedure, with the port in place of 6443 where the API listens on another port
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foundation4ai-app-egress
spec:
  podSelector:
    matchExpressions:
    - key: app.kubernetes.io/name
      operator: In
      values: [api-server, api-server-worker]
    - key: app.kubernetes.io/instance
      operator: In
      values: [foundation4ai]
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: postgres
          app.kubernetes.io/instance: foundation4ai-core
    ports:
    - protocol: TCP
      port: 5432
  - to:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: redis
          app.kubernetes.io/instance: foundation4ai-core
    ports:
    - protocol: TCP
      port: 6379
  - to:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: nats
          app.kubernetes.io/component: nats
    ports:
    - protocol: TCP
      port: 4222
  - to:
    - ipBlock:
        cidr: <model-server-cidr>
    ports:
    - protocol: TCP
      port: 443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foundation4ai-api-server-ingress
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: api-server
      app.kubernetes.io/instance: foundation4ai
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: <ingress-namespace>
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus
          app.kubernetes.io/component: server
    ports:
    - protocol: TCP
      port: 8000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foundation4ai-worker-ingress
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: api-server-worker
      app.kubernetes.io/instance: foundation4ai
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus
          app.kubernetes.io/component: server
    ports:
    - protocol: TCP
      port: 9090
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foundation4ai-dashboard-ingress
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: dashboard
      app.kubernetes.io/instance: foundation4ai
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: <ingress-namespace>
    ports:
    - protocol: TCP
      port: 3000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foundation4ai-core-nats
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: nats
      app.kubernetes.io/component: nats
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values: [api-server, api-server-worker]
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: nats
          app.kubernetes.io/component: nats-box
    ports:
    - protocol: TCP
      port: 4222
  - from:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: nats
          app.kubernetes.io/component: nats
    ports:
    - protocol: TCP
      port: 6222
  egress:
  - to:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: nats
          app.kubernetes.io/component: nats
    ports:
    - protocol: TCP
      port: 6222
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foundation4ai-core-nats-box-egress
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: nats
      app.kubernetes.io/component: nats-box
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: nats
          app.kubernetes.io/component: nats
    ports:
    - protocol: TCP
      port: 4222
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foundation4ai-core-data-ingress
spec:
  podSelector:
    matchExpressions:
    - key: app.kubernetes.io/name
      operator: In
      values: [redis, postgres]
    - key: app.kubernetes.io/instance
      operator: In
      values: [foundation4ai-core]
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values: [api-server, api-server-worker]
    ports:
    - protocol: TCP
      port: 6379
    - protocol: TCP
      port: 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foundation4ai-core-prometheus-egress
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/component: server
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: api-server
    ports:
    - protocol: TCP
      port: 8000
  - to:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: api-server-worker
    ports:
    - protocol: TCP
      port: 9090
  - to:
    - ipBlock:
        cidr: <kubernetes-api-address>/32
    ports:
    - protocol: TCP
      port: 6443

The policies apply as follows:

  1. Show the labels that the policies select:

    kubectl get pods -n foundation4ai \
      -L app.kubernetes.io/name,app.kubernetes.io/instance,app.kubernetes.io/component

    Expected result: the API server, worker and dashboard pods carry the names api-server, api-server-worker and dashboard with the instance foundation4ai. The NATS pods carry the name nats with the components nats and nats-box, and the Valkey, Prometheus server and, in the evaluation profile, PostgreSQL pods carry the names redis, prometheus and postgres, all with the instance foundation4ai-core.

  2. Read the address and port of the Kubernetes API, which the Prometheus server contacts for pod discovery:

    kubectl get endpoints kubernetes -n default \
      -o jsonpath='{.subsets[0].addresses[0].ip}{" "}{.subsets[0].ports[0].port}{"\n"}'

    Expected result: one address and one port, such as 10.0.0.1 6443. A control plane with several addresses needs one ipBlock entry per address.

  3. Save the policy set as foundation4ai-network-policies.yaml, replace the placeholders and apply the file:

    kubectl apply -n foundation4ai -f foundation4ai-network-policies.yaml

    Expected result: one networkpolicy.networking.k8s.io/<name> created line for each of the 10 policies.

  4. Check the pods, NATS JetStream and the API:

    kubectl get pods -n foundation4ai
    kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- nats server check connection

    Expected result: every pod stays Running with an unchanged ready count, and the NATS check reports OK. The request of step 3 of the ingress procedure prints 201, and a document submitted as in Ingest documents reliably reaches the status success.

The following points apply to the policy set:

  • External PostgreSQL. The production profile replaces the postgres destination of foundation4ai-app-egress with an ipBlock entry for the database address range on port 5432.
  • Health probes. The kubelet runs the HTTP and gRPC probes from the node. Most network plugins let node traffic reach local pods regardless of policy. A plugin that applies policies to node traffic needs an ingress rule for the node addresses, and the ready counts in step 4 show whether the probes pass.
  • Port forwarding. kubectl port-forward connects inside the network namespace of the pod, so the port forwards in API and dashboard access are expected to work under the policies (inferred).
  • Diagnostic pods. A temporary pod started in the namespace, such as a PostgreSQL client, matches no allow rule and reaches only DNS. An operator adds a policy for such a pod or runs the check from outside the namespace.
  • Internet access. The set allows no connection to the internet. FastEmbed models whose files are not in the image cannot download, as described in Models, packages and air-gapped installs.

Pod security​

The charts set no security context for the Foundation4 pods: podSecurityContext and securityContext are empty in the api-server and dashboard values. The API server and gRPC service images declare no user and run as user 0, and the dashboard image runs as user 1001. The installation Jobs take no security context from the values. The core charts set security contexts: Valkey runs as user 1000 with a read-only root file system and no capabilities, and the bundled PostgreSQL and the Prometheus server run as users 999 and 65534 with runAsNonRoot.

The following values restrict privilege escalation and system calls without changing the user of the Foundation4 containers:

api-server:
  podSecurityContext:
    seccompProfile:
      type: RuntimeDefault
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
dashboard:
  podSecurityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL

api-server.securityContext applies to the server, worker and grpc containers alike. A non-root user or a read-only root file system for these containers has not been tested with the published images.

The Pod Security Admission labels of the namespace enforce a Pod Security Standard. The restricted standard rejects the installation Jobs and the API server pods, which run as user 0, so the namespace enforces baseline and reports restricted violations as warnings:

  1. Preview the effect on the running pods:

    kubectl label --dry-run=server --overwrite namespace foundation4ai \
      pod-security.kubernetes.io/enforce=baseline

    Expected result: namespace/foundation4ai labeled (server dry run), with no warning about existing pods.

  2. Apply the labels:

    kubectl label --overwrite namespace foundation4ai \
      pod-security.kubernetes.io/enforce=baseline pod-security.kubernetes.io/warn=restricted

    Expected result: namespace/foundation4ai labeled. Later Helm upgrades print restricted warnings for the Foundation4 pods and still succeed.

Service account tokens​

The Foundation4 processes make no calls to the Kubernetes API, so the Foundation4 pods need no service account token. The API server and dashboard charts create service accounts that mount the token by default, and the worker pods, the installation Jobs and the NATS JetStream pods use the default service account of the namespace. The following values and patch stop the token mounts:

api-server:
  serviceAccount:
    automount: false
dashboard:
  serviceAccount:
    automount: false
kubectl patch serviceaccount default -n foundation4ai -p '{"automountServiceAccountToken": false}'

Expected result: serviceaccount/default patched. The setting applies to pods created after the change. After the next restart, kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ls /var/run/secrets/kubernetes.io/serviceaccount reports that the directory does not exist. The Prometheus server keeps the token of the Prometheus service account, which pod discovery needs.

Images​

  • Tags. The charts provide no default image tags, so each values file names the tags, as listed in Values reference. A production deployment uses tags that the registry never reassigns.
  • Digests. The charts join the repository and the tag with a colon, so a tag of the form <tag>@sha256:<digest> also pins the image digest (inferred).
  • Registry. Production clusters pull from an internal registry or mirror, as described in Charts, images and installation bundle.

Availability protection​

The charts create no PodDisruptionBudget and no priority class for the API server, the workers or the dashboard. The NATS chart creates the budget foundation4ai-core-nats, which allows one of the three NATS servers to be unavailable at a time. The Prometheus budget is disabled by default, and Valkey runs as one replica.

The following budgets, saved as foundation4ai-disruption-budgets.yaml, keep API server and worker capacity during node drains and cluster upgrades:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: foundation4ai-api-server
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: api-server
      app.kubernetes.io/instance: foundation4ai
      app.kubernetes.io/managed-by: Helm
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: foundation4ai-api-server-worker
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: api-server-worker
      app.kubernetes.io/instance: foundation4ai
kubectl apply -n foundation4ai -f foundation4ai-disruption-budgets.yaml
kubectl get pdb -n foundation4ai

Expected result: foundation4ai-api-server, foundation4ai-api-server-worker and foundation4ai-core-nats are listed, each with ALLOWED DISRUPTIONS of 1 when every pod is ready.

  • Job pods. The API server pods carry app.kubernetes.io/managed-by: Helm and the installation Job pods do not, so the API server budget covers only the API server pods.
  • Replica count. A budget keeps the API available only with 2 or more API server replicas. With api-server.replicaCount.server at the default of 1, a drain stops the API server until the pod is scheduled again.
  • Queue replicas. Foundation4 creates the DOCUMENTS stream and the documents object store without a replica setting, so NATS JetStream keeps one copy of each on one of the three servers (inferred). While that server is unavailable, document submission and processing stop. nats stream info DOCUMENTS, described in Operator command reference, shows the replica count, and a drain of the node that holds the stream is planned as maintenance.
  • Worker shutdown. The worker process does not handle the termination signal, so a worker pod stops at the end of the termination grace period, 30 seconds by default. The queue then delivers the documents of the interrupted batch again (inferred).

A priority class protects the Foundation4 pods from preemption by less important workloads when nodes run short of resources. The following class is saved as foundation4ai-priority-class.yaml:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: foundation4ai
value: 100000
globalDefault: false
description: Foundation4 services
kubectl apply -f foundation4ai-priority-class.yaml

Expected result: priorityclass.scheduling.k8s.io/foundation4ai created.

The core charts accept the class in the values of the core release. The NATS chart takes the class through a pod template merge, which has not been tested:

redis:
  priorityClassName: foundation4ai
postgres:
  priorityClassName: foundation4ai
prometheus:
  server:
    priorityClassName: foundation4ai
nats:
  podTemplate:
    merge:
      spec:
        priorityClassName: foundation4ai

The Foundation4 Deployments have no value for the class, so the operator patches the Deployments after each install or upgrade:

for d in foundation4ai-api-server foundation4ai-api-server-worker foundation4ai-dashboard; do
  kubectl patch deployment "$d" -n foundation4ai --type merge \
    -p '{"spec":{"template":{"spec":{"priorityClassName":"foundation4ai"}}}}'
done
kubectl get deployments -n foundation4ai \
  -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.template.spec.priorityClassName

Expected result: three deployment.apps/<name> patched lines, then foundation4ai in the PRIORITY column of each Foundation4 Deployment. Each patched Deployment replaces the pods of the Deployment.

API keys​

Foundation4 authenticates every request with an API key and checks the permissions of the key on each object. A production deployment applies the following rules, which Manage API keys and permissions describes step by step:

  • One key per application. Each client application or service holds a separate key, so that one key can be replaced or deleted without affecting other applications.
  • Object permissions. Each key holds permissions on the individual objects that the application uses, at the lowest level that each operation needs, as listed in Access control.
  • Master key. The master key serves only for administration and does not belong in client application configuration.
  • Expiration and rotation. Each key carries an expiration date and is replaced on a schedule, as described in the key rotation section of the guide.
  • Review. The operator reviews the key list and the permissions of each key on a schedule.

Secret storage​

The Kubernetes Secrets in the foundation4ai namespace and the file foundation4ai.secrets.env hold the database and cache passwords, the application secret, the master key and the license. Secrets and keys lists each Secret and the handling rules for command output and support requests. The following controls apply to the storage:

  • Read access. Role-based access control (RBAC) grants get, list and watch on Secrets, and create on pods/exec, in the namespace only to the operators of the installation. kubectl auth can-i get secrets -n foundation4ai --as <user> and kubectl auth can-i create pods --subresource=exec -n foundation4ai --as <user> print no for every other user.
  • Encryption at rest. The cluster encrypts Secrets at rest, through the encryption configuration of the Kubernetes API server or the key management service of the platform.
  • Secrets file. foundation4ai.secrets.env is kept in the organization's secret store, and working copies are deleted after use.
  • Recovery copies. The secret store also holds the application secret, the master key identifier and secret, and the license outside the cluster, because a database restore needs the same application secret and master key, as described in Backup, restore and upgrades.
  • Helm values. Credentials stay out of values files and --set arguments, because Helm stores the values in the release history, and helm get values prints the values.

Logging​

The chart sets app.log_level to warn for the API server and the workers, so the processes log warnings and errors only. The sample values file supplied with the deployment files sets debug in f4ai-05-servers.yaml, and the production foundation4ai.values.yaml described in Install on Kubernetes omits that setting. The following command shows every log level that the values set:

helm get values foundation4ai -n foundation4ai | grep -n log_level

Expected result: no output, or only lines with warn.

Lower levels add a line for each processed document and, at debug and trace, lines for database statements. Each lower level increases the log volume, so a lower level is set only for a limited diagnosis and removed afterwards. Configuration reference describes how a configs file sets the level. Logs are operational records, so access to the log store is limited to the operators of the installation.

Model servers​

The API server pods call LLM servers and embedding servers, and the worker pods call embedding servers, over HTTP. The following rules keep these connections inside the organization's network:

  • Internal endpoints. Model servers run inside the organization's network, and the endpoint of each embedding model and LLM names an internal address. Providers and models lists the endpoint parameters.
  • Restricted listeners. Each model server accepts connections only from the cluster's pod or node addresses, and the egress rule of the policy set names only the model server addresses.
  • Encrypted endpoints. An endpoint on a network shared with other systems uses an https:// URL, because requests to model servers carry document text and prompts.