Operator command reference
This page lists the commands that operators run most often on a Foundation4 deployment, grouped by task, with the purpose of each command and the output of a healthy deployment. The page also describes every subcommand of the foundation4ai command-line interface (CLI) in the API server image, and marks the subcommands that the installation Jobs run. Operators use this page during installation, routine checks and Troubleshooting. The commands use the namespace foundation4ai and the releases foundation4ai-core and foundation4ai of Install on Kubernetes.
Pods and Jobs
| Command | Purpose | Expected output |
|---|---|---|
kubectl get pods -n foundation4ai -o wide | State, restarts and node of every pod | Every pod Running. API server and worker pods 2/2, dashboard pod 1/1, no rising RESTARTS count |
kubectl get deployments,statefulsets -n foundation4ai | Ready replicas of each workload | READY equals the desired count for every workload, such as 3/3 for foundation4ai-api-server-worker and the StatefulSet foundation4ai-core-nats |
kubectl get jobs -n foundation4ai | State of the installation Jobs | No resources found in foundation4ai namespace. after a completed installation or upgrade. A listed foundation4ai-api-server-db-migration, foundation4ai-api-server-license-check or foundation4ai-api-server-create-admin-api-key Job is running or has failed |
kubectl get pvc -n foundation4ai | State of the volume claims | 3 NATS JetStream claims, 1 Prometheus claim and, in the evaluation profile, 1 PostgreSQL claim, all Bound |
kubectl describe pod -n foundation4ai <pod-name> | Events, container states and the reason of the last termination | No Failed or BackOff events, Last State empty or Completed |
kubectl get events -n foundation4ai --sort-by=.lastTimestamp | Recent events of the namespace, oldest first | Normal events only |
kubectl top pods -n foundation4ai --containers | CPU and memory of each container; needs the metrics server | Memory of each container below the limit |
kubectl rollout restart -n foundation4ai deployment/<name> | Restarts the pods of a Deployment, for example after a configuration change | deployment.apps/<name> restarted |
kubectl rollout status -n foundation4ai deployment/<name> | Waits until the restarted pods are ready | deployment "<name>" successfully rolled out |
Helm deletes each installation Job after the Job succeeds. A failed Job and its log remain until the next installation or upgrade of the application release replaces the Job.
Logs
| Command | Purpose | Expected output |
|---|---|---|
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c server | API server log | At the default level warn, no lines while the API server is healthy other than the two OpenTelemetry warnings at each start, described in Monitoring and logging, and the Waiting for the database, cache or queue retries while PostgreSQL, the cache or NATS JetStream is not yet reachable |
kubectl logs -n foundation4ai deploy/foundation4ai-api-server-worker -c worker | Log of one worker | At the level warn, no lines while processing succeeds other than the same start-up warnings as the API server |
kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=200 --prefix | Logs of every worker, each line prefixed with the pod name | No Error processing documents or Failed processing document lines |
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c grpc | gRPC service log of an API server pod | INFO:processor:gRPC server started on 127.0.0.1:50051 |
kubectl logs -n foundation4ai <pod-name> -c <container> --previous | Log of the previous container after a restart | The error that stopped the container, such as Error: "<message>" |
A command with deploy/<name> reads the log of one pod of the Deployment. Monitoring and logging describes the log formats and levels.
Installation Job logs
| Command | Purpose | Expected output |
|---|---|---|
kubectl logs -n foundation4ai job/foundation4ai-api-server-db-migration | Result of the database migration | Migrated to latest schema. |
kubectl logs -n foundation4ai job/foundation4ai-api-server-license-check | System ID and license check result | SystemID: <system ID>, then Checking for a valid license... OK |
kubectl logs -n foundation4ai job/foundation4ai-api-server-create-admin-api-key | Creation or check of the master key | Master key created successfully., Master key secret updated to match the configuration. or Master key already matches the configuration. |
kubectl get pods -n foundation4ai -l job-name=<job-name> | Pods of one Job, including failed attempts | One pod per attempt; the pods of failed attempts show Error |
Helm deletes each Job after the Job succeeds, so these commands read a Job while the Job runs, or after a failure until the next installation or upgrade of the application release. kubectl logs job/<name> reads one pod of the Job; the pod list shows every attempt, and kubectl logs -n foundation4ai <pod-name> reads a specific attempt. The license check Job makes one attempt.
Helm releases
| Command | Purpose | Expected output |
|---|---|---|
helm list -n foundation4ai | Releases, revisions and chart versions | foundation4ai-core and foundation4ai, both with STATUS deployed |
helm status foundation4ai -n foundation4ai | State of the latest revision of the application release | STATUS: deployed and the revision number |
helm history foundation4ai -n foundation4ai | Revisions with status and description | The latest revision deployed, earlier revisions superseded. A failed revision describes the failed hook or resource. |
helm get values foundation4ai -n foundation4ai | Values supplied to the release | The content of the values file, without credentials |
The same commands with foundation4ai-core show the core release. helm get hooks and helm get all print the rendered Secrets, so the output of these commands is handled as described in Secrets and keys. helm rollback does not reverse database migrations: migrations only move forward, and an older image stops at startup against a database that a newer version has migrated. A rollback after an upgrade that migrated the database therefore needs the database backup taken before the upgrade, as described in Backup, restore and upgrades.
NATS JetStream
The core release deploys the utility pod foundation4ai-core-nats-box, which contains the nats command-line tool. NATS JetStream accepts only authenticated clients: the user is foundation4ai, and the password is the NATS_PASSWORD value of the Secret foundation4ai-secrets. The following commands read the password and run a subcommand in that pod:
NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \
nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" <subcommand>
| Subcommand | Purpose | Expected output |
|---|---|---|
nats server check connection | Connection from the namespace to NATS | A line that begins with OK Connection |
nats account info | JetStream availability and storage use of the account | JetStream account information with the storage in use below the limit |
nats stream ls | Streams of the account | DOCUMENTS and OBJ_documents, the stream of the documents object store |
nats stream info DOCUMENTS | Configuration and state of the processing queue | Subjects: document.>, Retention: WorkQueue, Maximum Age: 1d0h0m0s and Replicas: 3, the value of global.natsReplicas. Under the state, Messages is the number of waiting jobs. |
nats consumer info DOCUMENTS document-processor | State of the consumer that the workers share | An Ack Wait of 10 minutes and Max Ack Pending: 1,000. The state shows the outstanding acknowledgements, the redelivered messages and the unprocessed messages. |
The commands above read state only. nats stream purge, nats stream rm and nats consumer rm delete queued jobs, which leaves the affected documents pending, as described in Troubleshooting.
API checks
-
Forward a local port to the API server Service in a separate terminal:
kubectl port-forward -n foundation4ai svc/foundation4ai-api-server 8080:80Expected result:
Forwarding from 127.0.0.1:8080 -> 8000. -
Check an API key with the shared variables of Authenticate:
export FOUNDATION4_URL=http://localhost:8080curl -X POST "$FOUNDATION4_URL/login" \-H "x-api-key: $FOUNDATION4_API_KEY" \-H "x-api-key-secret: $FOUNDATION4_API_SECRET"Expected result: status 201 and a body whose
messageisOKand whosekeyis the name of the key, followed by the permissions of the key. -
Check the components that the API server uses:
curl -s -w "\n%{http_code}\n" "$FOUNDATION4_URL/healthz"Expected result:
{"status":"ok","database":"ok","message_queue":"ok","cache":"ok","llm_service":"ok"}, then200. A failing PostgreSQL, NATS JetStream, cache or gRPC service returns status 503 with"status":"error"and the value"error"for that component, as described in Monitoring and logging.POST /loginin step 2 is the authenticated check.
The worker metrics are read through a port forward to one worker pod:
kubectl port-forward -n foundation4ai deploy/foundation4ai-api-server-worker 9090:9090
curl -s http://localhost:9090/metrics | grep foundation4ai_document_queue
Expected result: a # HELP line, a # TYPE line and the values of each metric, such as foundation4ai_document_queue_processed 1200. The API server metrics are on the metrics port 9090 of the API server pod, which no Service exposes:
kubectl port-forward -n foundation4ai deploy/foundation4ai-api-server 9091:9090
curl -s http://localhost:9091/metrics | grep -E "^(axum_http_requests_total|foundation4ai_)"
Expected result: request counts per method, status and endpoint, and the document and agent execution metrics. Monitoring and logging describes each metric.
Configuration and database checks
| Command | Purpose | Expected output |
|---|---|---|
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ls /app/config | Configuration files of the API server, in load order | f4ai-00-defaults.yaml, the files of api-server.configs and api-server.secrets, and the directory secrets, which holds one file per connection URL and credential, as described in Configuration reference |
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai database check-connection | Connection to PostgreSQL with the URL and CA certificate of the configuration | Ok. |
database check-connection reports a failure as Error: "Failed connecting to database: <reason>", where the reason names the failure, such as a host name that cannot be resolved, a rejected password, a certificate error or a rule of pg_hba.conf. A PostgreSQL client in a temporary pod tests the same connection outside the API server image. The pod reads POSTGRES_URL from foundation4ai-secrets, so the check applies to the production profile with an external database; <postgresql-client-image> is any image that contains psql, taken from the registry that the cluster uses:
kubectl run pg-check -n foundation4ai --rm -i --restart=Never --image=<postgresql-client-image> \
--overrides='{"apiVersion":"v1","spec":{"containers":[{"name":"pg-check","image":"<postgresql-client-image>","command":["sh","-c","psql \"$POSTGRES_URL\" -c \"SELECT version();\""],"envFrom":[{"secretRef":{"name":"foundation4ai-secrets"}}]}]}}'
Expected result: one row with the PostgreSQL version, then pod "pg-check" deleted. A failed connection prints the reason, such as a refused connection, a failed password or a certificate error. With sslmode=verify-full, psql also needs the CA certificate through the sslrootcert parameter, which the pod does not have. Under the network policies of Security hardening, the temporary pod reaches only DNS unless a policy allows the connection.
Foundation4 CLI
Command-line interface is the full reference for the subcommands, the connections and the exit status. The API server image contains the foundation4ai binary in /app, the working directory of the containers. Every subcommand reads the configuration from ./config, which is /app/config in the server and worker containers and in the installation Jobs. Operators run subcommands in the server container of a running API server pod:
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai <subcommand>
A subcommand that fails prints Error: "<message>" and exits with status 1, and an invalid command line exits with status 2. license check and license details exit with status 1 when the license cannot be used, so scripts read the exit status. The admin subcommands connect to PostgreSQL and the Redis-compatible cache only, and admin keys-without-classifications to PostgreSQL only. data check-models builds the same connection as the API server and fails in the same ways, as listed in Troubleshooting. Connections lists the connection of each subcommand.
| Subcommand | Arguments | Purpose | Output on success | Run by |
|---|---|---|---|---|
server | None | Runs the API server | None at the level warn | server container |
worker | None | Runs a worker with a database pool of 1 connection | None at the level warn | worker container |
database migrate | None | Applies pending database migrations | Migrated to latest schema. | Migration Job (hook) |
database check-connection | None | Connects to PostgreSQL with the configured URL and CA certificate | Ok. | Operator |
license check | None | Prints the system ID and the result of the license check, and exits with status 1 after a failed check | SystemID: <system ID>, then Checking for a valid license... OK | License check Job (hook), operator |
license system-id | None | Prints the system ID of the database | 64 hexadecimal characters | Operator |
license details | <license> | Prints the content of a license that is valid for the system ID | License: <name> (expires on <date> <time> UTC), System ID:, Version: and Limits: lines, and optional Description: and Environment: lines | Operator |
admin create-admin-api-key | <name> [description] [expiration] | Makes the master key match app.master.key and app.master.secret: creates the key, or replaces the stored secret | Master key created successfully., Master key secret updated to match the configuration. or Master key already matches the configuration. | Master key Job (hook), with the name Master Admin Key |
admin update-admin-api-key | Options --name, --description, --clear-description, --active, --expiration, --clear-expiration | Changes the name, the description, the active flag or the expiry of the master key | Master key updated successfully. | Operator |
admin generate-admin-api-key | <name> [description] [expiration] | Creates an additional key with all permissions, a new identifier and a random 32-character secret | Administration key created successfully., then Key : <identifier> and Secret: <secret> | Operator |
admin keys-without-classifications | None | Lists the active keys whose pipeline grants name no classification, so that their searches find nothing | Every active API key that can read or search pipelines names classifications., or one line per grant | Operator |
data providers | None | Lists the providers of the embedding models and text splitters in the database | Found embedding providers: and Text splitter providers in use:, each followed by - <provider> lines | Operator |
data check-models | None | Checks that every configured embedding model and text splitter loads, as the API server and the workers load them | One line per model that ends with the way the model was checked, then Every configured model loads. | Operator |
The following points apply to the subcommands:
-
Hook subcommands.
database migrate,license checkandadmin create-admin-api-keyrun in the installation Jobs at every installation and upgrade of the application release.license checkreturns at once, so an operator also runs the command in a running API server pod to read the system ID and the result. -
License details. The license is passed as an argument. The following commands read the value from the secrets file:
LICENSE=$(grep '^FOUNDATION4AI_APP_LICENSE=' foundation4ai.secrets.env | cut -d= -f2-)kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \./foundation4ai license details "$LICENSE"Expected result: the license name with the expiry date or
(perpetual), the system ID, the version and the limits. For a license that cannot be used, the command printsError: "Expired"orError: "Invalid", for example for a license issued for another system ID, and exits with status 1. Licensing describes the license fields. -
System ID.
license system-id,license checkandlicense detailscompute the system ID for the database indatabase.urland the schema indatabase.schema, as the API server does. Licensing describes the binding. -
Master key. The master key remains active and unexpired, because the workers authenticate with the master key at startup and stop without the key, as described in Access control. The API server starts with an expired or inactive master key, logs an error and refuses requests made with that key.
admin update-admin-api-key --active trueor--clear-expirationrestores the key, as described in Master key update. -
Generated keys.
admin generate-admin-api-keyprints the new secret once. The command runs in a session whose output is not recorded, and the secret goes directly to the organization's secret store. The new key does not replace the master key. -
Model check.
data check-modelsdownloads no FastEmbed model file, so a FastEmbed model that passes the check loads without network access. The CLI has no command that downloads model files; model and package images deliver the files to every pod, as described in Models, packages and air-gapped installs. Model check describes the output and the failures. -
Keys without classifications.
admin keys-without-classificationschanges nothing. Keys without classifications describes the output, and Manage API keys and permissions describes how to grant classifications. -
Help.
./foundation4ai --helpand./foundation4ai <subcommand> --helpprint the usage of each subcommand../foundation4ai --versionprintsfoundation4ai <version>, such asfoundation4ai 1.1.0, as described in Command-line interface.