Operator command reference

This page lists the commands that operators run most often on a Foundation4 deployment, grouped by task, with the purpose of each command and the output of a healthy deployment. The page also describes every subcommand of the foundation4ai command-line interface (CLI) in the API server image, and marks the subcommands that the installation Jobs run. Operators use this page during installation, routine checks and Troubleshooting. The commands use the namespace foundation4ai and the releases foundation4ai-core and foundation4ai of Install on Kubernetes.

Pods and Jobs

Command	Purpose	Expected output
kubectl get pods -n foundation4ai -o wide	State, restarts and node of every pod	Every pod Running. API server and worker pods 2/2, dashboard pod 1/1, no rising RESTARTS count
kubectl get deployments,statefulsets -n foundation4ai	Ready replicas of each workload	READY equals the desired count for every workload, such as 3/3 for foundation4ai-api-server-worker and the StatefulSet foundation4ai-core-nats
kubectl get jobs -n foundation4ai	State of the installation Jobs	foundation4ai-api-server-db-migration, foundation4ai-api-server-license-check and foundation4ai-api-server-create-admin-api-key at 1/1 completions
kubectl get pvc -n foundation4ai	State of the volume claims	3 NATS JetStream claims and 1 Prometheus claim, all Bound
kubectl describe pod -n foundation4ai <pod-name>	Events, container states and the reason of the last termination	No Failed or BackOff events, Last State empty or Completed
kubectl get events -n foundation4ai --sort-by=.lastTimestamp	Recent events of the namespace, oldest first	Normal events only
kubectl top pods -n foundation4ai --containers	CPU and memory of each container; needs the metrics server	Memory of each container below the limit
kubectl rollout restart -n foundation4ai deployment/<name>	Restarts the pods of a Deployment, for example after a configuration change	deployment.apps/<name> restarted
kubectl rollout status -n foundation4ai deployment/<name>	Waits until the restarted pods are ready	deployment "<name>" successfully rolled out

The installation Jobs have no deletion policy. Each Job and the Job log remain after completion until the next installation or upgrade of the application release replaces the Job.

Logs

Command	Purpose	Expected output
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c server	API server log	No lines while the API server is healthy at the default level warn
kubectl logs -n foundation4ai deploy/foundation4ai-api-server-worker -c worker	Log of one worker	No lines while processing succeeds at the level warn
kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=200 --prefix	Logs of every worker, each line prefixed with the pod name	No Error processing documents or Failed processing document lines
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c grpc	gRPC service log of an API server pod	INFO:processor:gRPC server started on port 50051
kubectl logs -n foundation4ai <pod-name> -c <container> --previous	Log of the previous container after a restart	The error that stopped the container, such as Error: "<message>"

A command with deploy/<name> reads the log of one pod of the Deployment. Monitoring and logging describes the log formats and levels.

Installation Job logs

Command	Purpose	Expected output
kubectl logs -n foundation4ai job/foundation4ai-api-server-db-migration	Result of the database migration	Migrated to latest schema.
kubectl logs -n foundation4ai job/foundation4ai-api-server-license-check	System ID and license check result	SystemID: <system ID>, then Checking for a valid license... OK
kubectl logs -n foundation4ai job/foundation4ai-api-server-create-admin-api-key	Creation of the master key	Admin API key created successfully.
kubectl get pods -n foundation4ai -l job-name=<job-name>	Pods of one Job, including failed attempts	One pod Completed
kubectl delete job -n foundation4ai foundation4ai-api-server-license-check	Removes a license check that waits after a failed check	job.batch "foundation4ai-api-server-license-check" deleted

kubectl logs job/<name> reads one pod of the Job; the pod list shows every attempt, and kubectl logs -n foundation4ai <pod-name> reads a specific attempt.

Helm releases

Command	Purpose	Expected output
helm list -n foundation4ai	Releases, revisions and chart versions	foundation4ai-core and foundation4ai, both with STATUS deployed
helm status foundation4ai -n foundation4ai	State of the latest revision of the application release	STATUS: deployed and the revision number
helm history foundation4ai -n foundation4ai	Revisions with status and description	The latest revision deployed, earlier revisions superseded. A failed revision describes the failed hook or resource.
helm get values foundation4ai -n foundation4ai	Values supplied to the release	The content of the values file, without credentials

The same commands with foundation4ai-core show the core release. helm get hooks and helm get all print the rendered Secrets, so the output of these commands is handled as described in Secrets and keys. helm rollback does not reverse database migrations, and an older image does not start against a database that a newer version has migrated (inferred), so Backup, restore and upgrades describes the recovery from a failed upgrade.

NATS JetStream

The core release deploys the utility pod foundation4ai-core-nats-box, which contains the nats command-line tool with a default context that points to nats://foundation4ai-core-nats. The following commands run in that pod:

kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- nats <subcommand>

Subcommand	Purpose	Expected output
nats server check connection	Connection from the namespace to NATS	A line that begins with OK Connection
nats account info	JetStream availability and storage use of the account	JetStream account information with the storage in use below the limit
nats stream ls	Streams of the account	DOCUMENTS and OBJ_documents, the stream of the documents object store
nats stream info DOCUMENTS	Configuration and state of the processing queue	Subjects: document.>, Retention: WorkQueue, Maximum Age: 1d0h0m0s and Replicas: 1. Under the state, Messages is the number of waiting jobs.
nats consumer info DOCUMENTS document-processor	State of the consumer that the workers share	Ack Wait: 30.00s and Max Ack Pending: 1,000. The state shows the outstanding acknowledgements, the redelivered messages and the unprocessed messages.

Without the default context, each subcommand takes --server nats://foundation4ai-core-nats:4222. The commands above read state only. nats stream purge, nats stream rm and nats consumer rm delete queued jobs, which leaves the affected documents pending, as described in Troubleshooting.

API checks

1. Forward a local port to the API server Service in a separate terminal:

   kubectl port-forward -n foundation4ai svc/foundation4ai-api-server 8080:80

   Expected result: Forwarding from 127.0.0.1:8080 -> 8000.

2. Check an API key with the shared variables of Authenticate:

   export FOUNDATION4_URL=http://localhost:8080
   curl -X POST "$FOUNDATION4_URL/login" \
     -H "x-api-key: $FOUNDATION4_API_KEY" \
     -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

   Expected result: status 201 and a body whose message is OK and whose key is the name of the key, followed by the permissions of the key.

3. Check that the API server process answers:

   curl -s "$FOUNDATION4_URL/healthz"

   Expected result: {"status":"ok","database":"ok","message_queue":"ok","cache":"ok","llm_service":"ok"}. The fields report ok whatever the state of the components, so the response confirms only that the API server process answers. POST /login in step 2 is the authenticated check, and the database check below tests the database connection.

The worker metrics are read through a port forward to one worker pod:

kubectl port-forward -n foundation4ai deploy/foundation4ai-api-server-worker 9090:9090
curl -s http://localhost:9090/metrics | grep foundation4ai_document_queue

Expected result: one line per metric, such as foundation4ai_document_queue_processed{description="Number of documents processed"} 1200. Monitoring and logging describes each metric.

Configuration and database checks

Command	Purpose	Expected output
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ls /app/config	Configuration files of the API server, in load order	f4ai-00-defaults.yaml, the files of api-server.configs, and five f4ai-zz-*-credentials.yaml files
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai database check-connection	Connection to PostgreSQL with the URL and CA certificate of the configuration	Ok.

database check-connection reports every failure as Error: "Failed connecting to database", without the reason. A PostgreSQL client in a temporary pod shows the reason. The pod reads POSTGRES_URL from foundation4ai-secrets, so the check applies to the production profile with an external database; <postgresql-client-image> is any image that contains psql, taken from the registry that the cluster uses:

kubectl run pg-check -n foundation4ai --rm -i --restart=Never --image=<postgresql-client-image> \
  --overrides='{"apiVersion":"v1","spec":{"containers":[{"name":"pg-check","image":"<postgresql-client-image>","command":["sh","-c","psql \"$POSTGRES_URL\" -c \"SELECT version();\""],"envFrom":[{"secretRef":{"name":"foundation4ai-secrets"}}]}]}}'

Expected result: one row with the PostgreSQL version, then pod "pg-check" deleted. A failed connection prints the reason, such as a refused connection, a failed password or a certificate error. With sslmode=verify-full, psql also needs the CA certificate through the sslrootcert parameter, which the pod does not have. Under the network policies of Security hardening, the temporary pod reaches only DNS unless a policy allows the connection.

Foundation4 CLI

Command-line interface is the full reference for the subcommands, the connections and the exit status. The API server image contains the foundation4ai binary in /app, the working directory of the containers. Every subcommand reads the configuration from ./config, which is /app/config in the server and worker containers and in the installation Jobs. Operators run subcommands in the server container of a running API server pod:

kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai <subcommand>

A subcommand that fails prints Error: "<message>" and exits with status 1, and an invalid command line exits with status 2. license details exits with status 0 after printing Invalid or Expired, so scripts read the printed result. The admin subcommands and data download-models build the same connection as the API server and fail in the same ways, as listed in Troubleshooting.

Subcommand	Arguments	Purpose	Output on success	Run by
server	None	Runs the API server	None at the level warn	server container
worker	None	Runs a worker with a database pool of 1 connection	None at the level warn	worker container
database migrate	None	Applies pending database migrations	Migrated to latest schema.	Migration Job (hook)
database check-connection	None	Connects to PostgreSQL with the configured URL and CA certificate	Ok.	Operator
license check	None	Prints the system ID and the result of the license check, and waits 365 days after a failed check	SystemID: <system ID>, then Checking for a valid license... OK	License check Job (hook)
license system-id	None	Prints the system ID of the database	64 hexadecimal characters	Operator
license details	<license>	Prints the content of a license that is valid for the system ID	License: <name> (expires on <date>), System ID:, Version: and Limits: lines, and optional Description: and Environment: lines	Operator
admin create-admin-api-key	<name> [description] [expiration]	Creates the master key from app.master.key and app.master.secret	Admin API key created successfully.	Master key Job (hook), with the name Master Admin Key
admin update-admin-api-key	[name] [description]	Changes the name or the description of the master key	Admin API key updated successfully.	Operator
admin generate-admin-api-key	<name> [description] [expiration]	Creates an additional key with all permissions, a new identifier and a random 32-character secret	Admin API key created successfully., then Key : <identifier> and Secret: <secret>	Operator
data providers	None	Lists the providers of the embedding models and text splitters in the database	Found embedding providers: and Found text splitters providers:, each followed by - <provider> lines	Operator
data download-models	None	Downloads the files of the models and text splitters in the database into the model directories of the containers of the pod	Downloading data for embedding provider <provider>, model <model> lines	Operator, connected deployments only

The following points apply to the subcommands:

• Hook subcommands. database migrate, license check and admin create-admin-api-key run in the installation Jobs at every installation and upgrade of the application release. An operator does not run license check interactively, because the command waits 365 days after a failed check; license details prints the license content and returns.

• License details. The license is passed as an argument. The following commands read the value from the secrets file:

  LICENSE=$(grep '^FOUNDATION4AI_APP_LICENSE=' foundation4ai.secrets.env | cut -d= -f2-)
  kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \
    ./foundation4ai license details "$LICENSE"

  Expected result: the license name with the expiry date or (perpetual), the system ID, the version and the limits. The command prints Expired for an expired license and Invalid for a license issued for another system ID. Licensing describes the license fields.

• System ID. license system-id and license check compute the system ID for the schema public. With a database.schema other than public, the result can differ from the system ID that the API server computes, as described in Licensing.

• Master key. The master key must remain active and unexpired, because the API server and the workers authenticate with the master key at startup, as described in Access control. admin update-admin-api-key accepts only the name and the description, as positional arguments.

• Generated keys. admin generate-admin-api-key prints the new secret once. The command runs in a session whose output is not recorded, and the secret goes directly to the organization's secret store. The new key does not replace the master key.

• Model downloads. data download-models writes into the containers of the pod where the command runs, and the files are lost when the containers restart. Air-gapped deployments use model and package images instead, as described in Models, packages and air-gapped installs.

• Help. ./foundation4ai --help and ./foundation4ai <subcommand> --help print the usage of each subcommand. ./foundation4ai --version prints foundation4ai 0.1.0, which does not identify the Foundation4 release.