Skip to main content

Operator command reference

This page lists the commands that operators run most often on a Foundation4 deployment, grouped by task, with the purpose of each command and the output of a healthy deployment. The page also describes every subcommand of the foundation4ai command-line interface (CLI) in the API server image, and marks the subcommands that the installation Jobs run. Operators use this page during installation, routine checks and Troubleshooting. The commands use the namespace foundation4ai and the releases foundation4ai-core and foundation4ai of Install on Kubernetes.

Pods and Jobs​

CommandPurposeExpected output
kubectl get pods -n foundation4ai -o wideState, restarts and node of every podEvery pod Running. API server and worker pods 2/2, dashboard pod 1/1, no rising RESTARTS count
kubectl get deployments,statefulsets -n foundation4aiReady replicas of each workloadREADY equals the desired count for every workload, such as 3/3 for foundation4ai-api-server-worker and the StatefulSet foundation4ai-core-nats
kubectl get jobs -n foundation4aiState of the installation JobsNo resources found in foundation4ai namespace. after a completed installation or upgrade. A listed foundation4ai-api-server-db-migration, foundation4ai-api-server-license-check or foundation4ai-api-server-create-admin-api-key Job is running or has failed
kubectl get pvc -n foundation4aiState of the volume claims3 NATS JetStream claims, 1 Prometheus claim and, in the evaluation profile, 1 PostgreSQL claim, all Bound
kubectl describe pod -n foundation4ai <pod-name>Events, container states and the reason of the last terminationNo Failed or BackOff events, Last State empty or Completed
kubectl get events -n foundation4ai --sort-by=.lastTimestampRecent events of the namespace, oldest firstNormal events only
kubectl top pods -n foundation4ai --containersCPU and memory of each container; needs the metrics serverMemory of each container below the limit
kubectl rollout restart -n foundation4ai deployment/<name>Restarts the pods of a Deployment, for example after a configuration changedeployment.apps/<name> restarted
kubectl rollout status -n foundation4ai deployment/<name>Waits until the restarted pods are readydeployment "<name>" successfully rolled out

Helm deletes each installation Job after the Job succeeds. A failed Job and its log remain until the next installation or upgrade of the application release replaces the Job.

Logs​

CommandPurposeExpected output
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c serverAPI server logAt the default level warn, no lines while the API server is healthy other than the two OpenTelemetry warnings at each start, described in Monitoring and logging, and the Waiting for the database, cache or queue retries while PostgreSQL, the cache or NATS JetStream is not yet reachable
kubectl logs -n foundation4ai deploy/foundation4ai-api-server-worker -c workerLog of one workerAt the level warn, no lines while processing succeeds other than the same start-up warnings as the API server
kubectl logs -n foundation4ai -l app.kubernetes.io/name=api-server-worker -c worker --tail=200 --prefixLogs of every worker, each line prefixed with the pod nameNo Error processing documents or Failed processing document lines
kubectl logs -n foundation4ai deploy/foundation4ai-api-server -c grpcgRPC service log of an API server podINFO:processor:gRPC server started on 127.0.0.1:50051
kubectl logs -n foundation4ai <pod-name> -c <container> --previousLog of the previous container after a restartThe error that stopped the container, such as Error: "<message>"

A command with deploy/<name> reads the log of one pod of the Deployment. Monitoring and logging describes the log formats and levels.

Installation Job logs​

CommandPurposeExpected output
kubectl logs -n foundation4ai job/foundation4ai-api-server-db-migrationResult of the database migrationMigrated to latest schema.
kubectl logs -n foundation4ai job/foundation4ai-api-server-license-checkSystem ID and license check resultSystemID: <system ID>, then Checking for a valid license... OK
kubectl logs -n foundation4ai job/foundation4ai-api-server-create-admin-api-keyCreation or check of the master keyMaster key created successfully., Master key secret updated to match the configuration. or Master key already matches the configuration.
kubectl get pods -n foundation4ai -l job-name=<job-name>Pods of one Job, including failed attemptsOne pod per attempt; the pods of failed attempts show Error

Helm deletes each Job after the Job succeeds, so these commands read a Job while the Job runs, or after a failure until the next installation or upgrade of the application release. kubectl logs job/<name> reads one pod of the Job; the pod list shows every attempt, and kubectl logs -n foundation4ai <pod-name> reads a specific attempt. The license check Job makes one attempt.

Helm releases​

CommandPurposeExpected output
helm list -n foundation4aiReleases, revisions and chart versionsfoundation4ai-core and foundation4ai, both with STATUS deployed
helm status foundation4ai -n foundation4aiState of the latest revision of the application releaseSTATUS: deployed and the revision number
helm history foundation4ai -n foundation4aiRevisions with status and descriptionThe latest revision deployed, earlier revisions superseded. A failed revision describes the failed hook or resource.
helm get values foundation4ai -n foundation4aiValues supplied to the releaseThe content of the values file, without credentials

The same commands with foundation4ai-core show the core release. helm get hooks and helm get all print the rendered Secrets, so the output of these commands is handled as described in Secrets and keys. helm rollback does not reverse database migrations: migrations only move forward, and an older image stops at startup against a database that a newer version has migrated. A rollback after an upgrade that migrated the database therefore needs the database backup taken before the upgrade, as described in Backup, restore and upgrades.

NATS JetStream​

The core release deploys the utility pod foundation4ai-core-nats-box, which contains the nats command-line tool. NATS JetStream accepts only authenticated clients: the user is foundation4ai, and the password is the NATS_PASSWORD value of the Secret foundation4ai-secrets. The following commands read the password and run a subcommand in that pod:

NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
kubectl exec -n foundation4ai deploy/foundation4ai-core-nats-box -- \
nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" <subcommand>
SubcommandPurposeExpected output
nats server check connectionConnection from the namespace to NATSA line that begins with OK Connection
nats account infoJetStream availability and storage use of the accountJetStream account information with the storage in use below the limit
nats stream lsStreams of the accountDOCUMENTS and OBJ_documents, the stream of the documents object store
nats stream info DOCUMENTSConfiguration and state of the processing queueSubjects: document.>, Retention: WorkQueue, Maximum Age: 1d0h0m0s and Replicas: 3, the value of global.natsReplicas. Under the state, Messages is the number of waiting jobs.
nats consumer info DOCUMENTS document-processorState of the consumer that the workers shareAn Ack Wait of 10 minutes and Max Ack Pending: 1,000. The state shows the outstanding acknowledgements, the redelivered messages and the unprocessed messages.

The commands above read state only. nats stream purge, nats stream rm and nats consumer rm delete queued jobs, which leaves the affected documents pending, as described in Troubleshooting.

API checks​

  1. Forward a local port to the API server Service in a separate terminal:

    kubectl port-forward -n foundation4ai svc/foundation4ai-api-server 8080:80

    Expected result: Forwarding from 127.0.0.1:8080 -> 8000.

  2. Check an API key with the shared variables of Authenticate:

    export FOUNDATION4_URL=http://localhost:8080
    curl -X POST "$FOUNDATION4_URL/login" \
    -H "x-api-key: $FOUNDATION4_API_KEY" \
    -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

    Expected result: status 201 and a body whose message is OK and whose key is the name of the key, followed by the permissions of the key.

  3. Check the components that the API server uses:

    curl -s -w "\n%{http_code}\n" "$FOUNDATION4_URL/healthz"

    Expected result: {"status":"ok","database":"ok","message_queue":"ok","cache":"ok","llm_service":"ok"}, then 200. A failing PostgreSQL, NATS JetStream, cache or gRPC service returns status 503 with "status":"error" and the value "error" for that component, as described in Monitoring and logging. POST /login in step 2 is the authenticated check.

The worker metrics are read through a port forward to one worker pod:

kubectl port-forward -n foundation4ai deploy/foundation4ai-api-server-worker 9090:9090
curl -s http://localhost:9090/metrics | grep foundation4ai_document_queue

Expected result: a # HELP line, a # TYPE line and the values of each metric, such as foundation4ai_document_queue_processed 1200. The API server metrics are on the metrics port 9090 of the API server pod, which no Service exposes:

kubectl port-forward -n foundation4ai deploy/foundation4ai-api-server 9091:9090
curl -s http://localhost:9091/metrics | grep -E "^(axum_http_requests_total|foundation4ai_)"

Expected result: request counts per method, status and endpoint, and the document and agent execution metrics. Monitoring and logging describes each metric.

Configuration and database checks​

CommandPurposeExpected output
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ls /app/configConfiguration files of the API server, in load orderf4ai-00-defaults.yaml, the files of api-server.configs and api-server.secrets, and the directory secrets, which holds one file per connection URL and credential, as described in Configuration reference
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai database check-connectionConnection to PostgreSQL with the URL and CA certificate of the configurationOk.

database check-connection reports a failure as Error: "Failed connecting to database: <reason>", where the reason names the failure, such as a host name that cannot be resolved, a rejected password, a certificate error or a rule of pg_hba.conf. A PostgreSQL client in a temporary pod tests the same connection outside the API server image. The pod reads POSTGRES_URL from foundation4ai-secrets, so the check applies to the production profile with an external database; <postgresql-client-image> is any image that contains psql, taken from the registry that the cluster uses:

kubectl run pg-check -n foundation4ai --rm -i --restart=Never --image=<postgresql-client-image> \
--overrides='{"apiVersion":"v1","spec":{"containers":[{"name":"pg-check","image":"<postgresql-client-image>","command":["sh","-c","psql \"$POSTGRES_URL\" -c \"SELECT version();\""],"envFrom":[{"secretRef":{"name":"foundation4ai-secrets"}}]}]}}'

Expected result: one row with the PostgreSQL version, then pod "pg-check" deleted. A failed connection prints the reason, such as a refused connection, a failed password or a certificate error. With sslmode=verify-full, psql also needs the CA certificate through the sslrootcert parameter, which the pod does not have. Under the network policies of Security hardening, the temporary pod reaches only DNS unless a policy allows the connection.

Foundation4 CLI​

Command-line interface is the full reference for the subcommands, the connections and the exit status. The API server image contains the foundation4ai binary in /app, the working directory of the containers. Every subcommand reads the configuration from ./config, which is /app/config in the server and worker containers and in the installation Jobs. Operators run subcommands in the server container of a running API server pod:

kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai <subcommand>

A subcommand that fails prints Error: "<message>" and exits with status 1, and an invalid command line exits with status 2. license check and license details exit with status 1 when the license cannot be used, so scripts read the exit status. The admin subcommands connect to PostgreSQL and the Redis-compatible cache only, and admin keys-without-classifications to PostgreSQL only. data check-models builds the same connection as the API server and fails in the same ways, as listed in Troubleshooting. Connections lists the connection of each subcommand.

SubcommandArgumentsPurposeOutput on successRun by
serverNoneRuns the API serverNone at the level warnserver container
workerNoneRuns a worker with a database pool of 1 connectionNone at the level warnworker container
database migrateNoneApplies pending database migrationsMigrated to latest schema.Migration Job (hook)
database check-connectionNoneConnects to PostgreSQL with the configured URL and CA certificateOk.Operator
license checkNonePrints the system ID and the result of the license check, and exits with status 1 after a failed checkSystemID: <system ID>, then Checking for a valid license... OKLicense check Job (hook), operator
license system-idNonePrints the system ID of the database64 hexadecimal charactersOperator
license details<license>Prints the content of a license that is valid for the system IDLicense: <name> (expires on <date> <time> UTC), System ID:, Version: and Limits: lines, and optional Description: and Environment: linesOperator
admin create-admin-api-key<name> [description] [expiration]Makes the master key match app.master.key and app.master.secret: creates the key, or replaces the stored secretMaster key created successfully., Master key secret updated to match the configuration. or Master key already matches the configuration.Master key Job (hook), with the name Master Admin Key
admin update-admin-api-keyOptions --name, --description, --clear-description, --active, --expiration, --clear-expirationChanges the name, the description, the active flag or the expiry of the master keyMaster key updated successfully.Operator
admin generate-admin-api-key<name> [description] [expiration]Creates an additional key with all permissions, a new identifier and a random 32-character secretAdministration key created successfully., then Key : <identifier> and Secret: <secret>Operator
admin keys-without-classificationsNoneLists the active keys whose pipeline grants name no classification, so that their searches find nothingEvery active API key that can read or search pipelines names classifications., or one line per grantOperator
data providersNoneLists the providers of the embedding models and text splitters in the databaseFound embedding providers: and Text splitter providers in use:, each followed by - <provider> linesOperator
data check-modelsNoneChecks that every configured embedding model and text splitter loads, as the API server and the workers load themOne line per model that ends with the way the model was checked, then Every configured model loads.Operator

The following points apply to the subcommands:

  • Hook subcommands. database migrate, license check and admin create-admin-api-key run in the installation Jobs at every installation and upgrade of the application release. license check returns at once, so an operator also runs the command in a running API server pod to read the system ID and the result.

  • License details. The license is passed as an argument. The following commands read the value from the secrets file:

    LICENSE=$(grep '^FOUNDATION4AI_APP_LICENSE=' foundation4ai.secrets.env | cut -d= -f2-)
    kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- \
    ./foundation4ai license details "$LICENSE"

    Expected result: the license name with the expiry date or (perpetual), the system ID, the version and the limits. For a license that cannot be used, the command prints Error: "Expired" or Error: "Invalid", for example for a license issued for another system ID, and exits with status 1. Licensing describes the license fields.

  • System ID. license system-id, license check and license details compute the system ID for the database in database.url and the schema in database.schema, as the API server does. Licensing describes the binding.

  • Master key. The master key remains active and unexpired, because the workers authenticate with the master key at startup and stop without the key, as described in Access control. The API server starts with an expired or inactive master key, logs an error and refuses requests made with that key. admin update-admin-api-key --active true or --clear-expiration restores the key, as described in Master key update.

  • Generated keys. admin generate-admin-api-key prints the new secret once. The command runs in a session whose output is not recorded, and the secret goes directly to the organization's secret store. The new key does not replace the master key.

  • Model check. data check-models downloads no FastEmbed model file, so a FastEmbed model that passes the check loads without network access. The CLI has no command that downloads model files; model and package images deliver the files to every pod, as described in Models, packages and air-gapped installs. Model check describes the output and the failures.

  • Keys without classifications. admin keys-without-classifications changes nothing. Keys without classifications describes the output, and Manage API keys and permissions describes how to grant classifications.

  • Help. ./foundation4ai --help and ./foundation4ai <subcommand> --help print the usage of each subcommand. ./foundation4ai --version prints foundation4ai <version>, such as foundation4ai 1.1.0, as described in Command-line interface.