Command-line interface
This page is the reference for foundation4ai, the command-line interface (CLI) of Foundation4. The page describes the global options, the configuration and the connections that the CLI uses, and the arguments, output, errors and exit status of every subcommand. Operators use this page to run maintenance commands in an installation and to read the logs of the installation Jobs. Operator command reference lists the subcommands next to the other operator commands.
Synopsis
foundation4ai [-h | --help] [-V | --version]
foundation4ai <command> [<subcommand>] [<arguments>]
| Command | Arguments | Purpose | Connection |
|---|---|---|---|
server | None | Runs the API server. The binary without a command runs the same process. | Full |
worker | None | Runs a worker | Full |
database migrate | None | Applies pending database migrations | Database only |
database check-connection | None | Tests the database connection | Database only |
license system-id | None | Prints the system ID | Database only |
license check | None | Checks the configured license | Database only |
license details | <LICENSE> | Prints the contents of a license | Database only |
data providers | None | Lists the providers of the configured embedding models and text splitters | Database only |
data check-models | None | Checks that the configured embedding models and text splitters load | Full |
admin create-admin-api-key | <NAME> [DESCRIPTION] [EXPIRATION] | Creates the master key, or updates its secret to match the configuration | Database and cache |
admin update-admin-api-key | Options, as described in Master key update | Changes the name, the description, the active flag or the expiry of the master key | Database and cache |
admin generate-admin-api-key | <NAME> [DESCRIPTION] [EXPIRATION] | Creates an additional administration key | Database and cache |
admin keys-without-classifications | None | Lists the active keys that can read or search pipelines but name no classification | Database only |
help | [<command>] [<subcommand>] | Prints the help of the binary or of a command | None |
Arguments are positional, except the options of admin update-admin-api-key. Angle brackets mark a required argument, and square brackets mark an optional argument. An argument that contains spaces is quoted, as in "Master Admin Key". The other subcommands accept no option other than -h and --help, and no subcommand reads standard input or asks for confirmation. Connections describes the three connection types.
Invocation
The API server image contains the binary as /app/foundation4ai, and /app is the working directory of the API server, worker and installation Job containers. Operators run subcommands in the server container of a running API server pod:
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai license system-id
Expected result: the output of the subcommand, here one line of 64 hexadecimal characters. kubectl exec returns the exit status of the subcommand and prints command terminated with exit code <status> for any status other than 0.
The binary runs in the following places:
- Containers. The
servercontainer runs./foundation4ai server, and theworkercontainer runs./foundation4ai worker. Operators do not start either subcommand throughkubectl exec, because the containers already run both processes. - Installation Jobs. The application release runs
database migrate,license checkandadmin create-admin-api-key "Master Admin Key"as installation Jobs at every installation and upgrade, in the order described in Install on Kubernetes. - Image default. A container started from the API server image without arguments runs the binary without a command, which runs the API server as
serverdoes. The charts always pass a command.
Global options
| Option | Effect | Exit status |
|---|---|---|
-h, --help | Prints the help of the binary, or of the command or subcommand before the option, on standard output | 0 |
-V, --version | Prints foundation4ai <version>, the version of the program package, such as foundation4ai 1.1.0 | 0 |
help [<command>] [<subcommand>] | Prints the same help as --help | 0 |
The version of the program package names the Foundation4 release. The 1.1.0 images print the release only, so the tag of the API server image is what tells apart the builds of one release, as described in Charts, images and installation bundle. A binary built with a build identifier adds a plus sign and the identifier, as in foundation4ai 1.1.0+<build>.
./foundation4ai --help prints the following text:
Foundation4.ai API Server
Usage: foundation4ai [COMMAND]
Commands:
admin Admin commands
license License commands
database Database operations
data Data operations
server Run the API Server
worker Run the Worker
help Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help
-V, --version Print version
The following rules apply to the command line:
- Command groups.
admin,license,databaseanddataare groups of subcommands. A group without a subcommand prints the help of the group on standard error and exits with status 2. - No command. The binary without a command runs the API server, as
serverdoes. - Unknown names. An unknown command or subcommand prints
error: unrecognized subcommand '<name>', followed by similar names when a similar name exists, and exits with status 2.
Configuration
Every subcommand reads the configuration from the working directory in the order that Configuration reference describes: defaults.yaml, config/settings.yaml, the config/f4ai-*.yaml files in name order, the files under config/secrets named by key path, then the FOUNDATION4AI__<SECTION>__<KEY> environment variables. The configuration directory is fixed at ./config, and the CLI has no option that changes the directory or a configuration key.
The following rules apply to the configuration of the CLI:
- Complete configuration. Every subcommand parses the whole configuration, including the keys that the subcommand does not use. A missing or invalid required key stops the subcommand with a message such as
Error: "Failed to parse config: missing field `license` for key \"default.app\"". - Start time. A subcommand reads the configuration when the subcommand starts, and the API server and worker processes read the configuration when the processes start. Configuration change describes the restart that applies a change to the running processes.
- Database settings. The database-only subcommands and the admin subcommands use
database.url,database.ca_cert,database.schemaanddatabase.embeddings_schema, and ignoredatabase.pool_size. They use the schemas of the API server, sodatabase migratecreates the tables indatabase.schema, and the license subcommands compute the system ID for that schema, as described in Licensing. Withdatabase.ca_cert, thesslmodevaluerequirealso verifies the server certificate against the certificate authority (CA) certificate, as the connection of the API server does. - Worker pool.
workeruses a database pool of 1 connection, whatever the value ofdatabase.pool_size. - Log level.
app.log_levelapplies toserverandworker. The other subcommands write log lines to standard error at the levelinfoand above, or atapp.log_levelwhen that level isdebugortrace, next to the output that this page lists. The warning with the system ID that the full connection logs after a license failure therefore appears in the output ofdata check-models.
Connections
Each subcommand uses one of three connection types:
- Database only.
database migrate,database check-connection, the three license subcommands,data providersandadmin keys-without-classificationsopen one PostgreSQL connection with the database settings of Configuration. These subcommands report a connection failure once, asError: "Failed connecting to database: <reason>", where the reason tells apart name resolution, authentication, Transport Layer Security (TLS) and access rule failures. - Database and cache.
admin create-admin-api-key,admin update-admin-api-keyandadmin generate-admin-api-keyopen a PostgreSQL connection as the database-only subcommands do, and connect to the Redis-compatible cache. These subcommands need no license, no NATS JetStream and no application secret. While PostgreSQL or the cache does not answer, the subcommands logWaiting for the database or cache; retrying in <delay>and try again, with a delay that doubles up to 10 seconds, for about four minutes. The last failure printsError: "Failed connecting to database: <reason>"orError: "Failed connecting to the cache: <reason>". - Full connection.
server,workeranddata check-modelsbuild the same connection as the API server. The full connection needs PostgreSQL with thevector,vchord,pg_tokenizerandvchord_bm25extensions, a valid license, the Redis-compatible cache, NATS JetStream and a valid application secret.
The full connection runs the following steps in order, and the first failure stops the subcommand:
- Connects to PostgreSQL with the pool size, the schemas and the CA certificate of the configuration, creates missing schemas and applies pending migrations.
- Computes the system ID from
database.schemaand checksapp.license. - Connects to the Redis-compatible cache.
- Connects to NATS JetStream, and creates the object store
nats.bucketand the streamnats.streamwhen missing. - Decodes the application secret
app.secret. - Prepares the address of the gRPC service in
grpc.url. The connection to the gRPC service opens on first use.
While PostgreSQL, the cache or NATS JetStream does not answer, the full connection logs Waiting for the database, cache or queue; retrying in <delay> and starts again, with a delay that doubles up to 10 seconds, for about four minutes. A license failure and the other failures, such as a failed migration, stop the subcommand without a retry.
data check-models reports a license failure as Error: "License check failed: <reason>" and a failure of another step as Error: "Failed to connect to Foundation4.ai's services: <reason>". The beginning of the reason names the step:
| Reason begins with | Step |
|---|---|
ORM error:, The database was migrated by a newer Foundation4 release | PostgreSQL connection or migration |
Invalid license, Expired license, Error computing system id | License check |
Redis error:, Invalid parameter: Invalid Redis URL | Redis-compatible cache |
Nats error: | NATS JetStream |
Decryption error: Error decoding secret key | Application secret |
gRPC error: | Address of the gRPC service |
Before these steps, the full connection reads the four paths keys, and a missing key stops the subcommand with Error: "missing field `<key>`". server prints these messages after its own prefix, as described in Server, and Troubleshooting lists the causes.
Output and exit status
Subcommands print results on standard output and errors on standard error:
| Exit status | Meaning | Standard error |
|---|---|---|
| 0 | The subcommand completed, or the help or version was printed on request | Empty, or the log lines described in Configuration |
| 1 | The subcommand failed | Error: "<message>" |
| 2 | The command line is invalid, or names a group without a subcommand | error: <reason> followed by the usage, or the help |
| 134 | The process stopped on an internal error with the abort signal | A message that contains panicked at |
The following rules apply to the output:
-
Quoted messages. The message after
Error:is printed as a quoted string, so a quotation mark inside the message appears as\"and a line break as\n. -
License results.
license checkandlicense detailsexit with status 1 when the license cannot be used, so scripts can read the exit status. -
Argument errors. An invalid argument prints the reason and a pointer to the help, as in the following output for an expiry without a time:
error: invalid value '2027-01-31' for '[EXPIRATION]': premature end of inputFor more information, try '--help'.
Process subcommands
Server
server runs the API server. The subcommand builds the full connection, checks the master key, serves the metrics on app.metrics_port and listens on app.host and app.port. The process logs at app.log_level, and at the level info the log shows Starting <branding.name> API on <host>:<port>. On the interrupt signal or the termination signal, the API server stops accepting connections, completes open requests and exits with status 0.
A master key that is expired, inactive or missing does not stop the API server: the API server logs an error that names admin update-admin-api-key --clear-expiration, and refuses requests made with the master key until the key is restored. A master key secret that does not match the stored key stops the API server.
The API server reports connection and master key failures after Error: "Failed to initialize server: , such as Failed to initialize server: Failed to initialize with the master key: <reason>. A listen address that cannot be used prints Failed to bind to address: <reason>, or Failed to bind the metrics address: <reason> for the metrics port. Troubleshooting lists the messages with the causes.
Worker
worker runs a worker. The subcommand opens the metrics listener on port 9090, builds the full connection with a database pool of 1 connection, and authenticates with the master key. After a wait of 2 seconds, the worker logs Starting Foundation4.ai worker at the level info and fetches processing jobs from NATS JetStream in batches of up to 100 jobs.
The following rules apply to the worker:
- Stop signal. On the interrupt signal or the termination signal that Kubernetes sends, the worker completes the batch in progress, logs
Received shutdown signal, stopping workerand exits with status 0. - Startup errors. The worker reports connection failures with the messages of Connections, such as
License check failed: <reason>, master key failures asFailed to initialize Foundation4.ai: <reason>, and queue consumer failures asError creating message queue consumer: <reason>. An expired or inactive master key therefore stops the worker. When port 9090 is already in use, the worker stops withError: "Failed to install metrics recorder: <reason>". - Queue errors. A failed read from the queue logs
Error retrieving messages from message queue, and the worker tries again after 2 seconds. - License expiry. The worker checks the license before each batch. When the license is no longer valid, for example after its expiry date, the worker logs
The license is no longer valid: the worker stops; queued documents wait for a renewed licenseand stops withError: "License check failed: <reason>". A restarted worker does not start until a valid license is installed.
Database subcommands
Database migration
database migrate applies the pending migrations of the database schema, records each migration in the table schema_migrations, and prints Migrated to latest schema. The migration Job runs the command before the license check at every installation and upgrade, and the API server, the workers and data check-models also apply pending migrations when they build the full connection.
The following rules apply to the migration:
- Schemas. The command creates the schemas
database.schemaanddatabase.embeddings_schemawhen missing, and applies the migrations indatabase.schema, as the API server does. A schema that cannot be created stops the command withError: "Failed creating schema <schema>: <reason>". - Earlier table format. When
database.schemaholds a table namedschema_migrationswith exactly one column, and every row of the table names a Foundation4 migration, the command drops that table before applying the migrations, because the table then has the format of an earlier Foundation4 version. When the table holds other rows, such as those of another migration tool, the command leaves the table in place and stops withError: "Failed applying migrations: the schema has a schema_migrations table of another tool. Give Foundation4 its own schema (database.schema).". - Direction. Each migration runs in one transaction, and migrations only move forward. The CLI has no command that reverts a migration, as described in Database. A database that a newer Foundation4 release migrated stops the command with
Error: "Failed applying migrations: the database was migrated by a newer Foundation4 release (<reason>). Run the newer release, or restore the backup taken before the upgrade, then roll back.". - Extensions. When a migration fails on a database extension, the message ends with a line that names the
vector,vchord,pg_tokenizerandvchord_bm25extensions, which only a superuser can create, and the bundled PostgreSQL image, which has them. - Errors.
Error: "Failed connecting to database: <reason>"means that the connection failed.Error: "Failed applying migrations: <reason>"means that the connection succeeded and a migration failed, as described in Troubleshooting.
Database connection check
database check-connection opens a connection with the database settings of Configuration and prints Ok. The check covers the address, the credentials and the TLS settings. The check does not cover the schemas, the migrations or the database extensions. A failure prints Error: "Failed connecting to database: <reason>", where the reason names the failure, such as a host name that cannot be resolved, a rejected password or a certificate error. Operator command reference tests the same connection with a PostgreSQL client.
License subcommands
The license subcommands compute the system ID for the database in database.url and the schema in database.schema, as the API server does. license system-id works before the first migration. The computation reads pg_control_system(), and a database user who cannot read it makes license system-id fail with Error: "Failed generating system id: the database user cannot read pg_control_system()". Licensing describes licenses and the system ID.
System ID
license system-id prints the system ID as one line of 64 hexadecimal characters. The Foundation4 provider issues a license for this value, as described in Licensing.
License check
license check checks the license in app.license against the system ID and prints two lines:
SystemID: <system ID>
Checking for a valid license... OK
When the system ID cannot be computed, the first line reads SystemID: unavailable (<reason>). The second line ends with one of the following results:
| Result | Meaning | Exit status |
|---|---|---|
OK | The license is valid for the system ID and has not expired | 0 |
Missing | app.license is an empty string | 1 |
Invalid | The license is malformed, of an unsupported version, or issued for another system ID | 1 |
Expired | The expiry date of the license has passed | 1 |
The system id for this license version could not be computed | The database does not provide the values that the license version binds to | 1 |
A result other than OK also prints Error: "License check failed: <result>" on standard error. The license check Job therefore fails at once, and its log keeps the system ID of the first line, so the application installation stops with a failed hook, as described in Troubleshooting. Under the charts, an empty license value in the secrets file reaches the check and reads as Missing.
License details
license details <LICENSE> checks the license given as the argument against the system ID and prints the contents of the license. The license is one argument in quotes:
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai license details '<license>'
Expected result for a license that matches the database:
License: <license holder> (expires on <date> <time> UTC)
System ID: <system ID>
Version: <license version>
Limits:
Pipelines: <limit>
Documents: <limit>
A license without an expiry date shows (perpetual), and optional Description: and Environment: lines appear when the license carries these fields. Licensing describes every line. For a license that cannot be used, the command prints Error: "<result>" on standard error instead of the contents, with one of the results of License check, and exits with status 1. Operator command reference reads the license value from the secrets file.
Data subcommands
Provider list
data providers lists the distinct providers of the embedding models and of the text splitters in the database, in no fixed order. A database with only the seeded objects of Providers and models produces the following output:
Found embedding providers:
- FastEmbedEmbeddings
Text splitter providers in use:
- RecursiveCharacterTextSplitter
- CharacterTextSplitter
- NLTKTextSplitter
The list names the providers that configured objects use, not every provider that Foundation4 supports. Providers and models lists the supported providers.
Model check
data check-models builds the full connection and checks that every configured embedding model and text splitter loads and works, as the API server and the workers load them. The command downloads no FastEmbed model file, so a FastEmbed model that passes the check loads without network access. A database with only the seeded objects of Providers and models and the model images in place produces the following output:
Checking that the configured models load. FastEmbed models are checked in this container without downloading; other providers are checked by the provider service at <grpc.url>.
Embedding models:
- all-MiniLM-L6-v2 (FastEmbedEmbeddings, Qdrant/all-MiniLM-L6-v2-onnx): ok, loads offline from <directory>
Text splitters:
- CharacterTextSplitter (<identifier>): ok, splits a sample text on the provider service
- NLTKTextSplitter (<identifier>): ok, splits a sample text on the provider service
- RecursiveCharacterTextSplitter (<identifier>): ok, splits a sample text on the provider service
Every configured model loads.
The command lists embedding models by name and text splitters by provider. Each line ends with the way the object was checked:
| Provider | Check | Line ending |
|---|---|---|
FastEmbedEmbeddings | Finds the model files in paths.fastembed or in the writable cache of the container that runs the command, then loads the model and embeds a sample text | loads offline from <directory> |
OpenAIEmbeddings, HuggingFaceEndpointEmbeddings | Embeds a sample text through the endpoint | remote endpoint answers |
HuggingFaceEmbeddings, GPT4AllEmbeddings | The gRPC service loads the model, from its local files first, and embeds a sample text | loaded by the provider service (set HF_HUB_OFFLINE=1 on it to forbid downloads) |
| Text splitter providers | The gRPC service creates the text splitter and splits a sample text | splits a sample text on the provider service |
The following rules apply to the model check:
- Failures. An object that fails prints
- <object>: FAILED: <reason>, and the command goes on with the next object. After the last object, the command printsError: "<count> model(s) cannot be loaded:followed by one line for each failed object, and exits with status 1. A FastEmbed model whose files are missing fails with a reason that names the directories searched. - Downloads. The CLI has no command that downloads model files. Without
HF_HUB_OFFLINE=1on the gRPC service, the gRPC service can still download a Hugging Face model that its local files lack while the command runs. Models, packages and air-gapped installs describes model and package images, which deliver the files to every pod. - One container. The FastEmbed check covers the files that the container running the command sees, and the other checks run in the gRPC service at
grpc.url. The charts mount each model image at the same path in the API server, worker and gRPC service containers. - Errors. Query failures print
Failed to retrieve embedding models: <reason>orFailed to retrieve text splitters: <reason>.
Admin subcommands
The admin subcommands create and change keys that hold every permission, and list the keys whose pipeline grants name no classification. Access control describes the master key, and Secrets and keys describes the master key values of an installation.
Master key creation
admin create-admin-api-key <NAME> [DESCRIPTION] [EXPIRATION] makes the master key match app.master.key and app.master.secret. The master key Job runs the command at every installation and upgrade with the name Master Admin Key and no other argument.
| Argument | Type | Required | Effect |
|---|---|---|---|
NAME | String | Yes | Name of a new key. Every key has a unique name. |
DESCRIPTION | String | No | Description of a new key |
EXPIRATION | Date and time in RFC 3339 form, such as 2027-01-31T00:00:00Z | No | Expiry of a new key. A value without a time or without a UTC offset fails with exit status 2. |
The following rules apply to the master key creation:
- New key. When no key has the identifier
app.master.key, the command creates the key with the arguments and printsMaster key created successfully.When another key already holds the name, such as the previous master key after a change ofapp.master.key, the command first renames that key to<NAME> (replaced by <new identifier>)and printsWarning: key <identifier> held the name "<NAME>" and was renamed; it keeps its admin permissions, so delete it once no client uses it. - Existing key. When the key exists and its stored secret does not match
app.master.secret, the command replaces the stored secret and printsMaster key secret updated to match the configuration.Otherwise the command printsMaster key already matches the configuration.The command leaves the name, the description, the active flag and the expiry of an existing key unchanged; Master key update changes them. - Permissions. The command sets the permissions of the key to read, write and execute on the nine object types that Access control lists, with the allow-list
*, removes grants on other object types, and removes the cached copy of the key from the Redis-compatible cache, so a replaced secret stops authenticating at once. - Expiry. The workers authenticate with the master key at startup, so the master key stays active and unexpired. The master key Job therefore passes no expiry.
- Errors. A connection failure prints the messages of Connections. A key that cannot be stored prints
Error: "Failed creating the master key: <reason>".
Master key update
admin update-admin-api-key [OPTIONS] changes the key with the identifier app.master.key and prints Master key updated successfully. The command takes the following options:
| Option | Effect |
|---|---|
--name <NAME> | Sets the name |
--description <DESCRIPTION> | Sets the description |
--clear-description | Removes the description |
--active <ACTIVE> | Activates the key with true or deactivates the key with false |
--expiration <EXPIRATION> | Sets the expiry, in RFC 3339 form such as 2027-01-31T00:00:00Z |
--clear-expiration | Removes the expiry |
The following rules apply to the master key update:
- Omitted options. An omitted option leaves the value unchanged, and the command without options changes no value.
--descriptionwith--clear-description, or--expirationwith--clear-expiration, fails with exit status 2. - Restoring the master key.
--active truereactivates a deactivated master key, and--clear-expirationremoves an expiry that has passed, so the workers start again and the API server accepts the master key. - Other effects. The command sets the permissions of the master key as
admin create-admin-api-keydoes, and removes the cached copy of the key. The command changes no other property of the master key. - Errors.
Error: "Failed updating the master key: API Key not found: <master key identifier>"means that no key has the identifierapp.master.key.Error: "Failed updating the master key: SQL error: <reason>"means that the change cannot be stored, for example because another key uses the name.
Administration key generation
admin generate-admin-api-key <NAME> [DESCRIPTION] [EXPIRATION] creates a new key with a random identifier, a random secret of 32 letters and digits, and the permissions of the master key. The arguments are those of admin create-admin-api-key:
kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai admin generate-admin-api-key "Operations key" "Administration key for the operations team"
Expected result:
Administration key created successfully.
Key : <key identifier>
Secret: <32-character secret>
The following rules apply to the generated key:
- Single output. The output is the only copy of the secret. The command runs in a session whose output is not recorded, and the secret goes directly to the organization's secret store.
- Use. The new key is an administration key for API requests, as described in Manage API keys and permissions. The new key does not replace the master key, because the workers authenticate with
app.master.keyat startup. - Errors. A connection failure prints the messages of Connections. A key that cannot be stored prints
Error: "Failed creating the administration key: <reason>", and a name that another key uses produces a reason that begins withConflict error (unique):.
Keys without classifications
admin keys-without-classifications lists the active, unexpired keys whose grants on pipelines give read or execute permission but name no classification, so their searches find nothing, as described in Classification allow-lists. The command uses the database-only connection. With no such key, the command prints:
Every active API key that can read or search pipelines names classifications.
Otherwise the command prints one line for each grant:
These active API keys can read or search pipelines but name no classification, so their searches find nothing. Grant them the classifications they need (POST /api-keys/<id>/permissions):
- <key name> (<key identifier>): pipeline <pipeline name> (<pipeline identifier>), access <access level>
- <key name> (<key identifier>): every pipeline without a grant of its own, access <access level>
The second form is a grant on the pipelines object type rather than on one pipeline. Manage API keys and permissions describes how to grant classifications. A failed query prints Error: "Failed to read the API key grants: <reason>".