Skip to main content

Command-line interface

This page is the reference for foundation4ai, the command-line interface (CLI) of Foundation4. The page describes the global options, the configuration and the connections that the CLI uses, and the arguments, output, errors and exit status of every subcommand. Operators use this page to run maintenance commands in an installation and to read the logs of the installation Jobs. Operator command reference lists the subcommands next to the other operator commands.

Synopsis​

foundation4ai [-h | --help] [-V | --version]
foundation4ai <command> [<subcommand>] [<arguments>]
CommandArgumentsPurposeConnection
serverNoneRuns the API server. The binary without a command runs the same process.Full
workerNoneRuns a workerFull
database migrateNoneApplies pending database migrationsDatabase only
database check-connectionNoneTests the database connectionDatabase only
license system-idNonePrints the system IDDatabase only
license checkNoneChecks the configured licenseDatabase only
license details<LICENSE>Prints the contents of a licenseDatabase only
data providersNoneLists the providers of the configured embedding models and text splittersDatabase only
data check-modelsNoneChecks that the configured embedding models and text splitters loadFull
admin create-admin-api-key<NAME> [DESCRIPTION] [EXPIRATION]Creates the master key, or updates its secret to match the configurationDatabase and cache
admin update-admin-api-keyOptions, as described in Master key updateChanges the name, the description, the active flag or the expiry of the master keyDatabase and cache
admin generate-admin-api-key<NAME> [DESCRIPTION] [EXPIRATION]Creates an additional administration keyDatabase and cache
admin keys-without-classificationsNoneLists the active keys that can read or search pipelines but name no classificationDatabase only
help[<command>] [<subcommand>]Prints the help of the binary or of a commandNone

Arguments are positional, except the options of admin update-admin-api-key. Angle brackets mark a required argument, and square brackets mark an optional argument. An argument that contains spaces is quoted, as in "Master Admin Key". The other subcommands accept no option other than -h and --help, and no subcommand reads standard input or asks for confirmation. Connections describes the three connection types.

Invocation​

The API server image contains the binary as /app/foundation4ai, and /app is the working directory of the API server, worker and installation Job containers. Operators run subcommands in the server container of a running API server pod:

kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai license system-id

Expected result: the output of the subcommand, here one line of 64 hexadecimal characters. kubectl exec returns the exit status of the subcommand and prints command terminated with exit code <status> for any status other than 0.

The binary runs in the following places:

  • Containers. The server container runs ./foundation4ai server, and the worker container runs ./foundation4ai worker. Operators do not start either subcommand through kubectl exec, because the containers already run both processes.
  • Installation Jobs. The application release runs database migrate, license check and admin create-admin-api-key "Master Admin Key" as installation Jobs at every installation and upgrade, in the order described in Install on Kubernetes.
  • Image default. A container started from the API server image without arguments runs the binary without a command, which runs the API server as server does. The charts always pass a command.

Global options​

OptionEffectExit status
-h, --helpPrints the help of the binary, or of the command or subcommand before the option, on standard output0
-V, --versionPrints foundation4ai <version>, the version of the program package, such as foundation4ai 1.1.00
help [<command>] [<subcommand>]Prints the same help as --help0

The version of the program package names the Foundation4 release. The 1.1.0 images print the release only, so the tag of the API server image is what tells apart the builds of one release, as described in Charts, images and installation bundle. A binary built with a build identifier adds a plus sign and the identifier, as in foundation4ai 1.1.0+<build>.

./foundation4ai --help prints the following text:

Foundation4.ai API Server

Usage: foundation4ai [COMMAND]

Commands:
admin Admin commands
license License commands
database Database operations
data Data operations
server Run the API Server
worker Run the Worker
help Print this message or the help of the given subcommand(s)

Options:
-h, --help Print help
-V, --version Print version

The following rules apply to the command line:

  • Command groups. admin, license, database and data are groups of subcommands. A group without a subcommand prints the help of the group on standard error and exits with status 2.
  • No command. The binary without a command runs the API server, as server does.
  • Unknown names. An unknown command or subcommand prints error: unrecognized subcommand '<name>', followed by similar names when a similar name exists, and exits with status 2.

Configuration​

Every subcommand reads the configuration from the working directory in the order that Configuration reference describes: defaults.yaml, config/settings.yaml, the config/f4ai-*.yaml files in name order, the files under config/secrets named by key path, then the FOUNDATION4AI__<SECTION>__<KEY> environment variables. The configuration directory is fixed at ./config, and the CLI has no option that changes the directory or a configuration key.

The following rules apply to the configuration of the CLI:

  • Complete configuration. Every subcommand parses the whole configuration, including the keys that the subcommand does not use. A missing or invalid required key stops the subcommand with a message such as Error: "Failed to parse config: missing field `license` for key \"default.app\"".
  • Start time. A subcommand reads the configuration when the subcommand starts, and the API server and worker processes read the configuration when the processes start. Configuration change describes the restart that applies a change to the running processes.
  • Database settings. The database-only subcommands and the admin subcommands use database.url, database.ca_cert, database.schema and database.embeddings_schema, and ignore database.pool_size. They use the schemas of the API server, so database migrate creates the tables in database.schema, and the license subcommands compute the system ID for that schema, as described in Licensing. With database.ca_cert, the sslmode value require also verifies the server certificate against the certificate authority (CA) certificate, as the connection of the API server does.
  • Worker pool. worker uses a database pool of 1 connection, whatever the value of database.pool_size.
  • Log level. app.log_level applies to server and worker. The other subcommands write log lines to standard error at the level info and above, or at app.log_level when that level is debug or trace, next to the output that this page lists. The warning with the system ID that the full connection logs after a license failure therefore appears in the output of data check-models.

Connections​

Each subcommand uses one of three connection types:

  • Database only. database migrate, database check-connection, the three license subcommands, data providers and admin keys-without-classifications open one PostgreSQL connection with the database settings of Configuration. These subcommands report a connection failure once, as Error: "Failed connecting to database: <reason>", where the reason tells apart name resolution, authentication, Transport Layer Security (TLS) and access rule failures.
  • Database and cache. admin create-admin-api-key, admin update-admin-api-key and admin generate-admin-api-key open a PostgreSQL connection as the database-only subcommands do, and connect to the Redis-compatible cache. These subcommands need no license, no NATS JetStream and no application secret. While PostgreSQL or the cache does not answer, the subcommands log Waiting for the database or cache; retrying in <delay> and try again, with a delay that doubles up to 10 seconds, for about four minutes. The last failure prints Error: "Failed connecting to database: <reason>" or Error: "Failed connecting to the cache: <reason>".
  • Full connection. server, worker and data check-models build the same connection as the API server. The full connection needs PostgreSQL with the vector, vchord, pg_tokenizer and vchord_bm25 extensions, a valid license, the Redis-compatible cache, NATS JetStream and a valid application secret.

The full connection runs the following steps in order, and the first failure stops the subcommand:

  1. Connects to PostgreSQL with the pool size, the schemas and the CA certificate of the configuration, creates missing schemas and applies pending migrations.
  2. Computes the system ID from database.schema and checks app.license.
  3. Connects to the Redis-compatible cache.
  4. Connects to NATS JetStream, and creates the object store nats.bucket and the stream nats.stream when missing.
  5. Decodes the application secret app.secret.
  6. Prepares the address of the gRPC service in grpc.url. The connection to the gRPC service opens on first use.

While PostgreSQL, the cache or NATS JetStream does not answer, the full connection logs Waiting for the database, cache or queue; retrying in <delay> and starts again, with a delay that doubles up to 10 seconds, for about four minutes. A license failure and the other failures, such as a failed migration, stop the subcommand without a retry.

data check-models reports a license failure as Error: "License check failed: <reason>" and a failure of another step as Error: "Failed to connect to Foundation4.ai's services: <reason>". The beginning of the reason names the step:

Reason begins withStep
ORM error:, The database was migrated by a newer Foundation4 releasePostgreSQL connection or migration
Invalid license, Expired license, Error computing system idLicense check
Redis error:, Invalid parameter: Invalid Redis URLRedis-compatible cache
Nats error:NATS JetStream
Decryption error: Error decoding secret keyApplication secret
gRPC error:Address of the gRPC service

Before these steps, the full connection reads the four paths keys, and a missing key stops the subcommand with Error: "missing field `<key>`". server prints these messages after its own prefix, as described in Server, and Troubleshooting lists the causes.

Output and exit status​

Subcommands print results on standard output and errors on standard error:

Exit statusMeaningStandard error
0The subcommand completed, or the help or version was printed on requestEmpty, or the log lines described in Configuration
1The subcommand failedError: "<message>"
2The command line is invalid, or names a group without a subcommanderror: <reason> followed by the usage, or the help
134The process stopped on an internal error with the abort signalA message that contains panicked at

The following rules apply to the output:

  • Quoted messages. The message after Error: is printed as a quoted string, so a quotation mark inside the message appears as \" and a line break as \n.

  • License results. license check and license details exit with status 1 when the license cannot be used, so scripts can read the exit status.

  • Argument errors. An invalid argument prints the reason and a pointer to the help, as in the following output for an expiry without a time:

    error: invalid value '2027-01-31' for '[EXPIRATION]': premature end of input

    For more information, try '--help'.

Process subcommands​

Server​

server runs the API server. The subcommand builds the full connection, checks the master key, serves the metrics on app.metrics_port and listens on app.host and app.port. The process logs at app.log_level, and at the level info the log shows Starting <branding.name> API on <host>:<port>. On the interrupt signal or the termination signal, the API server stops accepting connections, completes open requests and exits with status 0.

A master key that is expired, inactive or missing does not stop the API server: the API server logs an error that names admin update-admin-api-key --clear-expiration, and refuses requests made with the master key until the key is restored. A master key secret that does not match the stored key stops the API server.

The API server reports connection and master key failures after Error: "Failed to initialize server: , such as Failed to initialize server: Failed to initialize with the master key: <reason>. A listen address that cannot be used prints Failed to bind to address: <reason>, or Failed to bind the metrics address: <reason> for the metrics port. Troubleshooting lists the messages with the causes.

Worker​

worker runs a worker. The subcommand opens the metrics listener on port 9090, builds the full connection with a database pool of 1 connection, and authenticates with the master key. After a wait of 2 seconds, the worker logs Starting Foundation4.ai worker at the level info and fetches processing jobs from NATS JetStream in batches of up to 100 jobs.

The following rules apply to the worker:

  • Stop signal. On the interrupt signal or the termination signal that Kubernetes sends, the worker completes the batch in progress, logs Received shutdown signal, stopping worker and exits with status 0.
  • Startup errors. The worker reports connection failures with the messages of Connections, such as License check failed: <reason>, master key failures as Failed to initialize Foundation4.ai: <reason>, and queue consumer failures as Error creating message queue consumer: <reason>. An expired or inactive master key therefore stops the worker. When port 9090 is already in use, the worker stops with Error: "Failed to install metrics recorder: <reason>".
  • Queue errors. A failed read from the queue logs Error retrieving messages from message queue, and the worker tries again after 2 seconds.
  • License expiry. The worker checks the license before each batch. When the license is no longer valid, for example after its expiry date, the worker logs The license is no longer valid: the worker stops; queued documents wait for a renewed license and stops with Error: "License check failed: <reason>". A restarted worker does not start until a valid license is installed.

Database subcommands​

Database migration​

database migrate applies the pending migrations of the database schema, records each migration in the table schema_migrations, and prints Migrated to latest schema. The migration Job runs the command before the license check at every installation and upgrade, and the API server, the workers and data check-models also apply pending migrations when they build the full connection.

The following rules apply to the migration:

  • Schemas. The command creates the schemas database.schema and database.embeddings_schema when missing, and applies the migrations in database.schema, as the API server does. A schema that cannot be created stops the command with Error: "Failed creating schema <schema>: <reason>".
  • Earlier table format. When database.schema holds a table named schema_migrations with exactly one column, and every row of the table names a Foundation4 migration, the command drops that table before applying the migrations, because the table then has the format of an earlier Foundation4 version. When the table holds other rows, such as those of another migration tool, the command leaves the table in place and stops with Error: "Failed applying migrations: the schema has a schema_migrations table of another tool. Give Foundation4 its own schema (database.schema).".
  • Direction. Each migration runs in one transaction, and migrations only move forward. The CLI has no command that reverts a migration, as described in Database. A database that a newer Foundation4 release migrated stops the command with Error: "Failed applying migrations: the database was migrated by a newer Foundation4 release (<reason>). Run the newer release, or restore the backup taken before the upgrade, then roll back.".
  • Extensions. When a migration fails on a database extension, the message ends with a line that names the vector, vchord, pg_tokenizer and vchord_bm25 extensions, which only a superuser can create, and the bundled PostgreSQL image, which has them.
  • Errors. Error: "Failed connecting to database: <reason>" means that the connection failed. Error: "Failed applying migrations: <reason>" means that the connection succeeded and a migration failed, as described in Troubleshooting.

Database connection check​

database check-connection opens a connection with the database settings of Configuration and prints Ok. The check covers the address, the credentials and the TLS settings. The check does not cover the schemas, the migrations or the database extensions. A failure prints Error: "Failed connecting to database: <reason>", where the reason names the failure, such as a host name that cannot be resolved, a rejected password or a certificate error. Operator command reference tests the same connection with a PostgreSQL client.

License subcommands​

The license subcommands compute the system ID for the database in database.url and the schema in database.schema, as the API server does. license system-id works before the first migration. The computation reads pg_control_system(), and a database user who cannot read it makes license system-id fail with Error: "Failed generating system id: the database user cannot read pg_control_system()". Licensing describes licenses and the system ID.

System ID​

license system-id prints the system ID as one line of 64 hexadecimal characters. The Foundation4 provider issues a license for this value, as described in Licensing.

License check​

license check checks the license in app.license against the system ID and prints two lines:

SystemID: <system ID>
Checking for a valid license... OK

When the system ID cannot be computed, the first line reads SystemID: unavailable (<reason>). The second line ends with one of the following results:

ResultMeaningExit status
OKThe license is valid for the system ID and has not expired0
Missingapp.license is an empty string1
InvalidThe license is malformed, of an unsupported version, or issued for another system ID1
ExpiredThe expiry date of the license has passed1
The system id for this license version could not be computedThe database does not provide the values that the license version binds to1

A result other than OK also prints Error: "License check failed: <result>" on standard error. The license check Job therefore fails at once, and its log keeps the system ID of the first line, so the application installation stops with a failed hook, as described in Troubleshooting. Under the charts, an empty license value in the secrets file reaches the check and reads as Missing.

License details​

license details <LICENSE> checks the license given as the argument against the system ID and prints the contents of the license. The license is one argument in quotes:

kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai license details '<license>'

Expected result for a license that matches the database:

License: <license holder> (expires on <date> <time> UTC)
System ID: <system ID>
Version: <license version>
Limits:
Pipelines: <limit>
Documents: <limit>

A license without an expiry date shows (perpetual), and optional Description: and Environment: lines appear when the license carries these fields. Licensing describes every line. For a license that cannot be used, the command prints Error: "<result>" on standard error instead of the contents, with one of the results of License check, and exits with status 1. Operator command reference reads the license value from the secrets file.

Data subcommands​

Provider list​

data providers lists the distinct providers of the embedding models and of the text splitters in the database, in no fixed order. A database with only the seeded objects of Providers and models produces the following output:

Found embedding providers:
- FastEmbedEmbeddings

Text splitter providers in use:
- RecursiveCharacterTextSplitter
- CharacterTextSplitter
- NLTKTextSplitter

The list names the providers that configured objects use, not every provider that Foundation4 supports. Providers and models lists the supported providers.

Model check​

data check-models builds the full connection and checks that every configured embedding model and text splitter loads and works, as the API server and the workers load them. The command downloads no FastEmbed model file, so a FastEmbed model that passes the check loads without network access. A database with only the seeded objects of Providers and models and the model images in place produces the following output:

Checking that the configured models load. FastEmbed models are checked in this container without downloading; other providers are checked by the provider service at <grpc.url>.

Embedding models:
- all-MiniLM-L6-v2 (FastEmbedEmbeddings, Qdrant/all-MiniLM-L6-v2-onnx): ok, loads offline from <directory>

Text splitters:
- CharacterTextSplitter (<identifier>): ok, splits a sample text on the provider service
- NLTKTextSplitter (<identifier>): ok, splits a sample text on the provider service
- RecursiveCharacterTextSplitter (<identifier>): ok, splits a sample text on the provider service

Every configured model loads.

The command lists embedding models by name and text splitters by provider. Each line ends with the way the object was checked:

ProviderCheckLine ending
FastEmbedEmbeddingsFinds the model files in paths.fastembed or in the writable cache of the container that runs the command, then loads the model and embeds a sample textloads offline from <directory>
OpenAIEmbeddings, HuggingFaceEndpointEmbeddingsEmbeds a sample text through the endpointremote endpoint answers
HuggingFaceEmbeddings, GPT4AllEmbeddingsThe gRPC service loads the model, from its local files first, and embeds a sample textloaded by the provider service (set HF_HUB_OFFLINE=1 on it to forbid downloads)
Text splitter providersThe gRPC service creates the text splitter and splits a sample textsplits a sample text on the provider service

The following rules apply to the model check:

  • Failures. An object that fails prints - <object>: FAILED: <reason>, and the command goes on with the next object. After the last object, the command prints Error: "<count> model(s) cannot be loaded: followed by one line for each failed object, and exits with status 1. A FastEmbed model whose files are missing fails with a reason that names the directories searched.
  • Downloads. The CLI has no command that downloads model files. Without HF_HUB_OFFLINE=1 on the gRPC service, the gRPC service can still download a Hugging Face model that its local files lack while the command runs. Models, packages and air-gapped installs describes model and package images, which deliver the files to every pod.
  • One container. The FastEmbed check covers the files that the container running the command sees, and the other checks run in the gRPC service at grpc.url. The charts mount each model image at the same path in the API server, worker and gRPC service containers.
  • Errors. Query failures print Failed to retrieve embedding models: <reason> or Failed to retrieve text splitters: <reason>.

Admin subcommands​

The admin subcommands create and change keys that hold every permission, and list the keys whose pipeline grants name no classification. Access control describes the master key, and Secrets and keys describes the master key values of an installation.

Master key creation​

admin create-admin-api-key <NAME> [DESCRIPTION] [EXPIRATION] makes the master key match app.master.key and app.master.secret. The master key Job runs the command at every installation and upgrade with the name Master Admin Key and no other argument.

ArgumentTypeRequiredEffect
NAMEStringYesName of a new key. Every key has a unique name.
DESCRIPTIONStringNoDescription of a new key
EXPIRATIONDate and time in RFC 3339 form, such as 2027-01-31T00:00:00ZNoExpiry of a new key. A value without a time or without a UTC offset fails with exit status 2.

The following rules apply to the master key creation:

  • New key. When no key has the identifier app.master.key, the command creates the key with the arguments and prints Master key created successfully. When another key already holds the name, such as the previous master key after a change of app.master.key, the command first renames that key to <NAME> (replaced by <new identifier>) and prints Warning: key <identifier> held the name "<NAME>" and was renamed; it keeps its admin permissions, so delete it once no client uses it.
  • Existing key. When the key exists and its stored secret does not match app.master.secret, the command replaces the stored secret and prints Master key secret updated to match the configuration. Otherwise the command prints Master key already matches the configuration. The command leaves the name, the description, the active flag and the expiry of an existing key unchanged; Master key update changes them.
  • Permissions. The command sets the permissions of the key to read, write and execute on the nine object types that Access control lists, with the allow-list *, removes grants on other object types, and removes the cached copy of the key from the Redis-compatible cache, so a replaced secret stops authenticating at once.
  • Expiry. The workers authenticate with the master key at startup, so the master key stays active and unexpired. The master key Job therefore passes no expiry.
  • Errors. A connection failure prints the messages of Connections. A key that cannot be stored prints Error: "Failed creating the master key: <reason>".

Master key update​

admin update-admin-api-key [OPTIONS] changes the key with the identifier app.master.key and prints Master key updated successfully. The command takes the following options:

OptionEffect
--name <NAME>Sets the name
--description <DESCRIPTION>Sets the description
--clear-descriptionRemoves the description
--active <ACTIVE>Activates the key with true or deactivates the key with false
--expiration <EXPIRATION>Sets the expiry, in RFC 3339 form such as 2027-01-31T00:00:00Z
--clear-expirationRemoves the expiry

The following rules apply to the master key update:

  • Omitted options. An omitted option leaves the value unchanged, and the command without options changes no value. --description with --clear-description, or --expiration with --clear-expiration, fails with exit status 2.
  • Restoring the master key. --active true reactivates a deactivated master key, and --clear-expiration removes an expiry that has passed, so the workers start again and the API server accepts the master key.
  • Other effects. The command sets the permissions of the master key as admin create-admin-api-key does, and removes the cached copy of the key. The command changes no other property of the master key.
  • Errors. Error: "Failed updating the master key: API Key not found: <master key identifier>" means that no key has the identifier app.master.key. Error: "Failed updating the master key: SQL error: <reason>" means that the change cannot be stored, for example because another key uses the name.

Administration key generation​

admin generate-admin-api-key <NAME> [DESCRIPTION] [EXPIRATION] creates a new key with a random identifier, a random secret of 32 letters and digits, and the permissions of the master key. The arguments are those of admin create-admin-api-key:

kubectl exec -n foundation4ai deploy/foundation4ai-api-server -c server -- ./foundation4ai admin generate-admin-api-key "Operations key" "Administration key for the operations team"

Expected result:

Administration key created successfully.
Key : <key identifier>
Secret: <32-character secret>

The following rules apply to the generated key:

  • Single output. The output is the only copy of the secret. The command runs in a session whose output is not recorded, and the secret goes directly to the organization's secret store.
  • Use. The new key is an administration key for API requests, as described in Manage API keys and permissions. The new key does not replace the master key, because the workers authenticate with app.master.key at startup.
  • Errors. A connection failure prints the messages of Connections. A key that cannot be stored prints Error: "Failed creating the administration key: <reason>", and a name that another key uses produces a reason that begins with Conflict error (unique):.

Keys without classifications​

admin keys-without-classifications lists the active, unexpired keys whose grants on pipelines give read or execute permission but name no classification, so their searches find nothing, as described in Classification allow-lists. The command uses the database-only connection. With no such key, the command prints:

Every active API key that can read or search pipelines names classifications.

Otherwise the command prints one line for each grant:

These active API keys can read or search pipelines but name no classification, so their searches find nothing. Grant them the classifications they need (POST /api-keys/<id>/permissions):
- <key name> (<key identifier>): pipeline <pipeline name> (<pipeline identifier>), access <access level>
- <key name> (<key identifier>): every pipeline without a grant of its own, access <access level>

The second form is a grant on the pipelines object type rather than on one pipeline. Manage API keys and permissions describes how to grant classifications. A failed query prints Error: "Failed to read the API key grants: <reason>".