LLMs

An LLM in Foundation4 is a registered connection to a language model served through an OpenAI-compatible API. Three features use a registered LLM: agents generate answers through the LLM, the direct query endpoint sends a single prompt to the LLM and the OpenAI-compatible endpoint relays chat completion requests to the LLM. This page describes the fields of an LLM registration, the requirements on the model server, the response formats and the OpenAI-compatible endpoint. Integrators and administrators need this page to connect Foundation4 to a model server.

Model servers

Foundation4 works with any model server that exposes an OpenAI-compatible API and that the API server can reach over the network. The model server can be a hosted service or a server inside the customer's network, including a server in an air-gapped network. Foundation4 has no default model server and contacts a model server only when a feature uses a registered LLM.

The API endpoints that Foundation4 calls on the model server depend on the Foundation4 feature:

Foundation4 feature	Model server API	API key
Agents	Responses API	Required
Direct query	Responses API	Required
OpenAI-compatible endpoint	Chat completions (/chat/completions)	Optional

A model server that implements only chat completions therefore serves the OpenAI-compatible endpoint but not agents or direct queries. Agents and direct queries also require an API key in the registration, even when the model server does not check keys. Any non-empty value satisfies this requirement for a model server without authentication.

LLM registration

An LLM is registered with POST /llms. The request contains the following fields.

Field	Required	Description
name	Yes	A name that is unique across the deployment
endpoint	Yes	The base URL of the OpenAI-compatible API, without a trailing slash, such as http://llm.internal:8000/v1. Foundation4 appends the path of each API to this URL.
model	No	The model name that Foundation4 sends to the model server. When omitted, Foundation4 sends an empty model name.
api_key	No	The API key that Foundation4 sends to the model server as a bearer token. Required for agents and direct queries.
description	No	Free text for administrators

The following request registers a model served inside the customer's network. The shell variables in the examples are described in Authenticate.

curl -X POST "$FOUNDATION4_URL/llms" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "internal-general",
    "description": "General-purpose model on the internal model server",
    "endpoint": "http://llm.internal:8000/v1",
    "model": "example-model-8b-instruct",
    "api_key": "change-me"
  }'

The response has status 201 and contains the registration without the API key:

{
  "id": "3f0c9b6e-2d4a-4c1e-9b7f-5a8d2e6c1f04",
  "name": "internal-general",
  "description": "General-purpose model on the internal model server",
  "endpoint": "http://llm.internal:8000/v1",
  "model": "example-model-8b-instruct"
}

Registration does not contact the model server, so an incorrect endpoint or key appears only when the LLM is first used. The direct query described below tests a new registration.

Foundation4 stores no generation settings with an LLM. The temperature is set on each agent execution and direct query, and every other setting takes the model server's default.

PATCH /llms/{id} changes any field of a registration. A request with a new api_key value replaces the key, and a request with "api_key": null removes the key. DELETE /llms/{id} removes the registration. Agents do not store an LLM, so deleting a registration changes no agent; executions that name the deleted LLM fail.

API key storage

Foundation4 encrypts the API key of each LLM with the application secret of the deployment before storing the key in PostgreSQL. No endpoint returns the key. The application secret is therefore required to use stored keys: if the application secret changes or is lost, the stored keys cannot be decrypted and the API key of each LLM must be set again. Data protection describes the application secret.

Direct query

POST /llms/{id}/query sends one prompt to the LLM as a single user message, with no system message and no retrieval. The direct query serves to test a registration and to generate text that needs no context from a pipeline. The request contains the following fields.

Field	Required	Description
query	Yes	The prompt text
stream	No	The response format, described in Response formats. The default is newline-delimited JSON.
temperature	No	The sampling temperature. When omitted, the model server's default applies.

The following request tests a registration and waits for the complete answer:

curl -X POST "$FOUNDATION4_URL/llms/$LLM_ID/query" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{"query": "In one sentence, what is a text splitter?", "stream": false}'

The response has status 200 and contains the answer as plain text. A request with stream set to false returns HTTP 500 when the model server fails, with the model server's error in the message. Streamed responses cannot report such failures after the response has started, so false is the setting for testing a new registration.

Response formats

Agent executions and direct queries share three response formats, selected by the stream field of the request.

stream	Format	Content
Omitted, true or "stream"	Newline-delimited JSON (NDJSON)	One JSON object per line, each with a content field that holds the next part of the answer
"sse"	Server-sent events	One event per part of the answer, with the same JSON object as data. A keep-alive comment is sent after each second without an event.
false	Plain text	The complete answer in one response body, sent after generation completes

The following lines show the start of an NDJSON response:

{"content":"A text splitter"}
{"content":" divides a document"}
{"content":" into fragments."}

The following rules apply to both streamed formats:

• End of the answer. The response ends when the answer is complete. No closing event or summary line is sent.
• Empty parts. A line or event can carry an empty content value. Client applications skip empty values.
• Status code. The status code is 200 once the response has started. A failure of the model server during generation ends an NDJSON response early without an error message, and closes a server-sent events response before the response is complete.

OpenAI-compatible endpoint

POST /openai/v1/chat/completions relays a chat completion request to a registered LLM. Client applications and software development kits (SDKs) written for the OpenAI chat completions API can therefore use a model server through Foundation4, with Foundation4 authentication and permissions, while the model server's key stays inside Foundation4.

The request carries the Foundation4 API key headers and the x-llm-id header, which names the registered LLM. The body is a chat completion request in the OpenAI format. Foundation4 processes the request as follows:

• Body. Foundation4 forwards the body unchanged to the chat completions path of the LLM's endpoint. When the body has no model field, Foundation4 inserts the registered model name; a model field in the body takes precedence over the registered model.
• Authentication to the model server. Foundation4 sends the registered API key as a bearer token when the registration has a key. A bearer token sent by the client application is not forwarded.
• Response. Foundation4 returns the model server's status code and body. A non-streamed response also carries the model server's headers. A streamed response, requested with "stream": true, is relayed as server-sent events exactly as the model server sends them.

curl -X POST "$FOUNDATION4_URL/openai/v1/chat/completions" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "x-llm-id: $LLM_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "Answer in one sentence."},
      {"role": "user", "content": "What is retrieval-augmented generation?"}
    ],
    "temperature": 0.2
  }'

The endpoint performs no retrieval, prompt templating or tracing. Chat completions is the only OpenAI API that Foundation4 relays; paths such as /openai/v1/models do not exist. A request that carries any of the headers X-Agent-Id, X-Pipeline-Id or X-Pipeline-Classification returns an error.

Permissions

Registering an LLM requires write permission on LLMs. Agent executions, direct queries and the OpenAI-compatible endpoint require execute permission on the LLM that the request names. Execute permission on an LLM allows any model name that the model server accepts, because a model field in a relayed request takes precedence over the registered model. Access control describes permissions.