Skip to main content

Agents and prompt templates

An agent is a stored prompt template that combines retrieval from a pipeline with a call to a registered LLM, which produces an answer grounded in the pipeline's content. This retrieval-augmented generation (RAG) pattern is the basis of question answering over a knowledge base. This page describes the prompt template, the placeholders that insert request values and retrieved fragments, the execution of an agent, the tools for testing an agent and the attribution of an answer to the retrieved fragments. Integrators need this page to design agents and to call agents from a client application.

Agent definition​

An agent is created with POST /agents. The request contains the following fields.

FieldRequiredDescription
nameYesA name that is unique across the deployment
descriptionNoFree text for administrators
promptYesThe prompt template: an ordered list of messages
placeholdersYesThe named values that the templates insert: request values and retrieval results
metadataNoDescriptive data for the client application. Each top-level value is a JSON object. Foundation4 stores metadata and does not use metadata during execution.

An agent stores no pipeline and no LLM. Each execution names the pipeline and the LLM, so one agent can serve several pipelines and several models.

Prompt template​

The prompt is a list of messages in the order that the model receives them. Each message has the following fields:

  • role. system for instructions to the model, user for the request that the model answers, or assistant for an example answer.
  • template. The text of the message, with placeholders written as the placeholder name in braces, such as {question}.
  • include. Whether the message is sent to the model. The default is true. A message with false remains part of the stored agent but is never sent.

A prompt contains at least one system message and at least one user message, and the last message is a user message. Every name in braces in a template must be a placeholder of the agent. Foundation4 provides no built-in variables such as {context}; each name refers to a placeholder that the agent defines. Foundation4 rejects an agent that breaks one of these rules with HTTP 400. A template reads a name in braces only when the name consists of letters, digits and underscores and starts with a letter, such as {user_question}. Foundation4 does not check the names of placeholders against this rule, so a placeholder named user-question is stored, but {user-question} in a template is not read as a placeholder and never receives the value.

Placeholders​

A placeholder is a named value that Foundation4 inserts into the templates at execution time. Each placeholder has a name, a type and, depending on the type, a target and params.

TypeValue insertedFields
queryA value that the client application supplies in each execution request, such as the end user's questionname, type
similarityThe text of the fragments that a similarity search returnsname, type, target, params
mmrThe text of the fragments that a maximal marginal relevance (MMR) search returnsname, type, target, params
searchThe text of the fragments that a full-text search of the pipeline's default text index returnsname, type, target, params
hybridThe text of the fragments that a hybrid search of the pipeline's default vector index and default text index returnsname, type, target, params
historyThe text of the most recent fragments of the pipelinename, type, params

A retrieval placeholder (similarity, mmr, search or hybrid) names a query placeholder in target. At execution, Foundation4 uses the value of that query placeholder as the search text, so the end user's question both appears in the prompt and selects the fragments. Several retrieval placeholders can target the same query placeholder or different query placeholders.

Each placeholder has a name that no other placeholder of the agent uses. A retrieval placeholder requires a target, and a query or history placeholder takes none. Foundation4 rejects an agent that breaks one of these rules when the agent is created or changed, with HTTP 400, the message Invalid placeholder <name> and the error details placeholder, the placeholder name, and error, the reason, such as another placeholder has the same name.

The params object of a placeholder accepts the parameters of the corresponding search type, with the same defaults as described in Search and retrieval:

ParameterPlaceholder typesDefaultDescription
embeddingSimilarity, MMRThe model of the pipeline's default vector indexThe identifier of an embedding model. The placeholder searches the pipeline's ready vector index for that model: the default vector index when the default uses the model, otherwise a VectorChord index before a pgvector index.
kAll except query5The number of fragments to insert
fetch_kMMR10The number of candidates from which MMR selects
lambda_multMMR0.5The balance between relevance and diversity, from 0 to 1
thresholdSimilarity, MMR, searchNoneThe maximum distance of a fragment from the search text, or for search the minimum score

A parameter that a placeholder omits takes the default. A parameter that the placeholder type does not accept, such as threshold on a hybrid placeholder, returns HTTP 422, as does a field that the agent definition does not define. Foundation4 rejects a placeholder with k of 0, fetch_k of 0 or less than k, or lambda_mult outside 0 to 1, with HTTP 400. A placeholder cannot name a search index in index or indexes, because one agent runs against any pipeline that an execution request names; Foundation4 rejects such a placeholder with HTTP 400 as well. Retrieval in agents uses the default distance strategy of the vector index, which is cosine for a new index.

A search or hybrid placeholder requires the pipeline of the execution to have a default text index, and for hybrid also a default vector index. An execution against a pipeline without the required default returns HTTP 400. A similarity or mmr placeholder without embedding requires a default vector index, and an execution against a pipeline without one returns HTTP 400 with Missing embedding model ID for placeholder: <name>. Any other retrieval error returns the status that the same search returns on the pipeline, such as HTTP 409 for a search index that is still building. Search indexes describes the default indexes.

Foundation4 inserts the text of the retrieved fragments in rank order, separated by line breaks. The inserted text carries no fragment identifiers, scores or metadata. A template that needs the model to distinguish sources therefore labels the section that holds the placeholder, as in the example below.

A history placeholder inserts the most recent fragments of the pipeline, not the conversation with the end user. Agents are stateless, so a client application that continues a conversation supplies the earlier turns through a query placeholder.

The shell variables in the examples are described in Authenticate. The following request creates a question-answering agent with one query placeholder and one similarity placeholder:

curl -X POST "$FOUNDATION4_URL/agents" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "Content-Type: application/json" \
-d '{
"name": "support-answers",
"description": "Answers support questions from the product knowledge base",
"prompt": [
{
"role": "system",
"template": "Answer using only the reference text. If the reference text does not contain the answer, say that the answer is not available."
},
{
"role": "user",
"template": "Reference text:\n{context}\n\nQuestion: {question}"
}
],
"placeholders": [
{"name": "question", "type": "query"},
{"name": "context", "type": "similarity", "target": "question", "params": {"k": 5}}
]
}'

The response has status 201 and contains the stored agent, including the generated id, the include value of each message and every parameter of each retrieval placeholder.

Execution​

POST /agents/{id}/execute runs an agent. The request carries two headers in addition to the API key headers:

  • x-pipeline-id. The pipeline that the retrieval placeholders search. One execution searches one pipeline.
  • x-llm-id. The registered LLM that generates the answer. The LLMs page describes the requirements on the model server.

The request body contains the following fields.

FieldRequiredDescription
promptYesAn object that maps the name of each query placeholder to the value for this execution. Every query placeholder must have a value.
classificationYesThe classifications that the reader holds, in any form that search accepts. The Classifications page describes the forms.
filtersNoAn object that maps the name of a retrieval placeholder to a metadata filter for that placeholder. The Metadata, filters and taxonomies page describes the filter language.
pathNoA path pattern that applies to every retrieval placeholder, such as kb.billing.>. Without the field, the placeholders retrieve only from documents without a path. Document paths describes the patterns.
streamNoThe response format. The default is newline-delimited JSON.
tracingNoRecords a trace of the execution when true. The default is false.
attributionsNoAttributes each sentence of the answer to the fragments that support the sentence, in the trace, as described in Answer attributions. true also records a trace. The default is false.
temperatureNoThe sampling temperature. When omitted, the model server's default applies.

Filters apply per placeholder, so one agent can retrieve from different subsets of a pipeline in one execution. For example, one placeholder can retrieve from product documentation and a second placeholder from release notes, each with a filter on a source metadata field.

The following request runs the agent created above and waits for the complete answer:

curl -X POST "$FOUNDATION4_URL/agents/$AGENT_ID/execute" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
-H "x-pipeline-id: $PIPELINE_ID" \
-H "x-llm-id: $LLM_ID" \
-H "Content-Type: application/json" \
-d '{
"prompt": {"question": "How do I rotate the database credentials?"},
"classification": ["internal"],
"stream": false,
"tracing": true,
"temperature": 0.2
}'

The response has status 200 and contains the answer as plain text. The streamed formats and the handling of failures are the same as for direct queries, as described in Response formats on the LLMs page. A request that omits the value of a query placeholder returns HTTP 400, with the missing placeholder names in the error details. Executions and retrieval dry runs require a valid license, as direct queries do, and return HTTP 403 with Invalid license: <reason> otherwise.

Execution path​

The API server runs the retrieval placeholders in the order of the placeholders list. Each retrieval applies the classifications of the request and the filter for that placeholder, as a search request does. The API server then inserts the input values and the fragment text into the templates, removes messages with include set to false, and sends the messages to the model server. The system messages become the instructions of the model, the other messages form the conversation in order, and the last message is the request that the model answers.

The API server fills each placeholder of a template once. Text in braces inside an input value or a retrieved fragment, such as {id}, stays unchanged in the messages.

Stateless execution​

Agents keep no conversation state. Each execution is independent, and Foundation4 stores nothing about an execution except the optional trace. A client application that continues a conversation supplies the earlier turns in the input values of each execution, for example through a query placeholder that holds the conversation so far.

Testing and tracing​

Three endpoints support the design and debugging of an agent:

  • Prompt inspection. GET /agents/{id}/prompt returns the stored messages, the placeholders and input_variables, the names of the query placeholders that each execution must supply.
  • Retrieval dry run. POST /agents/{id}/search runs the retrieval placeholders without calling a model. The request takes the x-pipeline-id header and the prompt, classification, filters and path fields of an execution. The response maps each retrieval placeholder name to the fragments that the placeholder retrieves, with scores.
  • Execution trace. A successful execution with tracing or attributions set to true returns the trace identifier in the x-foundation4ai-tracing-id response header. A failed execution, such as one that returns HTTP 502, returns no header. GET /tracing/{id} returns the trace: the input values in inputs, the fragments retrieved for each placeholder in traces and the messages exactly as sent to the model in prompt. With attributions, the trace also holds the answer in answer and the attributions in attributions once the answer is complete.

A trace is retained for 60 minutes and can be read only with the API key that ran the execution. A trace holds the decrypted text of the retrieved fragments, encrypted with the application secret while stored.

Answer attributions​

The API server inserts the retrieved fragments into the prompt without labels, so the answer of the model cites no sources. Attributions link the answer to the fragments after the answer is complete. The API server splits the answer into sentences and matches each sentence to the fragments that the model received: by the words and word pairs that the sentence shares with each fragment, and, when the pipeline has a ready default vector index, by the similarity of the embeddings of the sentence and the fragment. Attributions are therefore an estimate of which fragments support which sentence.

An execution with attributions set to true adds the answer and the attributions to the trace when the answer is complete, before the response ends. An answer that fails part way, or whose client stops reading the response, receives no attributions. The run_agent tool of the MCP server returns the same attributions in the tool result.

Each entry of attributions describes one sentence of the answer:

FieldTypeDescription
startIntegerThe position of the first character of the sentence in the answer, counted in Unicode code points
endIntegerThe position after the last character of the sentence
textStringThe sentence
fragmentsArrayUp to three fragments that support the sentence, best first. The array is empty when no fragment supports the sentence, or when the sentence is too short to attribute, such as Sure!.

Each entry of fragments has the following fields:

FieldTypeDescription
placeholderStringThe retrieval placeholder that retrieved the fragment
idStringThe fragment identifier
scoreNumberThe combined score, from 0 to 1: the mean of lexical and semantic, or lexical alone when semantic is null
lexicalNumberThe share of the sentence's wording that the fragment contains, from 0 to 1
semanticNumber or nullThe cosine similarity of the embeddings of the sentence and the fragment, or null when no embedding is available, such as for a pipeline without a ready default vector index

Permissions​

Creating an agent requires write permission on agents. Executing an agent requires execute permission on the agent, on the pipeline and on the LLM that the request names. The retrieval dry run requires execute permission on the agent and on the pipeline, and prompt inspection requires read permission on the agent. Access control describes permissions.