Metrics
This page lists every metric that the Foundation4 processes export, with the exported name, the type, the labels, the emitting process, the event that changes the value and the meaning, and the fields of the /statistics route. Operators use this page to build dashboards and alerts, and integrators use this page to read processing counts during a load. Observability explains how the metrics relate to the other signals, and Monitoring and logging describes metric collection, queries and alerts.
Metric endpoints
| Process | Endpoint | Access | Content |
|---|---|---|---|
| API server | GET /metrics on the metrics port, set by app.metrics_port, 9090 by default and in the charts | No API key. Only /metrics answers on the port, and every other path returns status 404. No Service or ingress exposes the port, so the port is reachable inside the cluster or through a port forward. | HTTP request metrics, and the document and agent execution metrics |
| Worker | Port 9090 of each worker pod, any path except /health; the charts use /metrics | No authentication. No Service exposes the port, so the port is reachable inside the cluster or through a port forward. | Document processing metrics |
| Worker | /health on port 9090 | Same as the metrics path | The text OK |
| gRPC service and dashboard | None | Not applicable | Not applicable |
The API server listens for metrics on the address app.host with the port app.metrics_port, which must differ from app.port. The worker listens on 0.0.0.0:9090, a fixed address that no configuration key changes. The worker listener starts before the worker connects to PostgreSQL and NATS JetStream, and serves the metrics for the life of the process. The liveness probe of the worker container in the charts reads the heartbeat file that the processing loop writes, not the metrics listener, as described in Health checks.
The pod annotations of the charts name the path /metrics and the port 9090 for the API server and for the workers, so the bundled Prometheus server collects both. Security hardening lists the routes of the API port that need no API key.
Exposition format
Both endpoints return the Prometheus text exposition format, with the content type text/plain; charset=utf-8 from the API server and text/plain from the workers. The following rules apply to every metric on this page:
- Names. The exporter writes each name exactly as the code defines the name and adds no suffix. The
foundation4ai_*counters therefore have no_totalsuffix, andaxum_http_requests_totalends in_totalbecause the suffix is part of the defined name. - Metadata. Each metric has a
# TYPEline. Thefoundation4ai_*metrics also have a# HELPline that states the meaning. - Lifetime. The counters, and the
_bucket,_sumand_countseries, accumulate from the start of the process. The gauges hold a current value. Every value returns to 0 when the process restarts, and a series remains in the output for the life of the process. - First appearance. The worker metrics,
foundation4ai_document_queue_addedandfoundation4ai_agent_executionsappear when the process starts, the counters with the value 0.foundation4ai_agent_execution_latencyappears after the first agent execution, and an HTTP series after the first request with the label values of the series. - Histograms. A histogram exports
_bucketseries with the labelle, and_sumand_countseries. The histograms use the buckets 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120 and 300 seconds. - Summary.
axum_http_requests_duration_secondsis a summary: series with the labelquantilefor the quantiles 0, 0.5, 0.9, 0.95, 0.99, 0.999 and 1, computed by the process over a rolling window of about one minute, and_sumand_countseries.
API server metrics
The API server exports the following metrics on /metrics of the metrics port:
| Metric | Type | Labels | Change | Meaning |
|---|---|---|---|---|
axum_http_requests_total | Counter | method, status, endpoint | Increases by 1 when the API server returns the response head of a request on the API port | Requests handled, including requests rejected for a missing or invalid API key. Requests to the metrics port are not counted. |
axum_http_requests_duration_seconds | Summary | method, status, endpoint | Records one value when the API server returns the response head | Seconds from the arrival of the request to the response head. For a streamed response, the time spent sending the stream is not included. |
axum_http_requests_pending | Gauge | method, endpoint | Increases by 1 when a request arrives, and decreases by 1 when the response body is complete or the connection closes | Requests in progress, including streamed responses that are still sending |
foundation4ai_document_queue_added | Counter | None | Set to 0 at startup. Increases by 1 when POST /documents, POST /pipelines/{pipeline_id}/documents or the MCP tool add_document creates a document, including a document that could not be queued and has the status failed. | Documents accepted and queued for processing. Refused requests are not counted. |
foundation4ai_agent_executions | Counter | None | Set to 0 at startup. Increases by 1 when an agent execution starts through POST /agents/{agent_id}/execute or the MCP tool run_agent, after the agent lookup and before retrieval | Agent executions started, including executions that fail later |
foundation4ai_agent_execution_latency | Histogram | None | Records one value when the answer of an execution starts streaming | Seconds from the start of an agent execution until the answer starts streaming: retrieval, prompt rendering and the setup of the LLM call. Executions that fail before the answer are not recorded. |
HTTP label values
The HTTP labels take the following values:
| Label | Values |
|---|---|
method | The HTTP method in upper case: GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, CONNECT or TRACE. Any other method reports an empty value. |
status | The three-digit status code of the response head, such as 200, 401 or 500 |
endpoint | For a request to an API operation, the route template as the endpoint reference shows the path, with the parameter names in braces, such as /pipelines/{pipeline_id}/search or /tracing/{execution_id}. The other routes report /, /healthz, /mcp, /openapi.json, /docs, /_docs and /_docs-swagger. A request to a path that matches no route, and a request under /assets, reports unmatched, so unknown paths add no series. |
The following lines show the form of the API server output:
# TYPE axum_http_requests_total counter
axum_http_requests_total{method="POST",status="201",endpoint="/pipelines/{pipeline_id}/documents"} 1180
# HELP foundation4ai_document_queue_added Documents accepted and queued for processing (refused requests are not counted)
# TYPE foundation4ai_document_queue_added counter
foundation4ai_document_queue_added 1180
Worker metrics
Each worker exports the following metrics on port 9090. A worker fetches up to 100 jobs at a time and processes the jobs of each pipeline as one group. Every worker metric has a # HELP line with the text in the table.
| Metric | Type | # HELP text | Change | Meaning |
|---|---|---|---|---|
foundation4ai_document_queue_processed | Counter | Processing attempts, retries included: each document is counted every time it is tried | Increases by the number of documents of a pipeline group each time the worker processes the group, before the outcome is known | Processing attempts, including retries and failed attempts |
foundation4ai_document_queue_success | Counter | Documents processed successfully (empty documents included), each counted once | Increases by 1 for each document whose fragments the worker stored, and for each document with empty text | Documents completed |
foundation4ai_document_queue_removed | Counter | Queued documents found deleted (or their pipeline deleted) when the worker took them | Increases by 1 for each document that was deleted, or whose pipeline was deleted, before processing | Jobs of deleted documents |
foundation4ai_document_queue_skip | Counter | Deliveries skipped because the document was already processed, replaced or canceled | Increases by 1 for each document that is no longer pending | Jobs completed without processing |
foundation4ai_document_queue_retries | Counter | Failed attempts that will be retried later; one per document and attempt | Increases by 1 for each document that fails an attempt before the 12th, including every document of a failed pipeline group | Failed attempts that return to the queue |
foundation4ai_document_queue_failed | Counter | Documents given up on: marked failed after their last attempt, or undecodable queue messages; each counted once | Increases by 1 for each document that fails its 12th attempt, and for each queue message that cannot be read | Documents given up on |
foundation4ai_document_queue_processed_latency | Histogram | Time to process one batch of documents | Records the seconds of each attempt to process a pipeline group, successful or failed | Processing time of a pipeline group |
foundation4ai_document_queue_text_splitter_latency | Gauge | Text splitting time of the last processed pipeline group | Set to the seconds spent splitting the text of a pipeline group that the worker processed without a group failure | Splitting time of the last group |
foundation4ai_document_queue_embedding_latency | Gauge | Embedding time of the last processed pipeline group | Set to the seconds spent computing the embeddings of a pipeline group, for every embedding model of the pipeline, when the worker processed the group without a group failure | Embedding time of the last group |
foundation4ai_worker_heartbeat_timestamp_seconds | Gauge | Last time the processing loop ran (Unix time); a stale value means a stuck worker | Set to the current Unix time at startup, and at most every 10 seconds while the processing loop runs | Last activity of the processing loop |
Latency metrics
foundation4ai_document_queue_processed_latency is a histogram, so the rates of the _sum and _count series give the average time per pipeline group, and histogram_quantile gives a percentile. The two gauges hold the times of the last group only, and a later group replaces the values. Monitoring and logging describes the queries.
The following lines show the form of the worker output:
# HELP foundation4ai_document_queue_processed Processing attempts, retries included: each document is counted every time it is tried
# TYPE foundation4ai_document_queue_processed counter
foundation4ai_document_queue_processed 1200
# HELP foundation4ai_document_queue_embedding_latency Embedding time of the last processed pipeline group
# TYPE foundation4ai_document_queue_embedding_latency gauge
foundation4ai_document_queue_embedding_latency 3.82
Processing outcomes
Each document of a pipeline group ends each attempt in one of the following outcomes. Every outcome increases foundation4ai_document_queue_processed by 1 for the document.
| Outcome | Counter | Worker log line | Job |
|---|---|---|---|
| Fragments stored | success | INFO Successfully processed document <document> for pipeline <pipeline> | Acknowledged |
| Empty document text | success | INFO Successfully processed document <document> for pipeline <pipeline>. The document status becomes success. | Acknowledged |
Document no longer pending, such as a job delivered again after processing | skip | WARN Skipped document <document> for pipeline <pipeline>: <status> | Acknowledged |
| Document or pipeline deleted before processing | removed | WARN Document <document> for pipeline <pipeline> was removed before processing | Acknowledged |
| Version replaced or expired before processing | skip | WARN Skipped document <document> for pipeline <pipeline>: canceled. The document status becomes canceled. | Acknowledged |
| Document failed | retries, or failed at the 12th attempt | WARN Failed processing document <document> for pipeline <pipeline>: <reason> | Returned to the queue and delivered again after a delay that starts at 10 seconds and doubles up to 10 minutes. At the 12th failed attempt, the worker logs Giving up on a document after repeated processing failures, sets the status to failed with the message Processing failed after 12 attempts: <reason> and removes the job. |
| Pipeline group failed | retries, or failed at the 12th attempt, for every document of the group | ERROR Error processing documents for pipeline | Every job of the group returned to the queue, as for a failed document |
A pipeline group fails as a whole when the worker cannot complete a step that covers every document of the group, such as the license check, the pipeline lookup or the database transaction. Text splitting and embedding errors fail only the document concerned. A queue message that the worker cannot read is removed from the queue and counted in failed without a document outcome. The INFO line appears only when app.log_level is info, debug or trace.
Labels added at collection
The scrape job foundation4ai-pods of the core chart adds the labels job, instance, namespace, pod and node and every pod label, with dots, slashes and hyphens replaced by underscores. The label app_kubernetes_io_name separates the API server (api-server) from the workers (api-server-worker). The worker pods carry fewer pod labels than the API server pod, so app_kubernetes_io_managed_by, app_kubernetes_io_version and helm_sh_chart appear on the API server series only. The job keeps the labels of the exported series when the names collide. Prometheus scrape scope describes the job.
Statistics route
GET /statistics returns document processing and agent execution figures for a period, read from Prometheus, and the number of jobs in the queue. The route answers on the API port and requires an admin API key, because the figures cover every pipeline and agent. The dashboard does not use the route.
The query parameter interval is required and holds the length of the period, which ends at the time of the request. The value is a Prometheus duration: one or more of <n>y, <n>w, <n>d, <n>h, <n>m, <n>s and <n>ms, from the largest unit to the smallest, such as 15m, 1h30m or 24h. The value 0 passes the check, but Prometheus refuses a period of length 0, so a request with interval=0 returns HTTP 503 Prometheus could not be queried; see the server logs.
The API server sends each Prometheus query to the URL of the core release, waits up to 10 seconds for each answer, and runs the queries at the same time. A query that fails or times out makes its field null, and the API server logs Prometheus query failed or Prometheus query timed out at WARN. A query that Prometheus answers without data, such as a metric without series yet, gives 0 or an empty list, so null and 0 differ: null means that the figure is unknown, and 0 means no activity in the period.
Response fields
The response contains the following fields:
| Field | Type | Meaning |
|---|---|---|
documents_added | Number or null | Documents accepted and queued in the period, from foundation4ai_document_queue_added |
documents_processed | Number or null | Documents processed successfully in the period, each counted once, from foundation4ai_document_queue_success |
documents_processed_latency | Number or null | Average time in seconds to process one pipeline group of a batch, from foundation4ai_document_queue_processed_latency |
documents_failed | Number or null | Documents given up on in the period, each counted once, from foundation4ai_document_queue_failed |
agent_executions | Number or null | Agent executions started in the period |
agent_execution_latency | Number or null | Average time in seconds from the start of an agent execution until the answer starts streaming |
agent_execution_latencies | Array of pairs of a bucket bound and a count, or null | Distribution of agent execution times: the upper bound of each bucket in seconds, +Inf for the last one, and the number of executions up to the bound |
document_queue | Integer | Number of messages in the NATS JetStream stream named by nats.stream, DOCUMENTS in the charts: the jobs waiting in the queue and the jobs that a worker is processing |
document_queue_processed_latencies | Array of pairs of a bucket bound and a count, or null | Distribution of processing times: the upper bound of each bucket in seconds, +Inf for the last one, and the number of pipeline groups up to the bound |
The figures over a period are increases computed by Prometheus, so a value can carry decimals.
Example
The following request reads the figures of the last hour with an admin API key, using the shared variables of Authenticate:
curl -s "$FOUNDATION4_URL/statistics?interval=1h" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"
The response has the following form:
{
"documents_added": 120.0,
"documents_processed": 118.0,
"documents_processed_latency": 1.9,
"documents_failed": 0.0,
"agent_executions": 14.0,
"agent_execution_latency": 0.8,
"agent_execution_latencies": [["0.05", 0.0], ["0.1", 0.0], ["0.25", 2.0], ["0.5", 6.0], ["1", 11.0], ["2.5", 14.0], ["5", 14.0], ["10", 14.0], ["30", 14.0], ["60", 14.0], ["120", 14.0], ["300", 14.0], ["+Inf", 14.0]],
"document_queue": 2,
"document_queue_processed_latencies": [["0.05", 0.0], ["0.1", 0.0], ["0.25", 0.0], ["0.5", 1.0], ["1", 3.0], ["2.5", 9.0], ["5", 12.0], ["10", 12.0], ["30", 12.0], ["60", 12.0], ["120", 12.0], ["300", 12.0], ["+Inf", 12.0]]
}
Errors
The route returns the following errors:
| Status | Body | Cause |
|---|---|---|
| 400 | JSON error body with the message Invalid interval format: <value>. Must be a valid Prometheus duration (e.g., 15m, 1h, 24h). | interval is not a Prometheus duration |
| 400 | JSON error body with the message of the query parameter error | The request has no interval parameter |
| 401 | JSON error body | The request has no valid API key |
| 403 | JSON error body with the message Forbidden: an admin API key is required | The API key is not an admin key |
| 500 | JSON error body with the message Failed to get message queue information | The API server could not read the stream state from NATS JetStream, and logs Failed to get NATS stream info at ERROR |
| 503 | JSON error body with the message Prometheus could not be queried; see the server logs | Every Prometheus query failed |
The queue figures of Queue state and the Prometheus queries of Recommended panels give the same figures per pod and over any period.