Skip to main content

Metrics

This page lists every metric that the Foundation4 processes export, with the exported name, the type, the labels, the emitting process, the event that changes the value and the meaning, and the fields of the /statistics route. Operators use this page to build dashboards and alerts, and integrators use this page to read processing counts during a load. Observability explains how the metrics relate to the other signals, and Monitoring and logging describes metric collection, queries and alerts.

Metric endpoints​

ProcessEndpointAccessContent
API serverGET /metrics on the metrics port, set by app.metrics_port, 9090 by default and in the chartsNo API key. Only /metrics answers on the port, and every other path returns status 404. No Service or ingress exposes the port, so the port is reachable inside the cluster or through a port forward.HTTP request metrics, and the document and agent execution metrics
WorkerPort 9090 of each worker pod, any path except /health; the charts use /metricsNo authentication. No Service exposes the port, so the port is reachable inside the cluster or through a port forward.Document processing metrics
Worker/health on port 9090Same as the metrics pathThe text OK
gRPC service and dashboardNoneNot applicableNot applicable

The API server listens for metrics on the address app.host with the port app.metrics_port, which must differ from app.port. The worker listens on 0.0.0.0:9090, a fixed address that no configuration key changes. The worker listener starts before the worker connects to PostgreSQL and NATS JetStream, and serves the metrics for the life of the process. The liveness probe of the worker container in the charts reads the heartbeat file that the processing loop writes, not the metrics listener, as described in Health checks.

The pod annotations of the charts name the path /metrics and the port 9090 for the API server and for the workers, so the bundled Prometheus server collects both. Security hardening lists the routes of the API port that need no API key.

Exposition format​

Both endpoints return the Prometheus text exposition format, with the content type text/plain; charset=utf-8 from the API server and text/plain from the workers. The following rules apply to every metric on this page:

  • Names. The exporter writes each name exactly as the code defines the name and adds no suffix. The foundation4ai_* counters therefore have no _total suffix, and axum_http_requests_total ends in _total because the suffix is part of the defined name.
  • Metadata. Each metric has a # TYPE line. The foundation4ai_* metrics also have a # HELP line that states the meaning.
  • Lifetime. The counters, and the _bucket, _sum and _count series, accumulate from the start of the process. The gauges hold a current value. Every value returns to 0 when the process restarts, and a series remains in the output for the life of the process.
  • First appearance. The worker metrics, foundation4ai_document_queue_added and foundation4ai_agent_executions appear when the process starts, the counters with the value 0. foundation4ai_agent_execution_latency appears after the first agent execution, and an HTTP series after the first request with the label values of the series.
  • Histograms. A histogram exports _bucket series with the label le, and _sum and _count series. The histograms use the buckets 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120 and 300 seconds.
  • Summary. axum_http_requests_duration_seconds is a summary: series with the label quantile for the quantiles 0, 0.5, 0.9, 0.95, 0.99, 0.999 and 1, computed by the process over a rolling window of about one minute, and _sum and _count series.

API server metrics​

The API server exports the following metrics on /metrics of the metrics port:

MetricTypeLabelsChangeMeaning
axum_http_requests_totalCountermethod, status, endpointIncreases by 1 when the API server returns the response head of a request on the API portRequests handled, including requests rejected for a missing or invalid API key. Requests to the metrics port are not counted.
axum_http_requests_duration_secondsSummarymethod, status, endpointRecords one value when the API server returns the response headSeconds from the arrival of the request to the response head. For a streamed response, the time spent sending the stream is not included.
axum_http_requests_pendingGaugemethod, endpointIncreases by 1 when a request arrives, and decreases by 1 when the response body is complete or the connection closesRequests in progress, including streamed responses that are still sending
foundation4ai_document_queue_addedCounterNoneSet to 0 at startup. Increases by 1 when POST /documents, POST /pipelines/{pipeline_id}/documents or the MCP tool add_document creates a document, including a document that could not be queued and has the status failed.Documents accepted and queued for processing. Refused requests are not counted.
foundation4ai_agent_executionsCounterNoneSet to 0 at startup. Increases by 1 when an agent execution starts through POST /agents/{agent_id}/execute or the MCP tool run_agent, after the agent lookup and before retrievalAgent executions started, including executions that fail later
foundation4ai_agent_execution_latencyHistogramNoneRecords one value when the answer of an execution starts streamingSeconds from the start of an agent execution until the answer starts streaming: retrieval, prompt rendering and the setup of the LLM call. Executions that fail before the answer are not recorded.

HTTP label values​

The HTTP labels take the following values:

LabelValues
methodThe HTTP method in upper case: GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, CONNECT or TRACE. Any other method reports an empty value.
statusThe three-digit status code of the response head, such as 200, 401 or 500
endpointFor a request to an API operation, the route template as the endpoint reference shows the path, with the parameter names in braces, such as /pipelines/{pipeline_id}/search or /tracing/{execution_id}. The other routes report /, /healthz, /mcp, /openapi.json, /docs, /_docs and /_docs-swagger. A request to a path that matches no route, and a request under /assets, reports unmatched, so unknown paths add no series.

The following lines show the form of the API server output:

# TYPE axum_http_requests_total counter
axum_http_requests_total{method="POST",status="201",endpoint="/pipelines/{pipeline_id}/documents"} 1180
# HELP foundation4ai_document_queue_added Documents accepted and queued for processing (refused requests are not counted)
# TYPE foundation4ai_document_queue_added counter
foundation4ai_document_queue_added 1180

Worker metrics​

Each worker exports the following metrics on port 9090. A worker fetches up to 100 jobs at a time and processes the jobs of each pipeline as one group. Every worker metric has a # HELP line with the text in the table.

MetricType# HELP textChangeMeaning
foundation4ai_document_queue_processedCounterProcessing attempts, retries included: each document is counted every time it is triedIncreases by the number of documents of a pipeline group each time the worker processes the group, before the outcome is knownProcessing attempts, including retries and failed attempts
foundation4ai_document_queue_successCounterDocuments processed successfully (empty documents included), each counted onceIncreases by 1 for each document whose fragments the worker stored, and for each document with empty textDocuments completed
foundation4ai_document_queue_removedCounterQueued documents found deleted (or their pipeline deleted) when the worker took themIncreases by 1 for each document that was deleted, or whose pipeline was deleted, before processingJobs of deleted documents
foundation4ai_document_queue_skipCounterDeliveries skipped because the document was already processed, replaced or canceledIncreases by 1 for each document that is no longer pendingJobs completed without processing
foundation4ai_document_queue_retriesCounterFailed attempts that will be retried later; one per document and attemptIncreases by 1 for each document that fails an attempt before the 12th, including every document of a failed pipeline groupFailed attempts that return to the queue
foundation4ai_document_queue_failedCounterDocuments given up on: marked failed after their last attempt, or undecodable queue messages; each counted onceIncreases by 1 for each document that fails its 12th attempt, and for each queue message that cannot be readDocuments given up on
foundation4ai_document_queue_processed_latencyHistogramTime to process one batch of documentsRecords the seconds of each attempt to process a pipeline group, successful or failedProcessing time of a pipeline group
foundation4ai_document_queue_text_splitter_latencyGaugeText splitting time of the last processed pipeline groupSet to the seconds spent splitting the text of a pipeline group that the worker processed without a group failureSplitting time of the last group
foundation4ai_document_queue_embedding_latencyGaugeEmbedding time of the last processed pipeline groupSet to the seconds spent computing the embeddings of a pipeline group, for every embedding model of the pipeline, when the worker processed the group without a group failureEmbedding time of the last group
foundation4ai_worker_heartbeat_timestamp_secondsGaugeLast time the processing loop ran (Unix time); a stale value means a stuck workerSet to the current Unix time at startup, and at most every 10 seconds while the processing loop runsLast activity of the processing loop

Latency metrics​

foundation4ai_document_queue_processed_latency is a histogram, so the rates of the _sum and _count series give the average time per pipeline group, and histogram_quantile gives a percentile. The two gauges hold the times of the last group only, and a later group replaces the values. Monitoring and logging describes the queries.

The following lines show the form of the worker output:

# HELP foundation4ai_document_queue_processed Processing attempts, retries included: each document is counted every time it is tried
# TYPE foundation4ai_document_queue_processed counter
foundation4ai_document_queue_processed 1200
# HELP foundation4ai_document_queue_embedding_latency Embedding time of the last processed pipeline group
# TYPE foundation4ai_document_queue_embedding_latency gauge
foundation4ai_document_queue_embedding_latency 3.82

Processing outcomes​

Each document of a pipeline group ends each attempt in one of the following outcomes. Every outcome increases foundation4ai_document_queue_processed by 1 for the document.

OutcomeCounterWorker log lineJob
Fragments storedsuccessINFO Successfully processed document <document> for pipeline <pipeline>Acknowledged
Empty document textsuccessINFO Successfully processed document <document> for pipeline <pipeline>. The document status becomes success.Acknowledged
Document no longer pending, such as a job delivered again after processingskipWARN Skipped document <document> for pipeline <pipeline>: <status>Acknowledged
Document or pipeline deleted before processingremovedWARN Document <document> for pipeline <pipeline> was removed before processingAcknowledged
Version replaced or expired before processingskipWARN Skipped document <document> for pipeline <pipeline>: canceled. The document status becomes canceled.Acknowledged
Document failedretries, or failed at the 12th attemptWARN Failed processing document <document> for pipeline <pipeline>: <reason>Returned to the queue and delivered again after a delay that starts at 10 seconds and doubles up to 10 minutes. At the 12th failed attempt, the worker logs Giving up on a document after repeated processing failures, sets the status to failed with the message Processing failed after 12 attempts: <reason> and removes the job.
Pipeline group failedretries, or failed at the 12th attempt, for every document of the groupERROR Error processing documents for pipelineEvery job of the group returned to the queue, as for a failed document

A pipeline group fails as a whole when the worker cannot complete a step that covers every document of the group, such as the license check, the pipeline lookup or the database transaction. Text splitting and embedding errors fail only the document concerned. A queue message that the worker cannot read is removed from the queue and counted in failed without a document outcome. The INFO line appears only when app.log_level is info, debug or trace.

Labels added at collection​

The scrape job foundation4ai-pods of the core chart adds the labels job, instance, namespace, pod and node and every pod label, with dots, slashes and hyphens replaced by underscores. The label app_kubernetes_io_name separates the API server (api-server) from the workers (api-server-worker). The worker pods carry fewer pod labels than the API server pod, so app_kubernetes_io_managed_by, app_kubernetes_io_version and helm_sh_chart appear on the API server series only. The job keeps the labels of the exported series when the names collide. Prometheus scrape scope describes the job.

Statistics route​

GET /statistics returns document processing and agent execution figures for a period, read from Prometheus, and the number of jobs in the queue. The route answers on the API port and requires an admin API key, because the figures cover every pipeline and agent. The dashboard does not use the route.

The query parameter interval is required and holds the length of the period, which ends at the time of the request. The value is a Prometheus duration: one or more of <n>y, <n>w, <n>d, <n>h, <n>m, <n>s and <n>ms, from the largest unit to the smallest, such as 15m, 1h30m or 24h. The value 0 passes the check, but Prometheus refuses a period of length 0, so a request with interval=0 returns HTTP 503 Prometheus could not be queried; see the server logs.

The API server sends each Prometheus query to the URL of the core release, waits up to 10 seconds for each answer, and runs the queries at the same time. A query that fails or times out makes its field null, and the API server logs Prometheus query failed or Prometheus query timed out at WARN. A query that Prometheus answers without data, such as a metric without series yet, gives 0 or an empty list, so null and 0 differ: null means that the figure is unknown, and 0 means no activity in the period.

Response fields​

The response contains the following fields:

FieldTypeMeaning
documents_addedNumber or nullDocuments accepted and queued in the period, from foundation4ai_document_queue_added
documents_processedNumber or nullDocuments processed successfully in the period, each counted once, from foundation4ai_document_queue_success
documents_processed_latencyNumber or nullAverage time in seconds to process one pipeline group of a batch, from foundation4ai_document_queue_processed_latency
documents_failedNumber or nullDocuments given up on in the period, each counted once, from foundation4ai_document_queue_failed
agent_executionsNumber or nullAgent executions started in the period
agent_execution_latencyNumber or nullAverage time in seconds from the start of an agent execution until the answer starts streaming
agent_execution_latenciesArray of pairs of a bucket bound and a count, or nullDistribution of agent execution times: the upper bound of each bucket in seconds, +Inf for the last one, and the number of executions up to the bound
document_queueIntegerNumber of messages in the NATS JetStream stream named by nats.stream, DOCUMENTS in the charts: the jobs waiting in the queue and the jobs that a worker is processing
document_queue_processed_latenciesArray of pairs of a bucket bound and a count, or nullDistribution of processing times: the upper bound of each bucket in seconds, +Inf for the last one, and the number of pipeline groups up to the bound

The figures over a period are increases computed by Prometheus, so a value can carry decimals.

Example​

The following request reads the figures of the last hour with an admin API key, using the shared variables of Authenticate:

curl -s "$FOUNDATION4_URL/statistics?interval=1h" \
-H "x-api-key: $FOUNDATION4_API_KEY" \
-H "x-api-key-secret: $FOUNDATION4_API_SECRET"

The response has the following form:

{
"documents_added": 120.0,
"documents_processed": 118.0,
"documents_processed_latency": 1.9,
"documents_failed": 0.0,
"agent_executions": 14.0,
"agent_execution_latency": 0.8,
"agent_execution_latencies": [["0.05", 0.0], ["0.1", 0.0], ["0.25", 2.0], ["0.5", 6.0], ["1", 11.0], ["2.5", 14.0], ["5", 14.0], ["10", 14.0], ["30", 14.0], ["60", 14.0], ["120", 14.0], ["300", 14.0], ["+Inf", 14.0]],
"document_queue": 2,
"document_queue_processed_latencies": [["0.05", 0.0], ["0.1", 0.0], ["0.25", 0.0], ["0.5", 1.0], ["1", 3.0], ["2.5", 9.0], ["5", 12.0], ["10", 12.0], ["30", 12.0], ["60", 12.0], ["120", 12.0], ["300", 12.0], ["+Inf", 12.0]]
}

Errors​

The route returns the following errors:

StatusBodyCause
400JSON error body with the message Invalid interval format: <value>. Must be a valid Prometheus duration (e.g., 15m, 1h, 24h).interval is not a Prometheus duration
400JSON error body with the message of the query parameter errorThe request has no interval parameter
401JSON error bodyThe request has no valid API key
403JSON error body with the message Forbidden: an admin API key is requiredThe API key is not an admin key
500JSON error body with the message Failed to get message queue informationThe API server could not read the stream state from NATS JetStream, and logs Failed to get NATS stream info at ERROR
503JSON error body with the message Prometheus could not be queried; see the server logsEvery Prometheus query failed

The queue figures of Queue state and the Prometheus queries of Recommended panels give the same figures per pod and over any period.