Skip to main content

Monitoring and logging

This page describes the metrics, queue state, health checks, logs and traces of a Foundation4 deployment, and the queries and alerts that an operator builds from these sources. Operators who run Foundation4 in production need this page, and operators who diagnose a problem use the commands. Troubleshooting lists the diagnoses by symptom.

Monitoring sources​

The core release installs a Prometheus server. The charts install no Alertmanager, no Grafana, no log collector and no OpenTelemetry collector; a production deployment connects Foundation4 to the monitoring stack of the organization.

SourceEndpointCollected by the bundled Prometheus
API server metrics/metrics on the metrics port 9090 of each API server podYes
Worker metrics/metrics on port 9090 of each worker podYes
NATS JetStreamThe nats command-line tool, and an optional exporterNo
gRPC serviceNone; the Kubernetes liveness probe uses the gRPC health serviceNot applicable
Redis-compatible cache and PostgreSQLNo exporter installed by the chartsNo
LogsContainer output of every componentNot applicable
TracesOpenTelemetry export from the API server and the workersNot applicable
Statistics/statistics on the API, with an admin API keyNot applicable; the route reads the bundled Prometheus and NATS JetStream

The API server serves /metrics on a separate metrics port, set by app.metrics_port (9090), and not on the API port 8000. The Kubernetes Service and the ingress expose only the API port. Security hardening lists the routes that need no API key.

Prometheus scrape scope​

The core chart turns off each default job of the Prometheus chart and adds one job, foundation4ai-pods. The job discovers pods in every namespace and keeps a pod only when the pod carries both annotations prometheus.io/app: foundation4ai and prometheus.io/scrape: "true". The job reads the path and port from the prometheus.io/path and prometheus.io/port annotations, keeps the labels that a target exposes, scrapes every 10 seconds with a 2-second timeout, and ignores pods that are not running.

Each series carries the pod labels, with dots, slashes and hyphens replaced by underscores, and the labels namespace, pod and node. The label app_kubernetes_io_name separates the API server (api-server) from the workers (api-server-worker).

The API server and worker pods carry both annotations with path /metrics and port 9090, so the job collects the metrics of both. Queries, dashboards and alerts that select the job kubernetes-pods or foundation4ai-workers select job="foundation4ai-pods" instead.

Prometheus access and targets​

  1. Forward a local port to Prometheus in a separate terminal:

    kubectl -n foundation4ai port-forward svc/foundation4ai-core-prometheus-server 9090:80

    Expected result: Forwarding from 127.0.0.1:9090 -> 9090. The Prometheus user interface is at http://localhost:9090.

  2. List the active targets and the health of each:

    curl -s 'http://localhost:9090/api/v1/targets?state=active' | grep -o '"scrapeUrl":"[^"]*"\|"health":"[^"]*"'

    Expected result: one scrapeUrl ending in :9090/metrics with "health":"up" for each API server pod and for each worker pod.

Worker scrape job​

The foundation4ai-pods job collects the worker metrics, so the core release needs no separate worker job. A values file that still holds a worker job under prometheus.extraScrapeConfigs, such as foundation4ai-workers, collects every worker series twice, under two job names. The following steps remove the job:

  1. Delete the worker job from prometheus.extraScrapeConfigs in the values file of the core release, and delete prometheus.extraScrapeConfigs when no other job remains.

    Expected result: the values file holds no foundation4ai-workers job.

  2. Upgrade the core release:

    helm upgrade --install foundation4ai-core ./charts/foundation4ai-core \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

    Expected result: Helm reports STATUS: deployed.

  3. Repeat step 2 of Prometheus access and targets.

    Expected result: one scrapeUrl ending in :9090/metrics for each worker pod, not two.

API server metrics​

Metrics is the complete metric catalog, with labels, label values and counting rules. The following tables summarize the metrics for the queries on this page.

MetricTypeLabelsMeaning
axum_http_requests_totalCountermethod, endpoint, statusHTTP requests handled. endpoint is the route template, such as /pipelines/{pipeline_id}/search, or unmatched for a path that matches no route.
axum_http_requests_duration_secondsSummarymethod, endpoint, statusSeconds from the arrival of the request to the response head, as quantiles computed by each API server pod over about the last minute, with _sum and _count series. The time spent sending a streamed response is not included.
axum_http_requests_pendingGaugemethod, endpointRequests in progress, including streamed responses that are still sending
foundation4ai_document_queue_addedCounterNoneDocuments accepted and queued for processing, through the REST API or the MCP server. Refused requests are not counted.
foundation4ai_agent_executionsCounterNoneAgent executions started, through the REST API or the MCP server
foundation4ai_agent_execution_latencyHistogramNoneSeconds from the start of an agent execution until the answer starts streaming: retrieval, prompt rendering and the setup of the LLM call

The requests to the metrics port are not counted.

Worker metrics​

The worker exports the following metrics on port 9090, each with a help line. The counter names have no _total suffix.

MetricTypeMeaning
foundation4ai_document_queue_processedCounterProcessing attempts, retries included: each document is counted every time the worker tries the document
foundation4ai_document_queue_successCounterDocuments processed successfully, documents with empty text included, each counted once
foundation4ai_document_queue_removedCounterQueued documents found deleted, or whose pipeline was deleted, when the worker took the job
foundation4ai_document_queue_skipCounterDeliveries skipped because the document was already processed, replaced or canceled
foundation4ai_document_queue_retriesCounterFailed attempts that the worker retries later, one per document and attempt
foundation4ai_document_queue_failedCounterDocuments given up on, each counted once: documents marked failed after the last attempt, and queue messages that cannot be read
foundation4ai_document_queue_processed_latencyHistogramSeconds to process one pipeline group of a batch
foundation4ai_document_queue_text_splitter_latencyGaugeSeconds spent splitting text in the last processed pipeline group
foundation4ai_document_queue_embedding_latencyGaugeSeconds spent computing embeddings in the last processed pipeline group
foundation4ai_worker_heartbeat_timestamp_secondsGaugeLast time the processing loop ran, in Unix time. A value that stops increasing means a stuck worker.

The histograms of the API server and the worker use the buckets 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120 and 300 seconds. Every value restarts at 0 when the process restarts.

Accumulating values​

The Foundation4 counters, and the _sum, _count and _bucket series of the histograms and of the summary, accumulate from process start, so a raw value depends on the uptime of the pod. Dashboards and alerts use the following functions:

  • Rates. rate(<metric>[5m]) gives the change per second over 5 minutes and treats a drop to 0 as a restart.
  • Totals over a period. increase(<metric>[1h]) gives the change over 1 hour.
  • Deployment totals. sum(rate(<metric>[5m])) adds the rates of all pods.
  • Averages. The rate of a _sum series divided by the rate of the matching _count series gives the average duration.
  • Percentiles. histogram_quantile over the rates of the _bucket series of a histogram gives a percentile across all pods.

The gauges hold a current value and are read without rate. The quantile series of axum_http_requests_duration_seconds are computed by each pod and are also read without rate; quantiles of several pods are not added. Prometheus can add an informational note to rate queries on the counters, because the counter names do not end in _total.

PanelQuery
Documents submitted per minutesum(rate(foundation4ai_document_queue_added[5m])) * 60
Documents processed per minutesum(rate(foundation4ai_document_queue_success[5m])) * 60
Retried attempts per minutesum(rate(foundation4ai_document_queue_retries[5m])) * 60
Documents failed per hoursum(increase(foundation4ai_document_queue_failed[1h]))
Skipped jobs per minutesum(rate(foundation4ai_document_queue_skip[5m])) * 60
Processing seconds per pipeline group, averagesum(rate(foundation4ai_document_queue_processed_latency_sum[5m])) / sum(rate(foundation4ai_document_queue_processed_latency_count[5m]))
Processing seconds per pipeline group, 95th percentilehistogram_quantile(0.95, sum by (le) (rate(foundation4ai_document_queue_processed_latency_bucket[5m])))
Embedding and splitting seconds of the last group, by worker podfoundation4ai_document_queue_embedding_latency and foundation4ai_document_queue_text_splitter_latency
Worker heartbeat age, by worker podtime() - foundation4ai_worker_heartbeat_timestamp_seconds
Agent executions per minutesum(rate(foundation4ai_agent_executions[5m])) * 60
Agent time to first answer, 95th percentilehistogram_quantile(0.95, sum by (le) (rate(foundation4ai_agent_execution_latency_bucket[5m])))
API requests per second by statussum by (status) (rate(axum_http_requests_total[5m]))
API server error ratiosum(rate(axum_http_requests_total{status=~"5.."}[5m])) / sum(rate(axum_http_requests_total[5m]))
API latency, average by routesum by (endpoint) (rate(axum_http_requests_duration_seconds_sum[5m])) / sum by (endpoint) (rate(axum_http_requests_duration_seconds_count[5m]))
API latency, 95th percentile by route, slowest podmax by (endpoint) (axum_http_requests_duration_seconds{quantile="0.95"})
API requests in progresssum(axum_http_requests_pending)
Targets upup{job="foundation4ai-pods"}

The queue depth comes from NATS JetStream, as described in Queue state. No metric reports the database connection pool; the API server error ratio and the pool warning described in Log messages show pool pressure.

The thresholds depend on the workload and are set after a period of measurement:

AlertCondition
Processing failuressum(increase(foundation4ai_document_queue_failed[15m])) > <failures>
Processing retriessum(increase(foundation4ai_document_queue_retries[15m])) > <retries>
Processing stoppedsum(rate(foundation4ai_document_queue_added[30m])) > 0 and sum(rate(foundation4ai_document_queue_success[30m])) == 0
Worker stucktime() - foundation4ai_worker_heartbeat_timestamp_seconds > <seconds>, below the 900 seconds after which the liveness probe restarts the worker
Worker target downup{job="foundation4ai-pods", app_kubernetes_io_name="api-server-worker"} == 0 for 5 minutes
API server target downup{job="foundation4ai-pods", app_kubernetes_io_name="api-server"} == 0 for 5 minutes
API server errorsThe error ratio query of Recommended panels above <ratio> for 10 minutes
Oldest queued jobAge of the first message of the stream DOCUMENTS above <hours>, well below 24 hours, from Queue state

The bundled Prometheus evaluates rules from prometheus.serverFiles.alerting_rules.yml, but the core chart disables Alertmanager, so the bundled installation delivers no alert. A production deployment evaluates these conditions in the monitoring stack of the organization, which scrapes the same targets or receives the series from the bundled Prometheus.

Queue state​

The utility Deployment foundation4ai-core-nats-box carries the nats command-line tool. The stream DOCUMENTS holds the processing jobs, and the consumer document-processor delivers the jobs to the workers. NATS JetStream accepts only authenticated clients: the user is foundation4ai and the password is the NATS_PASSWORD value of the operator Secret foundation4ai-secrets.

NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
kubectl -n foundation4ai exec deploy/foundation4ai-core-nats-box -- \
nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" \
stream info DOCUMENTS
kubectl -n foundation4ai exec deploy/foundation4ai-core-nats-box -- \
nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" \
consumer info DOCUMENTS document-processor

Expected result: the stream information shows the number of messages and the sequence and time of the first message. The consumer information shows delivery and acknowledgement state. The field names below are those of the nats tool and can differ between versions of the tool:

FieldReading
Stream MessagesJobs in the queue, including jobs that a worker is processing. This value is the queue depth.
Stream First Sequence timeSubmission time of the oldest job. A job older than 24 hours is removed.
Consumer Unprocessed MessagesJobs not yet delivered to a worker. A value that rises during a load means that the workers process slower than the client application submits.
Consumer Outstanding AcksJobs delivered to a worker and not yet acknowledged
Consumer Redelivered MessagesJobs delivered more than once, after a failure or an expired acknowledgement wait
Consumer Last Ack timeTime of the last acknowledgement. An old time while jobs remain means that no worker completes jobs.
Consumer Waiting PullsFetch requests from workers. A value of 0 while workers should be running means that no worker is connected.

The NATS chart includes an exporter that publishes NATS server and JetStream metrics, including stream and consumer state. The core scrape job keeps the NATS pods only with the Foundation4 annotations, which the following untested values add:

nats:
promExporter:
enabled: true
podTemplate:
merge:
metadata:
annotations:
prometheus.io/app: foundation4ai
prometheus.io/scrape: "true"
prometheus.io/port: "7777"

The metric names depend on the exporter version; the exporter lists the names at http://<nats pod>:7777/metrics.

Health checks​

/healthz on the API server checks the components that the API server uses: a query to PostgreSQL, the connection to NATS JetStream, a PING to the Redis-compatible cache and the gRPC health service of the gRPC service in the same pod. The checks run at the same time, and a component that gives no answer within 3 seconds counts as failed. With every component available, /healthz returns status 200:

{"status":"ok","database":"ok","message_queue":"ok","cache":"ok","llm_service":"ok"}

When a component fails, /healthz returns status 503 with "status":"error" and the value "error" for the failing component, and the API server logs Health check failed at WARN with the component and the error. The chart uses /healthz for the readiness probe of the server container, so an API server pod with a failing component leaves the Service without a restart. The probes of the pods are the following:

  • API server liveness. The liveness probe calls the root route /, which answers Foundation4.ai API while the process serves HTTP. An outage of a shared component therefore restarts no API server pod.
  • API server startup. A startup probe on /, every 10 seconds for up to 30 failures, gives the API server about 5 minutes to start, while startup waits for PostgreSQL, the cache and NATS JetStream. api-server.server.startupProbe sets the probe, and the value null removes the probe.
  • Worker liveness. The processing loop of the worker writes the current time to the heartbeat file /tmp/foundation4ai-worker-heartbeat, at most every 10 seconds, and to the metric foundation4ai_worker_heartbeat_timestamp_seconds. The liveness probe reads the file and restarts a worker whose loop has not run for 15 minutes.
  • gRPC service liveness. The liveness probe of the grpc container calls the gRPC health service.

The following checks test the components:

CheckCommandExpected result
Components of the API servercurl -s -w "\n%{http_code}\n" "$FOUNDATION4_URL/healthz"The JSON response with "status":"ok", then 200
Database connection from an API server podkubectl -n foundation4ai exec deploy/foundation4ai-api-server -c server -- ./foundation4ai database check-connectionOk.
Authenticated API request that reads the databasecurl -s -o /dev/null -w "%{http_code}\n" -H "x-api-key: $FOUNDATION4_API_KEY" -H "x-api-key-secret: $FOUNDATION4_API_SECRET" "$FOUNDATION4_URL/pipelines?first=1"200
QueueThe stream command of Queue stateThe stream information, with no error
gRPC servicekubectl -n foundation4ai get podsAPI server and worker pods 2/2 ready; the liveness probe calls the gRPC health service
Worker processing loopkubectl -n foundation4ai exec deploy/foundation4ai-api-server-worker -c worker -- cat /tmp/foundation4ai-worker-heartbeatA Unix time less than a minute old
Worker metrics listenerkubectl -n foundation4ai port-forward deploy/foundation4ai-api-server-worker 9091:9090, then curl -s http://localhost:9091/healthOK
PrometheusWith the port forward of Prometheus access and targets, curl -s http://localhost:9090/-/readyA message that Prometheus is ready

An external availability check calls /healthz through the ingress, and the authenticated request with a dedicated API key that holds read permission on one pipeline, which also tests the API key check. Manage API keys and permissions describes such keys. The /statistics route returns document and agent execution totals and latencies over an interval, read from the bundled Prometheus, and the number of queued jobs, and requires an admin API key. Metrics describes each field.

Logs​

Every component writes logs to the container output only; no component writes log files. The retention of the logs follows the log collection of the cluster.

ComponentCommand
API serverkubectl -n foundation4ai logs deploy/foundation4ai-api-server -c server
gRPC service in the API server podskubectl -n foundation4ai logs deploy/foundation4ai-api-server -c grpc
Workers, all podskubectl -n foundation4ai logs -l app.kubernetes.io/name=api-server-worker -c worker --prefix --tail=500
Hook Jobskubectl -n foundation4ai logs job/foundation4ai-api-server-db-migration, and the same for foundation4ai-api-server-license-check and foundation4ai-api-server-create-admin-api-key. Helm deletes a Job that succeeds, so the log is available while the Job runs, and after a failure until the next installation or upgrade.
Dashboardkubectl -n foundation4ai logs deploy/foundation4ai-dashboard
NATS JetStreamkubectl -n foundation4ai logs foundation4ai-core-nats-0 -c nats

app.log_level sets the level of the API server and the workers, with the default warn; the gRPC service logs at INFO. Configuration reference describes the levels, the output formats and the procedure that changes the level. A production deployment keeps warn outside a limited diagnosis period, as described in Security hardening.

The API server returns an x-request-id header on every response of the API port. The exported traces carry the same value in the attribute request_id, and a log line that the API server writes while handling a request carries request_id="<value>", with the value in quotation marks, in the prefix that names the HTTP request span. The same prefix carries the query string of the request as url.query. The API server writes one line for each request: request failed at WARN for a status 5xx, request refused at INFO for a status 4xx, and request at DEBUG for other statuses. The line carries the method, the path without the query string as path, the status, the duration in milliseconds as elapsed_ms, and the request ID. At warn, only the requests that fail with a status 5xx produce a line.

Log messages​

ComponentLevelMessageMeaning
WorkerINFOSuccessfully processed document <document> for pipeline <pipeline>Document processed, including a document with empty text
WorkerWARNFailed processing document <document> for pipeline <pipeline>: <reason>One document failed; the job returns after a delay that starts at 10 seconds and doubles up to 10 minutes
WorkerWARNGiving up on a document after repeated processing failuresThe document failed its 12th attempt and is now failed, with the status message Processing failed after 12 attempts: <reason>
WorkerERRORError processing documents for pipelineA pipeline group failed; every job of the group returns after the same delay. For an internal error, the reason stored on the documents is internal error; see the worker logs.
WorkerWARNSkipped document <document> for pipeline <pipeline>: <status>The document was no longer pending, such as a job delivered again after the document was processed
WorkerWARNDocument <document> for pipeline <pipeline> was removed before processingThe document or the pipeline was deleted before processing
WorkerERRORFailed to acknowledge message: <error>The worker could not acknowledge a job to NATS JetStream
WorkerERRORError retrieving messages from message queueThe worker could not fetch jobs from NATS JetStream; the worker tries again after 2 seconds and keeps running
WorkerERRORUndecodable queue message; terminating itA job could not be read; the job leaves the queue and counts in foundation4ai_document_queue_failed
WorkerERRORProcessing a batch panickedAn unexpected error stopped one batch; the jobs of the batch return to the queue and the worker keeps running
WorkerWARNCould not write the worker heartbeat fileThe liveness probe reads no current heartbeat and restarts the worker
WorkerERRORThe license is no longer valid: the worker stops; queued documents wait for a renewed licenseThe license expired while the worker ran. The worker exits, and a restarted worker does not start until a renewed license is deployed, as described in Licensing.
API server and workerWARNWaiting for the database, cache or queue; retrying in <delay>PostgreSQL, the cache or NATS JetStream does not answer at startup; the process tries 30 times, for about 4.5 minutes, before the process stops
API server and workerWARNFailed to validate license for system id: <system id>License rejected at startup; the process stops
API server and workerProcess exitLicense check failed: <reason>The license is invalid, expired, of an unsupported version or over a limit. The API server writes the message after Failed to initialize server: .
API server and workerProcess exitFailed to parse config: <error>A configuration key is missing or invalid
API server and workerProcess exitFailed to connect to Foundation4.ai's services: <error>Startup failed on the database, the cache, NATS JetStream or the application secret. The error starts with the failing part, such as ORM error:, Redis error:, Nats error: or Decryption error:, and the API server writes the message after Failed to initialize server: .
API serverERRORThe master key is expired, inactive or missing, so requests made with it are refused; ...The API server keeps running and refuses the master key until an operator restores the key, as described in Secrets and keys
API serverProcess exitFailed to initialize server: Failed to initialize with the master key: <error>The master key secret does not match the key in the database
WorkerProcess exitFailed to initialize Foundation4.ai: <error>The master key check failed in a worker, including for an expired, inactive or missing master key
API serverProcess exitFailed to bind the metrics address: <reason> or app.metrics_port must differ from app.portThe metrics port cannot be used
API serverERRORFailed to refresh license object count cacheThe refresh of the license object counts, every 10 minutes, failed
API serverERRORFailed to refresh license document count cacheThe refresh of the document and fragment counts, every 10 seconds, failed, as described in Licensing
API serverWARNrequest failedA request ended with a status 5xx
API serverERRORInternal server errorA request failed with status 500; the line carries the kind and the error that the response leaves out
API serverERRORCould not validate the API keyThe API key check could not read the database or the cache; the request receives status 503
API serverERRORFailed queueing document for processingA created document could not be queued in NATS JetStream and has the status failed
API serverWARNHealth check failedA component failed the /healthz check; the line names the component and the error
API server and workerWARNno env var set or infered for OTEL_EXPORTER_OTLP_<SIGNAL>_PROTOCOL or OTEL_EXPORTER_OTLP_PROTOCOL; no <kind> exporter will be createdNo OpenTelemetry exporter is configured. Two lines appear at each start of a process without the variables described in OpenTelemetry tracing: LOGS with log, and TRACES with span
API serverWARNacquired connection, but time to acquire exceeded slow thresholdA request waited more than 2 seconds for a database connection
gRPC serviceINFOINFO:processor:gRPC server started on 127.0.0.1:50051The gRPC service is ready
gRPC serviceINFOAdded package directory to site: <path>A package image was loaded

The lines at INFO from the API server and the workers appear only when app.log_level is info, debug or trace.

OpenTelemetry tracing​

The API server and the workers export spans and log records through the OpenTelemetry Protocol (OTLP) when an OTLP variable is set. The API server exports a span for each API request, with the request ID, and a request that carries trace context headers joins the trace of the caller. A document queued through the REST API or the MCP server carries the trace context of the request, so the process_documents span of the worker joins the trace of the request that queued the document. When one batch holds documents of several requests, the span links to each of these traces instead. The process_documents span has the level INFO, so a worker exports the span when app.log_level is info, debug or trace, and exports log records only at warn. The gRPC service and the dashboard export no traces, so the text splitting and embedding calls of the gRPC service appear in no trace. The charts install no collector, so the collector runs in the cluster or elsewhere inside the network. Configuration reference lists the variables.

  1. Add the variables to the values file of the application release. The entries of api-server.configs reach both the API server and the workers. Values that contain credentials, such as OTEL_EXPORTER_OTLP_HEADERS, go under api-server.secrets instead:

    api-server:
    configs:
    OTEL_EXPORTER_OTLP_ENDPOINT: "http://<collector host>:4317"
    OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"

    Expected result: the values file holds the two entries. Without OTEL_SERVICE_NAME, the API server reports the service name foundation4ai and the workers report foundation4ai-worker; OTEL_SERVICE_NAME sets one name for both.

  2. Upgrade the application release:

    helm upgrade --install foundation4ai ./charts/foundation4ai \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m
    kubectl -n foundation4ai rollout status deploy/foundation4ai-api-server
    kubectl -n foundation4ai rollout status deploy/foundation4ai-api-server-worker

    Expected result: Helm reports STATUS: deployed, and both status commands end with successfully rolled out. A changed configuration restarts the API server and worker pods during the upgrade.

  3. Confirm the variables in the server and worker containers:

    kubectl -n foundation4ai exec deploy/foundation4ai-api-server -c server -- env | grep '^OTEL_'
    kubectl -n foundation4ai exec deploy/foundation4ai-api-server-worker -c worker -- env | grep '^OTEL_'

    Expected result: the two variables with the values from step 1, for each container.

  4. Send a request and read the request ID from the response headers:

    curl -s -D - -o /dev/null -X POST "$FOUNDATION4_URL/login" \
    -H "x-api-key: $FOUNDATION4_API_KEY" \
    -H "x-api-key-secret: $FOUNDATION4_API_SECRET" | grep -i '^x-request-id'

    Expected result: an x-request-id header. The trace backend of the collector shows a trace of service foundation4ai whose span carries the same value in request_id. With app.log_level at info or a more detailed level, a document created after this step adds a process_documents span of service foundation4ai-worker to the trace of the create request.