Monitoring and logging
This page describes the metrics, queue state, health checks, logs and traces of a Foundation4 deployment, and the queries and alerts that an operator builds from these sources. Operators who run Foundation4 in production need this page, and operators who diagnose a problem use the commands. Troubleshooting lists the diagnoses by symptom.
Monitoring sources
The core release installs a Prometheus server. The charts install no Alertmanager, no Grafana, no log collector and no OpenTelemetry collector; a production deployment connects Foundation4 to the monitoring stack of the organization.
| Source | Endpoint | Collected by the bundled Prometheus |
|---|---|---|
| API server metrics | /metrics on the metrics port 9090 of each API server pod | Yes |
| Worker metrics | /metrics on port 9090 of each worker pod | Yes |
| NATS JetStream | The nats command-line tool, and an optional exporter | No |
| gRPC service | None; the Kubernetes liveness probe uses the gRPC health service | Not applicable |
| Redis-compatible cache and PostgreSQL | No exporter installed by the charts | No |
| Logs | Container output of every component | Not applicable |
| Traces | OpenTelemetry export from the API server and the workers | Not applicable |
| Statistics | /statistics on the API, with an admin API key | Not applicable; the route reads the bundled Prometheus and NATS JetStream |
The API server serves /metrics on a separate metrics port, set by app.metrics_port (9090), and not on the API port 8000. The Kubernetes Service and the ingress expose only the API port. Security hardening lists the routes that need no API key.
Prometheus scrape scope
The core chart turns off each default job of the Prometheus chart and adds one job, foundation4ai-pods. The job discovers pods in every namespace and keeps a pod only when the pod carries both annotations prometheus.io/app: foundation4ai and prometheus.io/scrape: "true". The job reads the path and port from the prometheus.io/path and prometheus.io/port annotations, keeps the labels that a target exposes, scrapes every 10 seconds with a 2-second timeout, and ignores pods that are not running.
Each series carries the pod labels, with dots, slashes and hyphens replaced by underscores, and the labels namespace, pod and node. The label app_kubernetes_io_name separates the API server (api-server) from the workers (api-server-worker).
The API server and worker pods carry both annotations with path /metrics and port 9090, so the job collects the metrics of both. Queries, dashboards and alerts that select the job kubernetes-pods or foundation4ai-workers select job="foundation4ai-pods" instead.
Prometheus access and targets
-
Forward a local port to Prometheus in a separate terminal:
kubectl -n foundation4ai port-forward svc/foundation4ai-core-prometheus-server 9090:80Expected result:
Forwarding from 127.0.0.1:9090 -> 9090. The Prometheus user interface is athttp://localhost:9090. -
List the active targets and the health of each:
curl -s 'http://localhost:9090/api/v1/targets?state=active' | grep -o '"scrapeUrl":"[^"]*"\|"health":"[^"]*"'Expected result: one
scrapeUrlending in:9090/metricswith"health":"up"for each API server pod and for each worker pod.
Worker scrape job
The foundation4ai-pods job collects the worker metrics, so the core release needs no separate worker job. A values file that still holds a worker job under prometheus.extraScrapeConfigs, such as foundation4ai-workers, collects every worker series twice, under two job names. The following steps remove the job:
-
Delete the worker job from
prometheus.extraScrapeConfigsin the values file of the core release, and deleteprometheus.extraScrapeConfigswhen no other job remains.Expected result: the values file holds no
foundation4ai-workersjob. -
Upgrade the core release:
helm upgrade --install foundation4ai-core ./charts/foundation4ai-core \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result: Helm reports
STATUS: deployed. -
Repeat step 2 of Prometheus access and targets.
Expected result: one
scrapeUrlending in:9090/metricsfor each worker pod, not two.
API server metrics
Metrics is the complete metric catalog, with labels, label values and counting rules. The following tables summarize the metrics for the queries on this page.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
axum_http_requests_total | Counter | method, endpoint, status | HTTP requests handled. endpoint is the route template, such as /pipelines/{pipeline_id}/search, or unmatched for a path that matches no route. |
axum_http_requests_duration_seconds | Summary | method, endpoint, status | Seconds from the arrival of the request to the response head, as quantiles computed by each API server pod over about the last minute, with _sum and _count series. The time spent sending a streamed response is not included. |
axum_http_requests_pending | Gauge | method, endpoint | Requests in progress, including streamed responses that are still sending |
foundation4ai_document_queue_added | Counter | None | Documents accepted and queued for processing, through the REST API or the MCP server. Refused requests are not counted. |
foundation4ai_agent_executions | Counter | None | Agent executions started, through the REST API or the MCP server |
foundation4ai_agent_execution_latency | Histogram | None | Seconds from the start of an agent execution until the answer starts streaming: retrieval, prompt rendering and the setup of the LLM call |
The requests to the metrics port are not counted.
Worker metrics
The worker exports the following metrics on port 9090, each with a help line. The counter names have no _total suffix.
| Metric | Type | Meaning |
|---|---|---|
foundation4ai_document_queue_processed | Counter | Processing attempts, retries included: each document is counted every time the worker tries the document |
foundation4ai_document_queue_success | Counter | Documents processed successfully, documents with empty text included, each counted once |
foundation4ai_document_queue_removed | Counter | Queued documents found deleted, or whose pipeline was deleted, when the worker took the job |
foundation4ai_document_queue_skip | Counter | Deliveries skipped because the document was already processed, replaced or canceled |
foundation4ai_document_queue_retries | Counter | Failed attempts that the worker retries later, one per document and attempt |
foundation4ai_document_queue_failed | Counter | Documents given up on, each counted once: documents marked failed after the last attempt, and queue messages that cannot be read |
foundation4ai_document_queue_processed_latency | Histogram | Seconds to process one pipeline group of a batch |
foundation4ai_document_queue_text_splitter_latency | Gauge | Seconds spent splitting text in the last processed pipeline group |
foundation4ai_document_queue_embedding_latency | Gauge | Seconds spent computing embeddings in the last processed pipeline group |
foundation4ai_worker_heartbeat_timestamp_seconds | Gauge | Last time the processing loop ran, in Unix time. A value that stops increasing means a stuck worker. |
The histograms of the API server and the worker use the buckets 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120 and 300 seconds. Every value restarts at 0 when the process restarts.
Accumulating values
The Foundation4 counters, and the _sum, _count and _bucket series of the histograms and of the summary, accumulate from process start, so a raw value depends on the uptime of the pod. Dashboards and alerts use the following functions:
- Rates.
rate(<metric>[5m])gives the change per second over 5 minutes and treats a drop to 0 as a restart. - Totals over a period.
increase(<metric>[1h])gives the change over 1 hour. - Deployment totals.
sum(rate(<metric>[5m]))adds the rates of all pods. - Averages. The rate of a
_sumseries divided by the rate of the matching_countseries gives the average duration. - Percentiles.
histogram_quantileover the rates of the_bucketseries of a histogram gives a percentile across all pods.
The gauges hold a current value and are read without rate. The quantile series of axum_http_requests_duration_seconds are computed by each pod and are also read without rate; quantiles of several pods are not added. Prometheus can add an informational note to rate queries on the counters, because the counter names do not end in _total.
Recommended panels
| Panel | Query |
|---|---|
| Documents submitted per minute | sum(rate(foundation4ai_document_queue_added[5m])) * 60 |
| Documents processed per minute | sum(rate(foundation4ai_document_queue_success[5m])) * 60 |
| Retried attempts per minute | sum(rate(foundation4ai_document_queue_retries[5m])) * 60 |
| Documents failed per hour | sum(increase(foundation4ai_document_queue_failed[1h])) |
| Skipped jobs per minute | sum(rate(foundation4ai_document_queue_skip[5m])) * 60 |
| Processing seconds per pipeline group, average | sum(rate(foundation4ai_document_queue_processed_latency_sum[5m])) / sum(rate(foundation4ai_document_queue_processed_latency_count[5m])) |
| Processing seconds per pipeline group, 95th percentile | histogram_quantile(0.95, sum by (le) (rate(foundation4ai_document_queue_processed_latency_bucket[5m]))) |
| Embedding and splitting seconds of the last group, by worker pod | foundation4ai_document_queue_embedding_latency and foundation4ai_document_queue_text_splitter_latency |
| Worker heartbeat age, by worker pod | time() - foundation4ai_worker_heartbeat_timestamp_seconds |
| Agent executions per minute | sum(rate(foundation4ai_agent_executions[5m])) * 60 |
| Agent time to first answer, 95th percentile | histogram_quantile(0.95, sum by (le) (rate(foundation4ai_agent_execution_latency_bucket[5m]))) |
| API requests per second by status | sum by (status) (rate(axum_http_requests_total[5m])) |
| API server error ratio | sum(rate(axum_http_requests_total{status=~"5.."}[5m])) / sum(rate(axum_http_requests_total[5m])) |
| API latency, average by route | sum by (endpoint) (rate(axum_http_requests_duration_seconds_sum[5m])) / sum by (endpoint) (rate(axum_http_requests_duration_seconds_count[5m])) |
| API latency, 95th percentile by route, slowest pod | max by (endpoint) (axum_http_requests_duration_seconds{quantile="0.95"}) |
| API requests in progress | sum(axum_http_requests_pending) |
| Targets up | up{job="foundation4ai-pods"} |
The queue depth comes from NATS JetStream, as described in Queue state. No metric reports the database connection pool; the API server error ratio and the pool warning described in Log messages show pool pressure.
Recommended alerts
The thresholds depend on the workload and are set after a period of measurement:
| Alert | Condition |
|---|---|
| Processing failures | sum(increase(foundation4ai_document_queue_failed[15m])) > <failures> |
| Processing retries | sum(increase(foundation4ai_document_queue_retries[15m])) > <retries> |
| Processing stopped | sum(rate(foundation4ai_document_queue_added[30m])) > 0 and sum(rate(foundation4ai_document_queue_success[30m])) == 0 |
| Worker stuck | time() - foundation4ai_worker_heartbeat_timestamp_seconds > <seconds>, below the 900 seconds after which the liveness probe restarts the worker |
| Worker target down | up{job="foundation4ai-pods", app_kubernetes_io_name="api-server-worker"} == 0 for 5 minutes |
| API server target down | up{job="foundation4ai-pods", app_kubernetes_io_name="api-server"} == 0 for 5 minutes |
| API server errors | The error ratio query of Recommended panels above <ratio> for 10 minutes |
| Oldest queued job | Age of the first message of the stream DOCUMENTS above <hours>, well below 24 hours, from Queue state |
The bundled Prometheus evaluates rules from prometheus.serverFiles.alerting_rules.yml, but the core chart disables Alertmanager, so the bundled installation delivers no alert. A production deployment evaluates these conditions in the monitoring stack of the organization, which scrapes the same targets or receives the series from the bundled Prometheus.
Queue state
The utility Deployment foundation4ai-core-nats-box carries the nats command-line tool. The stream DOCUMENTS holds the processing jobs, and the consumer document-processor delivers the jobs to the workers. NATS JetStream accepts only authenticated clients: the user is foundation4ai and the password is the NATS_PASSWORD value of the operator Secret foundation4ai-secrets.
NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
kubectl -n foundation4ai exec deploy/foundation4ai-core-nats-box -- \
nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" \
stream info DOCUMENTS
kubectl -n foundation4ai exec deploy/foundation4ai-core-nats-box -- \
nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" \
consumer info DOCUMENTS document-processor
Expected result: the stream information shows the number of messages and the sequence and time of the first message. The consumer information shows delivery and acknowledgement state. The field names below are those of the nats tool and can differ between versions of the tool:
| Field | Reading |
|---|---|
Stream Messages | Jobs in the queue, including jobs that a worker is processing. This value is the queue depth. |
Stream First Sequence time | Submission time of the oldest job. A job older than 24 hours is removed. |
Consumer Unprocessed Messages | Jobs not yet delivered to a worker. A value that rises during a load means that the workers process slower than the client application submits. |
Consumer Outstanding Acks | Jobs delivered to a worker and not yet acknowledged |
Consumer Redelivered Messages | Jobs delivered more than once, after a failure or an expired acknowledgement wait |
Consumer Last Ack time | Time of the last acknowledgement. An old time while jobs remain means that no worker completes jobs. |
Consumer Waiting Pulls | Fetch requests from workers. A value of 0 while workers should be running means that no worker is connected. |
The NATS chart includes an exporter that publishes NATS server and JetStream metrics, including stream and consumer state. The core scrape job keeps the NATS pods only with the Foundation4 annotations, which the following untested values add:
nats:
promExporter:
enabled: true
podTemplate:
merge:
metadata:
annotations:
prometheus.io/app: foundation4ai
prometheus.io/scrape: "true"
prometheus.io/port: "7777"
The metric names depend on the exporter version; the exporter lists the names at http://<nats pod>:7777/metrics.
Health checks
/healthz on the API server checks the components that the API server uses: a query to PostgreSQL, the connection to NATS JetStream, a PING to the Redis-compatible cache and the gRPC health service of the gRPC service in the same pod. The checks run at the same time, and a component that gives no answer within 3 seconds counts as failed. With every component available, /healthz returns status 200:
{"status":"ok","database":"ok","message_queue":"ok","cache":"ok","llm_service":"ok"}
When a component fails, /healthz returns status 503 with "status":"error" and the value "error" for the failing component, and the API server logs Health check failed at WARN with the component and the error. The chart uses /healthz for the readiness probe of the server container, so an API server pod with a failing component leaves the Service without a restart. The probes of the pods are the following:
- API server liveness. The liveness probe calls the root route
/, which answersFoundation4.ai APIwhile the process serves HTTP. An outage of a shared component therefore restarts no API server pod. - API server startup. A startup probe on
/, every 10 seconds for up to 30 failures, gives the API server about 5 minutes to start, while startup waits for PostgreSQL, the cache and NATS JetStream.api-server.server.startupProbesets the probe, and the valuenullremoves the probe. - Worker liveness. The processing loop of the worker writes the current time to the heartbeat file
/tmp/foundation4ai-worker-heartbeat, at most every 10 seconds, and to the metricfoundation4ai_worker_heartbeat_timestamp_seconds. The liveness probe reads the file and restarts a worker whose loop has not run for 15 minutes. - gRPC service liveness. The liveness probe of the
grpccontainer calls the gRPC health service.
The following checks test the components:
| Check | Command | Expected result |
|---|---|---|
| Components of the API server | curl -s -w "\n%{http_code}\n" "$FOUNDATION4_URL/healthz" | The JSON response with "status":"ok", then 200 |
| Database connection from an API server pod | kubectl -n foundation4ai exec deploy/foundation4ai-api-server -c server -- ./foundation4ai database check-connection | Ok. |
| Authenticated API request that reads the database | curl -s -o /dev/null -w "%{http_code}\n" -H "x-api-key: $FOUNDATION4_API_KEY" -H "x-api-key-secret: $FOUNDATION4_API_SECRET" "$FOUNDATION4_URL/pipelines?first=1" | 200 |
| Queue | The stream command of Queue state | The stream information, with no error |
| gRPC service | kubectl -n foundation4ai get pods | API server and worker pods 2/2 ready; the liveness probe calls the gRPC health service |
| Worker processing loop | kubectl -n foundation4ai exec deploy/foundation4ai-api-server-worker -c worker -- cat /tmp/foundation4ai-worker-heartbeat | A Unix time less than a minute old |
| Worker metrics listener | kubectl -n foundation4ai port-forward deploy/foundation4ai-api-server-worker 9091:9090, then curl -s http://localhost:9091/health | OK |
| Prometheus | With the port forward of Prometheus access and targets, curl -s http://localhost:9090/-/ready | A message that Prometheus is ready |
An external availability check calls /healthz through the ingress, and the authenticated request with a dedicated API key that holds read permission on one pipeline, which also tests the API key check. Manage API keys and permissions describes such keys. The /statistics route returns document and agent execution totals and latencies over an interval, read from the bundled Prometheus, and the number of queued jobs, and requires an admin API key. Metrics describes each field.
Logs
Every component writes logs to the container output only; no component writes log files. The retention of the logs follows the log collection of the cluster.
| Component | Command |
|---|---|
| API server | kubectl -n foundation4ai logs deploy/foundation4ai-api-server -c server |
| gRPC service in the API server pods | kubectl -n foundation4ai logs deploy/foundation4ai-api-server -c grpc |
| Workers, all pods | kubectl -n foundation4ai logs -l app.kubernetes.io/name=api-server-worker -c worker --prefix --tail=500 |
| Hook Jobs | kubectl -n foundation4ai logs job/foundation4ai-api-server-db-migration, and the same for foundation4ai-api-server-license-check and foundation4ai-api-server-create-admin-api-key. Helm deletes a Job that succeeds, so the log is available while the Job runs, and after a failure until the next installation or upgrade. |
| Dashboard | kubectl -n foundation4ai logs deploy/foundation4ai-dashboard |
| NATS JetStream | kubectl -n foundation4ai logs foundation4ai-core-nats-0 -c nats |
app.log_level sets the level of the API server and the workers, with the default warn; the gRPC service logs at INFO. Configuration reference describes the levels, the output formats and the procedure that changes the level. A production deployment keeps warn outside a limited diagnosis period, as described in Security hardening.
The API server returns an x-request-id header on every response of the API port. The exported traces carry the same value in the attribute request_id, and a log line that the API server writes while handling a request carries request_id="<value>", with the value in quotation marks, in the prefix that names the HTTP request span. The same prefix carries the query string of the request as url.query. The API server writes one line for each request: request failed at WARN for a status 5xx, request refused at INFO for a status 4xx, and request at DEBUG for other statuses. The line carries the method, the path without the query string as path, the status, the duration in milliseconds as elapsed_ms, and the request ID. At warn, only the requests that fail with a status 5xx produce a line.
Log messages
| Component | Level | Message | Meaning |
|---|---|---|---|
| Worker | INFO | Successfully processed document <document> for pipeline <pipeline> | Document processed, including a document with empty text |
| Worker | WARN | Failed processing document <document> for pipeline <pipeline>: <reason> | One document failed; the job returns after a delay that starts at 10 seconds and doubles up to 10 minutes |
| Worker | WARN | Giving up on a document after repeated processing failures | The document failed its 12th attempt and is now failed, with the status message Processing failed after 12 attempts: <reason> |
| Worker | ERROR | Error processing documents for pipeline | A pipeline group failed; every job of the group returns after the same delay. For an internal error, the reason stored on the documents is internal error; see the worker logs. |
| Worker | WARN | Skipped document <document> for pipeline <pipeline>: <status> | The document was no longer pending, such as a job delivered again after the document was processed |
| Worker | WARN | Document <document> for pipeline <pipeline> was removed before processing | The document or the pipeline was deleted before processing |
| Worker | ERROR | Failed to acknowledge message: <error> | The worker could not acknowledge a job to NATS JetStream |
| Worker | ERROR | Error retrieving messages from message queue | The worker could not fetch jobs from NATS JetStream; the worker tries again after 2 seconds and keeps running |
| Worker | ERROR | Undecodable queue message; terminating it | A job could not be read; the job leaves the queue and counts in foundation4ai_document_queue_failed |
| Worker | ERROR | Processing a batch panicked | An unexpected error stopped one batch; the jobs of the batch return to the queue and the worker keeps running |
| Worker | WARN | Could not write the worker heartbeat file | The liveness probe reads no current heartbeat and restarts the worker |
| Worker | ERROR | The license is no longer valid: the worker stops; queued documents wait for a renewed license | The license expired while the worker ran. The worker exits, and a restarted worker does not start until a renewed license is deployed, as described in Licensing. |
| API server and worker | WARN | Waiting for the database, cache or queue; retrying in <delay> | PostgreSQL, the cache or NATS JetStream does not answer at startup; the process tries 30 times, for about 4.5 minutes, before the process stops |
| API server and worker | WARN | Failed to validate license for system id: <system id> | License rejected at startup; the process stops |
| API server and worker | Process exit | License check failed: <reason> | The license is invalid, expired, of an unsupported version or over a limit. The API server writes the message after Failed to initialize server: . |
| API server and worker | Process exit | Failed to parse config: <error> | A configuration key is missing or invalid |
| API server and worker | Process exit | Failed to connect to Foundation4.ai's services: <error> | Startup failed on the database, the cache, NATS JetStream or the application secret. The error starts with the failing part, such as ORM error:, Redis error:, Nats error: or Decryption error:, and the API server writes the message after Failed to initialize server: . |
| API server | ERROR | The master key is expired, inactive or missing, so requests made with it are refused; ... | The API server keeps running and refuses the master key until an operator restores the key, as described in Secrets and keys |
| API server | Process exit | Failed to initialize server: Failed to initialize with the master key: <error> | The master key secret does not match the key in the database |
| Worker | Process exit | Failed to initialize Foundation4.ai: <error> | The master key check failed in a worker, including for an expired, inactive or missing master key |
| API server | Process exit | Failed to bind the metrics address: <reason> or app.metrics_port must differ from app.port | The metrics port cannot be used |
| API server | ERROR | Failed to refresh license object count cache | The refresh of the license object counts, every 10 minutes, failed |
| API server | ERROR | Failed to refresh license document count cache | The refresh of the document and fragment counts, every 10 seconds, failed, as described in Licensing |
| API server | WARN | request failed | A request ended with a status 5xx |
| API server | ERROR | Internal server error | A request failed with status 500; the line carries the kind and the error that the response leaves out |
| API server | ERROR | Could not validate the API key | The API key check could not read the database or the cache; the request receives status 503 |
| API server | ERROR | Failed queueing document for processing | A created document could not be queued in NATS JetStream and has the status failed |
| API server | WARN | Health check failed | A component failed the /healthz check; the line names the component and the error |
| API server and worker | WARN | no env var set or infered for OTEL_EXPORTER_OTLP_<SIGNAL>_PROTOCOL or OTEL_EXPORTER_OTLP_PROTOCOL; no <kind> exporter will be created | No OpenTelemetry exporter is configured. Two lines appear at each start of a process without the variables described in OpenTelemetry tracing: LOGS with log, and TRACES with span |
| API server | WARN | acquired connection, but time to acquire exceeded slow threshold | A request waited more than 2 seconds for a database connection |
| gRPC service | INFO | INFO:processor:gRPC server started on 127.0.0.1:50051 | The gRPC service is ready |
| gRPC service | INFO | Added package directory to site: <path> | A package image was loaded |
The lines at INFO from the API server and the workers appear only when app.log_level is info, debug or trace.
OpenTelemetry tracing
The API server and the workers export spans and log records through the OpenTelemetry Protocol (OTLP) when an OTLP variable is set. The API server exports a span for each API request, with the request ID, and a request that carries trace context headers joins the trace of the caller. A document queued through the REST API or the MCP server carries the trace context of the request, so the process_documents span of the worker joins the trace of the request that queued the document. When one batch holds documents of several requests, the span links to each of these traces instead. The process_documents span has the level INFO, so a worker exports the span when app.log_level is info, debug or trace, and exports log records only at warn. The gRPC service and the dashboard export no traces, so the text splitting and embedding calls of the gRPC service appear in no trace. The charts install no collector, so the collector runs in the cluster or elsewhere inside the network. Configuration reference lists the variables.
-
Add the variables to the values file of the application release. The entries of
api-server.configsreach both the API server and the workers. Values that contain credentials, such asOTEL_EXPORTER_OTLP_HEADERS, go underapi-server.secretsinstead:api-server:configs:OTEL_EXPORTER_OTLP_ENDPOINT: "http://<collector host>:4317"OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"Expected result: the values file holds the two entries. Without
OTEL_SERVICE_NAME, the API server reports the service namefoundation4aiand the workers reportfoundation4ai-worker;OTEL_SERVICE_NAMEsets one name for both. -
Upgrade the application release:
helm upgrade --install foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mkubectl -n foundation4ai rollout status deploy/foundation4ai-api-serverkubectl -n foundation4ai rollout status deploy/foundation4ai-api-server-workerExpected result: Helm reports
STATUS: deployed, and both status commands end withsuccessfully rolled out. A changed configuration restarts the API server and worker pods during the upgrade. -
Confirm the variables in the
serverandworkercontainers:kubectl -n foundation4ai exec deploy/foundation4ai-api-server -c server -- env | grep '^OTEL_'kubectl -n foundation4ai exec deploy/foundation4ai-api-server-worker -c worker -- env | grep '^OTEL_'Expected result: the two variables with the values from step 1, for each container.
-
Send a request and read the request ID from the response headers:
curl -s -D - -o /dev/null -X POST "$FOUNDATION4_URL/login" \-H "x-api-key: $FOUNDATION4_API_KEY" \-H "x-api-key-secret: $FOUNDATION4_API_SECRET" | grep -i '^x-request-id'Expected result: an
x-request-idheader. The trace backend of the collector shows a trace of servicefoundation4aiwhose span carries the same value inrequest_id. Withapp.log_levelatinfoor a more detailed level, a document created after this step adds aprocess_documentsspan of servicefoundation4ai-workerto the trace of the create request.