Skip to main content

Scaling and performance

This page describes how each Foundation4 component scales, how the workers process documents, and how resource settings, database connections, autoscaling and the queue retention limit the capacity of a deployment. Operators who size a production deployment or plan a large document load need this page. Values reference lists every value that this page names.

Components and replicas​

ComponentDefault replicasScaled byScaling notes
API server1api-server.replicaCount.server or autoscalingStateless for the REST API and the MCP endpoint, which keeps no sessions, so any replica answers any request and the ingress needs no session affinity. Each replica holds a database pool and loads each model that the replica uses.
Worker3api-server.replicaCount.worker or autoscalingEach replica processes one batch of jobs at a time with one database connection
gRPC serviceOne per API server pod and one per worker podThe pod countRuns as a sidecar container, a second container in each pod, with the resource settings of api-server.grpc.resources or, without them, api-server.resources
Dashboard1dashboard.replicaCountServes the dashboard files; carries no processing load
NATS JetStream3nats.config.cluster.replicasHolds the queue and the submitted document text for 24 hours, in global.natsReplicas copies, 3 by default
Redis-compatible cache1Not scaled by the chartsHolds caches of API keys, pipeline, agent and text splitter records, and execution traces in memory
PostgreSQL with the vector extensionsExternal in productionOutside the chartsPrimary data store; the connection budget limits the Foundation4 replicas

The API server and the workers scale independently. API server replicas add capacity for requests: search, document submission and agent execution. Worker replicas add capacity for document processing.

API server capacity​

Every search in similarity or MMR mode embeds the query text before PostgreSQL runs the vector search. The embedding runs where the embedding model's provider runs:

  • FastEmbed. Inside the API server process, on a separate thread pool, so the threads that serve requests stay free during inference. One process loads up to app.fastembed_instances copies of each model, 2 in the chart through api-server.fastembedInstances, and loads a further copy only when calls overlap. Each call embeds at most 32 texts, and a call that finds every copy busy waits its turn in arrival order, so a search waits for at most one part of a large embedding job.
  • gRPC service providers. In the gRPC service container of the same pod, which serves up to 32 requests at a time.
  • OpenAI-compatible embeddings. On the embeddings server, outside the pod.

For FastEmbed models, concurrent search capacity grows with the number of copies and with the number of API server replicas. Each copy takes as much memory as the model: about 90 MB for all-MiniLM-L6-v2, and over 1 GB for large models. Agents and direct queries add the latency of the registered model server, which the API server waits for.

Each API server and worker process loads an embedding model on first use and keeps up to 32 embedding models loaded, unloading a model that stays unused for 1 hour. The gRPC service keeps up to 8 embedding models and 8 text splitters loaded and unloads the least recently used one beyond that number; the variable F4AI_MODEL_CACHE_SIZE, set through api-server.grpcExtraEnv, changes the number. Memory therefore grows with the number of distinct embedding models and text splitters that the deployment uses, and the first request after a restart, a scale-up or an unload is slower while the model loads.

Worker processing​

Each worker process runs one loop:

  1. License. The worker checks the license. A license that is no longer valid stops the worker, as described in Licensing.
  2. Fetch. The worker takes up to 100 jobs from NATS JetStream, waiting at most 250 milliseconds for the batch to fill, and marks each job of the batch as in progress.
  3. Group. The worker groups the jobs of the batch by pipeline and processes the groups one after another.
  4. Split. For each document of a group, the worker sends the text to the text splitter in the gRPC service.
  5. Embed. The worker requests the embeddings of all fragments of the group for each embedding model, with at most 1,000 fragments in one request to the provider.
  6. Store. The worker writes the fragments and vectors to PostgreSQL in one transaction per group and acknowledges the jobs.

The batch size of 100 and the sequential processing are fixed in the code; no value changes the batch size or the order. Throughput therefore scales with the number of worker replicas. Next to the loop, each worker fills the search indexes that are added to pipelines that already hold documents, one batch at a time with a pause between batches, so a search index build shares the workers with document processing.

The following behaviors reduce throughput when processing fails:

  • Failed documents. A splitting or embedding error in one document fails that document only. The worker returns the job to the queue with a delay that starts at 10 seconds and doubles up to 10 minutes, and marks the document failed after 12 attempts, about one hour after the first attempt. An error that covers the whole group, such as a database error, returns every job of that pipeline group in the same way, and the other groups of the batch continue.
  • Long batches. The queue redelivers a job that stays unacknowledged longer than the acknowledgement wait of 10 minutes, counted from the in-progress mark of the fetch step, so a batch that runs longer can be processed twice.
  • Queue errors. A queue read error makes the worker wait 2 seconds and fetch again. A job that the worker cannot decode is removed from the queue and counted as failed, and the worker continues.

Ingest documents reliably describes the retries from the point of view of the client application.

Resource requests and limits​

The charts set no resource requests or limits for any container. The following keys set requests and limits:

ComponentKey
API server container serverapi-server.server.resources, or api-server.resources
Worker container workerapi-server.worker.resources, or api-server.resources
Both gRPC service containers grpcapi-server.grpc.resources, or api-server.resources
Hook Jobsapi-server.jobs.resources, or api-server.resources
Dashboarddashboard.resources
NATS JetStreamnats.container.resources
Redis-compatible cacheredis.resources
Prometheusprometheus.server.resources
Bundled PostgreSQLpostgres.resources

api-server.resources applies to every container of the api-server chart that has no resources value of its own. The containers have different needs:

  • FastEmbed models. Loaded in the server and worker containers, in up to app.fastembed_instances copies per model.
  • GPT4All and Hugging Face models, and every text splitter. Loaded in the grpc containers, which usually need more memory than the server and worker containers.
  • Worker batches. The text, fragments and vectors of up to 100 documents, so worker memory depends on document size and embedding dimensions.

Because no reference numbers exist, the operator measures a representative load before setting the values.

Resource settings​

  1. Run a representative load, such as a first round of documents and the expected search traffic, and read the usage of each container. The command needs the Kubernetes metrics server:

    kubectl -n foundation4ai top pods --containers

    Expected result: CPU and memory usage for the server, worker, grpc and dashboard containers.

  2. Set the values of each container from its highest usage, with headroom, in the values file of the application release:

    api-server:
    server:
    resources:
    requests:
    cpu: <cpu request>
    memory: <memory request>
    limits:
    memory: <memory limit>
    worker:
    resources:
    requests:
    cpu: <cpu request>
    memory: <memory request>
    limits:
    memory: <memory limit>
    grpc:
    resources:
    requests:
    cpu: <cpu request>
    memory: <memory request>
    limits:
    memory: <memory limit>

    Expected result: the values file holds a resources block under server, worker and grpc.

  3. Upgrade the application release:

    helm upgrade --install foundation4ai ./charts/foundation4ai \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

    Expected result: Helm reports STATUS: deployed, and the API server and worker pods are replaced.

  4. Confirm the settings of the worker pod template:

    kubectl -n foundation4ai get deploy foundation4ai-api-server-worker \
    -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}'

    Expected result: the worker line shows the values of api-server.worker.resources, and the grpc line shows the values of api-server.grpc.resources.

  5. After a new load, check for containers stopped for exceeding the memory limit:

    kubectl -n foundation4ai get pods \
    -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}{end}'

    Expected result: no line contains OOMKilled. A line with OOMKilled calls for a higher memory limit, fewer FastEmbed copies or smaller documents.

Database connections​

Each API server replica opens up to database.pool_size connections, 3 in the chart and 5 when unset, and each worker opens 1. The total at the maximum replica counts must stay below the connection limit of PostgreSQL. With autoscaling at the chart maximum of 100 replicas for both Deployments and the chart pool size, the deployment can open 400 connections. Database describes the connection budget and the HTTP 500 responses that pool exhaustion produces.

Autoscaling​

The chart creates a HorizontalPodAutoscaler for the API server when api-server.autoscaling.server.enabled is true, and one for the workers when api-server.autoscaling.worker.enabled is true. Each targets 80 percent CPU utilization by default and can add a memory target. While autoscaling is on, the Deployment carries no replica count.

The following conditions apply:

  • Requests. Utilization is measured against the requests, so autoscaling needs requests on both containers of the pod, server or worker and grpc, and the Kubernetes metrics server. Without requests, the autoscaler reports the target as unknown and does not scale.
  • Whole pod. The autoscaler computes utilization over all containers of the pod, including the gRPC service container.
  • Connection budget. The default maxReplicas of 100 exceeds the connection limit of most PostgreSQL servers, so maxReplicas is set from the budget in Database connections.
  • Queue depth. The worker autoscaler follows CPU, not the number of queued jobs. The charts include no autoscaling on queue depth.
  • Scale-down. On the termination signal that Kubernetes sends, the worker finishes the batch in progress and stops. A batch that outlasts the termination grace period, 30 seconds by default, ends with the pod, and the queue redelivers the jobs of that batch after the acknowledgement wait of 10 minutes. The jobs are delayed, not lost, while the jobs are younger than 24 hours.
  • Model loading. A new pod loads each model on first use, so the first requests on a new replica are slower.
  • Disruptions. api-server.podDisruptionBudget.server and api-server.podDisruptionBudget.worker limit voluntary evictions, such as node drains, and api-server.priorityClassName sets the scheduling priority of the API server and worker pods.

Worker autoscaling​

  1. Set requests for the worker and grpc containers as described in Resource settings, then enable the worker autoscaler with a maximum that fits the connection budget:

    api-server:
    autoscaling:
    worker:
    enabled: true
    minReplicas: 3
    maxReplicas: <maximum workers>
    targetCPUUtilizationPercentage: 80

    Expected result: the values file holds the autoscaling.worker block.

  2. Upgrade the application release:

    helm upgrade --install foundation4ai ./charts/foundation4ai \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

    Expected result: Helm reports STATUS: deployed.

  3. Check the autoscaler:

    kubectl -n foundation4ai get hpa foundation4ai-api-server-worker

    Expected result: the TARGETS column shows a percentage against 80%, such as cpu: 12%/80%, and not <unknown>.

Queue retention and backfills​

NATS JetStream keeps each processing job, and the submitted document text, for 24 hours. The queue removes a job that no worker completed within 24 hours, and the document stays pending until the client application submits the document again. The following situations let jobs reach that age:

  • Backlog. A backfill submits documents faster than the workers process the documents.
  • Repeated failures. Processing fails on every attempt, for example because a model is missing.
  • Expired license. The workers stop when the license expires, as described in Licensing. The queued documents are processed when a renewed license is deployed within 24 hours.

A backfill therefore keeps the backlog at a size that the workers clear well within 24 hours. The client application submits in rounds and reads the pending count between rounds, as described in Pacing and batching. The operator measures the processing rate during the first rounds, as described in Throughput measurement, and divides the planned backlog by that rate.

The API server and the workers create the stream DOCUMENTS and the object store documents with nats.replicas copies, 3 in the chart through global.natsReplicas, so the queue survives the loss of one of the 3 NATS servers. At startup, the processes set the replica count of an existing stream and object store to the same value and keep the other settings. The NATS servers require the client user foundation4ai and the password NATS_PASSWORD of foundation4ai-secrets. The following commands show the stream configuration and state:

NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
kubectl -n foundation4ai exec deploy/foundation4ai-core-nats-box -- \
nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" \
stream info DOCUMENTS

Expected result: the configuration shows work-queue retention, a maximum age of 24 hours and 3 replicas, and the state shows the number of queued messages.

Throughput measurement​

The worker metrics measure processing. The core chart's scrape job collects them from every worker pod, as described in Monitoring and logging. The following queries run in the Prometheus user interface during a load:

MeasureQuery
Documents processed per hoursum(rate(foundation4ai_document_queue_success[15m])) * 3600
Average processing seconds per pipeline groupsum(rate(foundation4ai_document_queue_processed_latency_sum[15m])) / sum(rate(foundation4ai_document_queue_processed_latency_count[15m]))
Average processing seconds per document attemptsum(rate(foundation4ai_document_queue_processed_latency_sum[15m])) / sum(rate(foundation4ai_document_queue_processed[15m]))
Embedding seconds of recent pipeline groupsavg(avg_over_time(foundation4ai_document_queue_embedding_latency[15m]))
Splitting seconds of recent pipeline groupsavg(avg_over_time(foundation4ai_document_queue_text_splitter_latency[15m]))
Retried attempts per hoursum(rate(foundation4ai_document_queue_retries[15m])) * 3600

foundation4ai_document_queue_processed_latency is a histogram with one observation per pipeline group, and foundation4ai_document_queue_processed counts every attempt, retries included. The embedding and splitting gauges hold the times of the last pipeline group of each worker, so the averages over time sample recent groups. Comparing the embedding and splitting seconds shows which step to address: a different embedding model or provider, a larger chunk size, or more workers. The measured rate for a given embedding model, text splitter and document size is the reference for pacing backfills and for the worker count.

Metadata indexes​

A metadata filter needs no index. The indexes that POST /pipelines/{id}/indexes creates are built on the same expression as the filters on their fields, and PostgreSQL decides for each query whether to use one, as described in Metadata indexes. Each index adds work to every fragment write of the pipeline, so a deployment creates metadata indexes only for a measured need.