Scaling and performance
This page describes how each Foundation4 component scales, how the workers process documents, and how resource settings, database connections, autoscaling and the queue retention limit the capacity of a deployment. Operators who size a production deployment or plan a large document load need this page. Values reference lists every value that this page names.
Components and replicas
| Component | Default replicas | Scaled by | Scaling notes |
|---|---|---|---|
| API server | 1 | api-server.replicaCount.server or autoscaling | Stateless for the REST API and the MCP endpoint, which keeps no sessions, so any replica answers any request and the ingress needs no session affinity. Each replica holds a database pool and loads each model that the replica uses. |
| Worker | 3 | api-server.replicaCount.worker or autoscaling | Each replica processes one batch of jobs at a time with one database connection |
| gRPC service | One per API server pod and one per worker pod | The pod count | Runs as a sidecar container, a second container in each pod, with the resource settings of api-server.grpc.resources or, without them, api-server.resources |
| Dashboard | 1 | dashboard.replicaCount | Serves the dashboard files; carries no processing load |
| NATS JetStream | 3 | nats.config.cluster.replicas | Holds the queue and the submitted document text for 24 hours, in global.natsReplicas copies, 3 by default |
| Redis-compatible cache | 1 | Not scaled by the charts | Holds caches of API keys, pipeline, agent and text splitter records, and execution traces in memory |
| PostgreSQL with the vector extensions | External in production | Outside the charts | Primary data store; the connection budget limits the Foundation4 replicas |
The API server and the workers scale independently. API server replicas add capacity for requests: search, document submission and agent execution. Worker replicas add capacity for document processing.
API server capacity
Every search in similarity or MMR mode embeds the query text before PostgreSQL runs the vector search. The embedding runs where the embedding model's provider runs:
- FastEmbed. Inside the API server process, on a separate thread pool, so the threads that serve requests stay free during inference. One process loads up to
app.fastembed_instancescopies of each model, 2 in the chart throughapi-server.fastembedInstances, and loads a further copy only when calls overlap. Each call embeds at most 32 texts, and a call that finds every copy busy waits its turn in arrival order, so a search waits for at most one part of a large embedding job. - gRPC service providers. In the gRPC service container of the same pod, which serves up to 32 requests at a time.
- OpenAI-compatible embeddings. On the embeddings server, outside the pod.
For FastEmbed models, concurrent search capacity grows with the number of copies and with the number of API server replicas. Each copy takes as much memory as the model: about 90 MB for all-MiniLM-L6-v2, and over 1 GB for large models. Agents and direct queries add the latency of the registered model server, which the API server waits for.
Each API server and worker process loads an embedding model on first use and keeps up to 32 embedding models loaded, unloading a model that stays unused for 1 hour. The gRPC service keeps up to 8 embedding models and 8 text splitters loaded and unloads the least recently used one beyond that number; the variable F4AI_MODEL_CACHE_SIZE, set through api-server.grpcExtraEnv, changes the number. Memory therefore grows with the number of distinct embedding models and text splitters that the deployment uses, and the first request after a restart, a scale-up or an unload is slower while the model loads.
Worker processing
Each worker process runs one loop:
- License. The worker checks the license. A license that is no longer valid stops the worker, as described in Licensing.
- Fetch. The worker takes up to 100 jobs from NATS JetStream, waiting at most 250 milliseconds for the batch to fill, and marks each job of the batch as in progress.
- Group. The worker groups the jobs of the batch by pipeline and processes the groups one after another.
- Split. For each document of a group, the worker sends the text to the text splitter in the gRPC service.
- Embed. The worker requests the embeddings of all fragments of the group for each embedding model, with at most 1,000 fragments in one request to the provider.
- Store. The worker writes the fragments and vectors to PostgreSQL in one transaction per group and acknowledges the jobs.
The batch size of 100 and the sequential processing are fixed in the code; no value changes the batch size or the order. Throughput therefore scales with the number of worker replicas. Next to the loop, each worker fills the search indexes that are added to pipelines that already hold documents, one batch at a time with a pause between batches, so a search index build shares the workers with document processing.
The following behaviors reduce throughput when processing fails:
- Failed documents. A splitting or embedding error in one document fails that document only. The worker returns the job to the queue with a delay that starts at 10 seconds and doubles up to 10 minutes, and marks the document
failedafter 12 attempts, about one hour after the first attempt. An error that covers the whole group, such as a database error, returns every job of that pipeline group in the same way, and the other groups of the batch continue. - Long batches. The queue redelivers a job that stays unacknowledged longer than the acknowledgement wait of 10 minutes, counted from the in-progress mark of the fetch step, so a batch that runs longer can be processed twice.
- Queue errors. A queue read error makes the worker wait 2 seconds and fetch again. A job that the worker cannot decode is removed from the queue and counted as failed, and the worker continues.
Ingest documents reliably describes the retries from the point of view of the client application.
Resource requests and limits
The charts set no resource requests or limits for any container. The following keys set requests and limits:
| Component | Key |
|---|---|
API server container server | api-server.server.resources, or api-server.resources |
Worker container worker | api-server.worker.resources, or api-server.resources |
Both gRPC service containers grpc | api-server.grpc.resources, or api-server.resources |
| Hook Jobs | api-server.jobs.resources, or api-server.resources |
| Dashboard | dashboard.resources |
| NATS JetStream | nats.container.resources |
| Redis-compatible cache | redis.resources |
| Prometheus | prometheus.server.resources |
| Bundled PostgreSQL | postgres.resources |
api-server.resources applies to every container of the api-server chart that has no resources value of its own. The containers have different needs:
- FastEmbed models. Loaded in the
serverandworkercontainers, in up toapp.fastembed_instancescopies per model. - GPT4All and Hugging Face models, and every text splitter. Loaded in the
grpccontainers, which usually need more memory than theserverandworkercontainers. - Worker batches. The text, fragments and vectors of up to 100 documents, so worker memory depends on document size and embedding dimensions.
Because no reference numbers exist, the operator measures a representative load before setting the values.
Resource settings
-
Run a representative load, such as a first round of documents and the expected search traffic, and read the usage of each container. The command needs the Kubernetes metrics server:
kubectl -n foundation4ai top pods --containersExpected result: CPU and memory usage for the
server,worker,grpcand dashboard containers. -
Set the values of each container from its highest usage, with headroom, in the values file of the application release:
api-server:server:resources:requests:cpu: <cpu request>memory: <memory request>limits:memory: <memory limit>worker:resources:requests:cpu: <cpu request>memory: <memory request>limits:memory: <memory limit>grpc:resources:requests:cpu: <cpu request>memory: <memory request>limits:memory: <memory limit>Expected result: the values file holds a
resourcesblock underserver,workerandgrpc. -
Upgrade the application release:
helm upgrade --install foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result: Helm reports
STATUS: deployed, and the API server and worker pods are replaced. -
Confirm the settings of the worker pod template:
kubectl -n foundation4ai get deploy foundation4ai-api-server-worker \-o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}'Expected result: the
workerline shows the values ofapi-server.worker.resources, and thegrpcline shows the values ofapi-server.grpc.resources. -
After a new load, check for containers stopped for exceeding the memory limit:
kubectl -n foundation4ai get pods \-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}{end}'Expected result: no line contains
OOMKilled. A line withOOMKilledcalls for a higher memory limit, fewer FastEmbed copies or smaller documents.
Database connections
Each API server replica opens up to database.pool_size connections, 3 in the chart and 5 when unset, and each worker opens 1. The total at the maximum replica counts must stay below the connection limit of PostgreSQL. With autoscaling at the chart maximum of 100 replicas for both Deployments and the chart pool size, the deployment can open 400 connections. Database describes the connection budget and the HTTP 500 responses that pool exhaustion produces.
Autoscaling
The chart creates a HorizontalPodAutoscaler for the API server when api-server.autoscaling.server.enabled is true, and one for the workers when api-server.autoscaling.worker.enabled is true. Each targets 80 percent CPU utilization by default and can add a memory target. While autoscaling is on, the Deployment carries no replica count.
The following conditions apply:
- Requests. Utilization is measured against the requests, so autoscaling needs requests on both containers of the pod,
serverorworkerandgrpc, and the Kubernetes metrics server. Without requests, the autoscaler reports the target as unknown and does not scale. - Whole pod. The autoscaler computes utilization over all containers of the pod, including the gRPC service container.
- Connection budget. The default
maxReplicasof 100 exceeds the connection limit of most PostgreSQL servers, somaxReplicasis set from the budget in Database connections. - Queue depth. The worker autoscaler follows CPU, not the number of queued jobs. The charts include no autoscaling on queue depth.
- Scale-down. On the termination signal that Kubernetes sends, the worker finishes the batch in progress and stops. A batch that outlasts the termination grace period, 30 seconds by default, ends with the pod, and the queue redelivers the jobs of that batch after the acknowledgement wait of 10 minutes. The jobs are delayed, not lost, while the jobs are younger than 24 hours.
- Model loading. A new pod loads each model on first use, so the first requests on a new replica are slower.
- Disruptions.
api-server.podDisruptionBudget.serverandapi-server.podDisruptionBudget.workerlimit voluntary evictions, such as node drains, andapi-server.priorityClassNamesets the scheduling priority of the API server and worker pods.
Worker autoscaling
-
Set requests for the
workerandgrpccontainers as described in Resource settings, then enable the worker autoscaler with a maximum that fits the connection budget:api-server:autoscaling:worker:enabled: trueminReplicas: 3maxReplicas: <maximum workers>targetCPUUtilizationPercentage: 80Expected result: the values file holds the
autoscaling.workerblock. -
Upgrade the application release:
helm upgrade --install foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result: Helm reports
STATUS: deployed. -
Check the autoscaler:
kubectl -n foundation4ai get hpa foundation4ai-api-server-workerExpected result: the
TARGETScolumn shows a percentage against80%, such ascpu: 12%/80%, and not<unknown>.
Queue retention and backfills
NATS JetStream keeps each processing job, and the submitted document text, for 24 hours. The queue removes a job that no worker completed within 24 hours, and the document stays pending until the client application submits the document again. The following situations let jobs reach that age:
- Backlog. A backfill submits documents faster than the workers process the documents.
- Repeated failures. Processing fails on every attempt, for example because a model is missing.
- Expired license. The workers stop when the license expires, as described in Licensing. The queued documents are processed when a renewed license is deployed within 24 hours.
A backfill therefore keeps the backlog at a size that the workers clear well within 24 hours. The client application submits in rounds and reads the pending count between rounds, as described in Pacing and batching. The operator measures the processing rate during the first rounds, as described in Throughput measurement, and divides the planned backlog by that rate.
The API server and the workers create the stream DOCUMENTS and the object store documents with nats.replicas copies, 3 in the chart through global.natsReplicas, so the queue survives the loss of one of the 3 NATS servers. At startup, the processes set the replica count of an existing stream and object store to the same value and keep the other settings. The NATS servers require the client user foundation4ai and the password NATS_PASSWORD of foundation4ai-secrets. The following commands show the stream configuration and state:
NATS_PASSWORD=$(kubectl -n foundation4ai get secret foundation4ai-secrets -o jsonpath='{.data.NATS_PASSWORD}' | base64 -d)
kubectl -n foundation4ai exec deploy/foundation4ai-core-nats-box -- \
nats --server nats://foundation4ai-core-nats:4222 --user foundation4ai --password "$NATS_PASSWORD" \
stream info DOCUMENTS
Expected result: the configuration shows work-queue retention, a maximum age of 24 hours and 3 replicas, and the state shows the number of queued messages.
Throughput measurement
The worker metrics measure processing. The core chart's scrape job collects them from every worker pod, as described in Monitoring and logging. The following queries run in the Prometheus user interface during a load:
| Measure | Query |
|---|---|
| Documents processed per hour | sum(rate(foundation4ai_document_queue_success[15m])) * 3600 |
| Average processing seconds per pipeline group | sum(rate(foundation4ai_document_queue_processed_latency_sum[15m])) / sum(rate(foundation4ai_document_queue_processed_latency_count[15m])) |
| Average processing seconds per document attempt | sum(rate(foundation4ai_document_queue_processed_latency_sum[15m])) / sum(rate(foundation4ai_document_queue_processed[15m])) |
| Embedding seconds of recent pipeline groups | avg(avg_over_time(foundation4ai_document_queue_embedding_latency[15m])) |
| Splitting seconds of recent pipeline groups | avg(avg_over_time(foundation4ai_document_queue_text_splitter_latency[15m])) |
| Retried attempts per hour | sum(rate(foundation4ai_document_queue_retries[15m])) * 3600 |
foundation4ai_document_queue_processed_latency is a histogram with one observation per pipeline group, and foundation4ai_document_queue_processed counts every attempt, retries included. The embedding and splitting gauges hold the times of the last pipeline group of each worker, so the averages over time sample recent groups. Comparing the embedding and splitting seconds shows which step to address: a different embedding model or provider, a larger chunk size, or more workers. The measured rate for a given embedding model, text splitter and document size is the reference for pacing backfills and for the worker count.
Metadata indexes
A metadata filter needs no index. The indexes that POST /pipelines/{id}/indexes creates are built on the same expression as the filters on their fields, and PostgreSQL decides for each query whether to use one, as described in Metadata indexes. Each index adds work to every fragment write of the pipeline, so a deployment creates metadata indexes only for a measured need.