Models, packages and air-gapped installs
This page describes how Foundation4 delivers embedding models and Python packages, how an operator adds model and package images to a deployment, and which images an air-gapped installation mirrors. The page also states which embedding providers, text splitters and language model features work without network access. Operators who plan an air-gapped installation or add embedding models need this page. Providers and models lists the parameters and network access of every provider.
Model and package delivery
Foundation4 runs the API server and the workers from the same image. Each API server pod and each worker pod also runs the gRPC service as a sidecar container, a second container in the same Kubernetes pod, which hosts the Python embedding providers and every text splitter.
| Source | Content | Consumers |
|---|---|---|
| API server image | The Foundation4 binary and the FastEmbed model files listed in Default model shadowing, in /data/fastembed | API server and worker processes |
| gRPC service image | The Python service, the LangChain libraries and the NLTK and tiktoken packages, without the GPT4All, Sentence Transformers and spaCy packages | gRPC service |
| Package image | One Python package with dependencies: gpt4all, sentence-transformers or spacy | gRPC service, mounted at /packages/<name> |
| Model image | Model files for one provider, such as GPT4All GGUF files, NLTK data or tiktoken encodings | Every container of the API server and worker pods, mounted at the path in mount |
Model and package images use the Kubernetes image volume source: the kubelet pulls an OCI image and mounts the image content read-only, without running the image. The cluster therefore needs image volume support in the API server, the kubelet and the container runtime. The application chart requires Kubernetes 1.31 or later, and the operator confirms that the image volume feature is enabled before adding a model or package image. Deployment overview and requirements lists the cluster requirements. An installation that uses only the models in the API server image, such as the evaluation profile in Install for evaluation, needs no image volumes.
Values for model and package images
Model and package images are declared under api-server in the values file of the application release:
api-server:
packages:
- name: gpt4all
image:
reference: <registry>/<package image repository>:<tag>
pullPolicy: IfNotPresent
models:
- name: gpt4all
image:
reference: <registry>/<model image repository>:<tag>
pullPolicy: IfNotPresent
mount: /data/gpt4all
| Key | Effect |
|---|---|
packages[].name | Names the volume and the mount directory /packages/<name> in both gRPC service containers |
packages[].image | Copied unchanged into the image volume source: reference and pullPolicy |
packages[].imagePullSecrets | Plain names or name entries, added to every pod of the api-server chart |
models[].name | Names the volume |
models[].image | Copied unchanged into the image volume source: reference and pullPolicy |
models[].mount | The mount path in every container of the API server and worker pods |
models[].subPath | Optional. A directory of the image to mount in place of the image root, for an image that keeps the files in a directory |
models[].imagePullSecrets | Plain names or name entries, added to every pod of the api-server chart |
Every pod of the api-server chart, including the worker pods and the hook Jobs, receives global.imagePullSecrets, api-server.imagePullSecrets and the pull secrets of every model and package image. Values reference describes every key of the charts.
A change to packages or models changes the pod templates, so helm upgrade replaces the API server and worker pods without a manual restart.
Container mounts
Every container of the API server and worker pods mounts each model image the same way, at mount, so a model has the same path in the server, worker and grpc containers:
| Container | Pod | Package images | Model images | Writable caches |
|---|---|---|---|---|
server | API server | Not mounted | The image at mount | /cache/fastembed, /cache/huggingface |
grpc | API server | /packages/<name> | The image at mount | /cache/huggingface |
worker | Worker | Not mounted | The image at mount | /cache/fastembed, /cache/huggingface |
grpc | Worker | /packages/<name> | The image at mount | /cache/huggingface |
The model images keep the model files at the image root. An image that keeps the files in a directory, such as a model image of an earlier build with the files under data, takes subPath: data in its models entry. Subpath mounts of image volumes need a Kubernetes version that supports subpaths on image volumes.
The writable caches are temporary volumes of each pod, and their content is lost when the pod restarts. The chart omits /cache/huggingface when api-server.huggingfaceCache.enabled is false or when a model image is mounted at /data/huggingface.
Model directories
The paths configuration keys tell Foundation4 where each provider reads model files. The API server and worker processes read paths.fastembed directly, use paths.huggingface for the tokenizers of BM25 indexes, and send paths.gpt4all, paths.huggingface and paths.nltk to the gRPC service with each request.
| Key | Chart value | Provider | Content |
|---|---|---|---|
paths.fastembed | /data/fastembed | FastEmbedEmbeddings | ONNX models in the Hugging Face cache layout |
paths.gpt4all | /data/gpt4all | GPT4AllEmbeddings | GGUF model files |
paths.huggingface | /cache/huggingface, or /data/huggingface when a model image is mounted there | The Hugging Face tokenizers of BM25 indexes, and, with the huggingface model image at /data/huggingface, HuggingFaceEmbeddings and SentenceTransformersTokenTextSplitter | Sentence Transformers models and tokenizers in the Hugging Face cache layout |
paths.nltk | /data/nltk | NLTKTextSplitter | NLTK data, such as punkt_tab |
The mount value of a model image equals the paths value of the provider that reads the files. The chart values match the mount paths that this page uses for the model images, so a deployment changes a path only for a different directory layout. Configuration reference describes how a configs file sets these keys.
In a chart installation, HuggingFaceEmbeddings and SentenceTransformersTokenTextSplitter read their models from the huggingface model image mounted at /data/huggingface. With that image mounted, the chart sets no writable Hugging Face cache, so the image also holds the tokenizer of every BM25 index that uses a Hugging Face tokenizer.
The gRPC service reads two further directories that no paths key names:
/data/tiktoken. The tiktoken encodings ofTokenTextSplitter, from thetiktokenmodel image. The variableTIKTOKEN_CACHE_DIR, set throughapi-server.grpcExtraEnv, names another directory./data/spacy/<pipeline>. A named spaCy pipeline ofSpacyTextSplitter. The default pipelinesentencizerneeds no files.
Default model shadowing
A model image mounted at /data/fastembed replaces the directory of the API server image. The FastEmbed files in the API server image are then hidden in every container that mounts the model image.
The seeded embedding model all-MiniLM-L6-v2 loads Qdrant/all-MiniLM-L6-v2-onnx. With a FastEmbed model image that lacks those files, each process downloads the seeded model into the writable cache /cache/fastembed on first use, which fails without internet access and leaves every pipeline that uses the seeded model unable to embed text. A FastEmbed model image therefore contains every FastEmbed model that the deployment uses, including the seeded model.
The API server image holds the following entries in /data/fastembed:
| Entry | Model |
|---|---|
models--qdrant--all-MiniLM-L6-v2-onnx | Qdrant/all-MiniLM-L6-v2-onnx, the seeded model |
models--Qdrant--all-MiniLM-L6-v2-onnx | A link to the previous entry, for the capitalized name |
models--Xenova--bge-small-en-v1.5 | Files of Xenova/bge-small-en-v1.5, which the built-in name BAAI/bge-small-en-v1.5 loads |
models--nomic-ai--nomic-embed-text-v1.5 | nomic-ai/nomic-embed-text-v1.5 |
models--sentence-transformers--all-MiniLM-L6-v2 | sentence-transformers/all-MiniLM-L6-v2, which loads as a custom model with mean pooling from onnx/model.onnx |
The following command lists the directory of a running API server:
kubectl -n foundation4ai exec deploy/foundation4ai-api-server -c server -- ls -la /data/fastembed
FastEmbed models
FastEmbed accepts the following kinds of models:
- Built-in models. Three of the ten built-in models, the seeded model,
BAAI/bge-small-en-v1.5andnomic-ai/nomic-embed-text-v1.5, load from the files in the API server image. FastEmbed downloads the files of the other seven from Hugging Face the first time each process loads the model. When the FastEmbed directory is read-only, such as a model image, the download goes to the writable cache/cache/fastembedof the pod, and each pod downloads again after a restart. Those models fail in an air-gapped deployment unless a FastEmbed model image holds their files. - Known Sentence Transformers names.
sentence-transformers/all-MiniLM-L6-v2andsentence-transformers/all-MiniLM-L12-v2, created without parameters, load as custom models with mean pooling fromonnx/model.onnx. The API server image holds the files ofsentence-transformers/all-MiniLM-L6-v2. FastEmbed never downloads these two models, and no Foundation4 image holds the files ofsentence-transformers/all-MiniLM-L12-v2, so that model loads only from files that a FastEmbed model image adds. - Custom ONNX models. FastEmbed reads a custom model only from local files in the FastEmbed model directory or in the writable cache, and never downloads the model. A custom model is the supported way to add a FastEmbed model to an air-gapped deployment.
A custom model occupies one directory in the Hugging Face cache layout, where the model name has each / replaced by --:
<paths.fastembed>/models--<model>/snapshots/<snapshot>/
The snapshot directory holds the ONNX file named by the onnx parameter and the files tokenizer.json, config.json, special_tokens_map.json and tokenizer_config.json. The snapshots directory holds exactly one snapshot directory. The loader reads only the ONNX file, so a model that stores weights in separate external data files cannot be loaded, and a data_files parameter that names a file returns HTTP 400. The model is exported as a single ONNX file. Providers and models lists the parameters.
Custom FastEmbed model
The following procedure builds a FastEmbed model image that contains the files of the API server image and one custom model, example-org/example-embed-onnx. The build runs on a host with Docker and access to the registry that the cluster uses.
-
Place the model files in the build directory and list the files:
find models--example-org--example-embed-onnx -type fExpected result: five files under
models--example-org--example-embed-onnx/snapshots/<snapshot>/: the ONNX file, such asonnx/model.onnx, and the four JSON files. -
Create a file named
Dockerfilein the build directory. The first stage reads the FastEmbed directory of the API server image, so the new image keeps the seeded model and the other included models:FROM <registry>/foundation4ai-api:<api server image tag> AS apiFROM scratchCOPY --from=api /data/fastembed/ /COPY models--example-org--example-embed-onnx/ /models--example-org--example-embed-onnx/Expected result: the build directory holds
Dockerfileand the model directory. -
Build and push the image:
docker build -t <registry>/fastembed-models:1.0.0 .docker push <registry>/fastembed-models:1.0.0Expected result: the push ends with a line that reports the image digest.
-
Add the image to the values file of the application release:
api-server:models:- name: fastembedimage:reference: <registry>/fastembed-models:1.0.0pullPolicy: IfNotPresentmount: /data/fastembedExpected result:
modelslists the entry once, next to any other model images of the deployment. -
Upgrade the application release:
helm upgrade --install foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result: Helm reports
STATUS: deployed. -
Check the pods:
kubectl -n foundation4ai get podsExpected result: new API server and worker pods show
2/2containers ready andRunning. -
Compare the FastEmbed directory in the
serverandworkercontainers:kubectl -n foundation4ai exec deploy/foundation4ai-api-server -c server -- ls /data/fastembedkubectl -n foundation4ai exec deploy/foundation4ai-api-server-worker -c worker -- ls /data/fastembedExpected result: both listings show the five entries of the API server image and
models--example-org--example-embed-onnx. A listing without these entries points to the image layout: an image that keeps the files in a directory needssubPath, as described in Container mounts. -
Create the embedding model. The shell variables are described in Authenticate:
curl -X POST "$FOUNDATION4_URL/embedding-models" \-H "x-api-key: $FOUNDATION4_API_KEY" \-H "x-api-key-secret: $FOUNDATION4_API_SECRET" \-H "Content-Type: application/json" \-d '{"name": "example-embed","description": "Custom ONNX model from the FastEmbed model image","provider": "FastEmbedEmbeddings","model": "example-org/example-embed-onnx","size": 0,"parameters": {"onnx": "onnx/model.onnx", "pooling": "mean"}}'Expected result: status 201 and a body whose
sizeis the dimension of the model. A missing file returns HTTP 400 withError reading <file>indetails.error. -
Check that the worker loads every configured model without network access:
kubectl -n foundation4ai exec deploy/foundation4ai-api-server-worker -c worker -- ./foundation4ai data check-modelsExpected result: a line that starts with
- example-embed (FastEmbedEmbeddings, example-org/example-embed-onnx): ok, loads offline from. When every other configured model also loads, the last line isEvery configured model loads.Command-line interface describes the output and the failures.
Providers in the gRPC service
The gRPC service loads Python packages from /packages at startup. For each directory in /packages, the service adds /packages/<name>/lib to the Python path when that directory exists, and /packages/<name> otherwise. A custom package image therefore contains the package directories of a Python 3.13 site-packages directory at the image root, built for the processor architecture of the cluster nodes.
| Provider or splitter | Package image | Model image and mount |
|---|---|---|
GPT4AllEmbeddings | gpt4all | gpt4all, at /data/gpt4all |
HuggingFaceEmbeddings | sentence-transformers | huggingface, at /data/huggingface |
SentenceTransformersTokenTextSplitter | sentence-transformers | huggingface, at /data/huggingface, which holds the default model sentence-transformers/all-mpnet-base-v2 |
NLTKTextSplitter | None: the gRPC service image includes the nltk package | nltk, at /data/nltk |
TokenTextSplitter | None: the gRPC service image includes the tiktoken package | tiktoken, at /data/tiktoken |
SpacyTextSplitter | spacy | None for the default sentencizer pipeline |
The seeded NLTKTextSplitter text splitter exists in every installation and splits text once the nltk model image provides the punkt_tab data.
GPT4All provider
-
Add the package and model images to the values file of the application release:
api-server:packages:- name: gpt4allimage:reference: <registry>/<gpt4all package image>:<tag>pullPolicy: IfNotPresentmodels:- name: gpt4allimage:reference: <registry>/<gpt4all model image>:<tag>pullPolicy: IfNotPresentmount: /data/gpt4allExpected result:
packagesandmodelseach list onegpt4allentry. -
Upgrade the application release:
helm upgrade --install foundation4ai ./charts/foundation4ai \-n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10mExpected result: Helm reports
STATUS: deployed, and the new API server and worker pods show2/2containers ready. -
Confirm that the gRPC service loaded the package:
kubectl -n foundation4ai logs deploy/foundation4ai-api-server -c grpc | grep "Added package directory"Expected result:
INFO:processor:Added package directory to site: /packages/gpt4allor the same line ending in/packages/gpt4all/lib. -
Confirm that the model files are present:
kubectl -n foundation4ai exec deploy/foundation4ai-api-server -c grpc -- ls /data/gpt4allExpected result:
all-MiniLM-L6-v2.gguf2.f16.ggufandnomic-embed-text-v1.5.f16.gguf.
The gRPC service loads GPT4All models with downloads turned off, so a model file that the model image lacks fails instead of downloading. The Hugging Face provider follows the same steps with the sentence-transformers package image and the huggingface model image at /data/huggingface. The variable HF_HUB_OFFLINE with the value 1 in api-server.grpcExtraEnv keeps the Hugging Face provider from contacting the Hugging Face Hub, so a model that the image lacks returns HTTP 503 instead of downloading. foundation4ai data check-models confirms that every configured model loads, as described in Command-line interface.
Offline capabilities
The following table states what works without access to the internet. The conditions match Providers and models.
| Capability | Without internet access | Condition or reason |
|---|---|---|
| FastEmbed, the models in the API server image | Yes | Files present in the API server image, or in the FastEmbed model image that replaces the directory |
| FastEmbed, the other built-in models | No, unless a FastEmbed model image holds the files | FastEmbed downloads missing files from Hugging Face on first use in each pod |
| FastEmbed, custom ONNX models | Yes | Files in a FastEmbed model image |
| OpenAI-compatible embeddings | Yes | endpoint names an embeddings server inside the network |
| Hugging Face Sentence Transformers | Yes, with the images | sentence-transformers package image, the model in the huggingface model image at /data/huggingface, and HF_HUB_OFFLINE=1 in api-server.grpcExtraEnv |
| GPT4All | Yes, with the images | gpt4all package image and the model file in the gpt4all model image; the gRPC service turns off GPT4All downloads |
| Hugging Face inference endpoint | Only with an internal endpoint | The endpoint must be reachable from the gRPC service |
| Recursive character, character, code and Markdown header splitters | Yes | None |
| Token splitter | Yes, with the tiktoken model image | Encoding files in /data/tiktoken, or in the directory that TIKTOKEN_CACHE_DIR names; otherwise tiktoken downloads the files |
| NLTK splitter | Yes, with the nltk model image | The gRPC service image includes the nltk package, and the punkt_tab data comes from the model image |
| spaCy splitter | Yes, with the spacy package image | The default sentencizer pipeline needs no files; a named pipeline is read from /data/spacy/<pipeline> and never downloaded |
| Sentence Transformers token splitter | Yes, with the images | As for Hugging Face Sentence Transformers |
| BM25 index, built-in tokenizer models | Yes | The four models are part of the PostgreSQL tokenizer extension |
| BM25 index, Hugging Face tokenizer | No | The API server downloads tokenizer.json from huggingface.co when the index is created, as described in BM25 tokenizers |
| Agents, direct queries and the OpenAI-compatible endpoint | Yes, with an internal model server | The model server must be reachable from the API server pods |
Model check, foundation4ai data check-models | Yes | Checks that every configured embedding model and text splitter loads, without downloading FastEmbed files, as described in Command-line interface |
An air-gapped deployment therefore works provided its pipelines use offline-capable embedding models and text splitters.
Language models in an air-gapped deployment
Foundation4 includes no language model and has no default model server. An administrator registers a model server inside the network with POST /llms, as described in LLMs. The API server calls the model server, so the network allows traffic from the API server pods to the model server.
Agents, direct queries and the OpenAI-compatible endpoint call the chat completions API of the model server. A registration without an API key serves a model server that does not check keys. Model servers on private addresses are accepted unless the configuration key app.allow_private_endpoints is set to false.
Network access in an air-gapped deployment
With the capabilities marked Yes in Offline capabilities, Foundation4 components connect only to services inside the network:
| Component | Destinations |
|---|---|
| API server | PostgreSQL, the Redis-compatible cache, NATS JetStream, the gRPC service in the same pod, the Prometheus server, registered model servers and embedding servers, and an OpenTelemetry collector when configured |
| Worker | PostgreSQL, the Redis-compatible cache, NATS JetStream, the gRPC service in the same pod, embedding servers, and an OpenTelemetry collector when configured |
| gRPC service | A Hugging Face inference endpoint, when a model uses one |
| Dashboard | None; the browser calls the API server |
A model server or embedding server that uses TLS presents a certificate that the API server image trusts. The charts provide no value that adds a certificate authority to the images, so a server certificate from a private certificate authority is an open point to settle with the Foundation4 provider before installation.
Images to mirror
An air-gapped installation copies every image that the releases use into a registry inside the network and points each values key at the copy. The chart directories include the archives of the PostgreSQL, Valkey, NATS and Prometheus subcharts, so the installation reads no Helm repository.
| Role | Values key | Image |
|---|---|---|
| API server, workers and hook Jobs | api-server.image.repository or global.foundation4aiImageRegistry, with global.serverImageTag | Supplied by the Foundation4 provider |
| gRPC service | api-server.image.repositoryGrpc or global.foundation4aiImageRegistry, with global.grpcImageTag | Supplied by the Foundation4 provider |
| Dashboard | dashboard.image.repository or global.foundation4aiImageRegistry, with global.dashboardImageTag | Supplied by the Foundation4 provider |
| Model images | api-server.models[].image.reference | fastembed, gpt4all, huggingface, nltk and tiktoken, supplied by the Foundation4 provider, or built as in Custom FastEmbed model |
| Package images | api-server.packages[].image.reference | gpt4all, sentence-transformers and spacy, supplied by the Foundation4 provider |
| NATS server | nats.container.image | nats, tag 2.12.4-alpine |
| NATS configuration reloader | nats.reloader.image | natsio/nats-server-config-reloader, tag 0.21.1 |
| NATS utility container | nats.natsBox.container.image | natsio/nats-box, tag 0.19.3 |
| Redis-compatible cache (Valkey) | redis.image | valkey/valkey, the chart default tag 9.0.1 |
| Prometheus server | prometheus.server.image | prometheus/prometheus, the chart default tag v3.9.1 |
| Prometheus configuration reloader | prometheus.configmapReload.prometheus.image | prometheus-operator/prometheus-config-reloader, tag v0.89.0 |
| PostgreSQL, evaluation profile only | postgres.image | tensorchord/vchord-suite, tag pg18-20260501 |
The dependency checks of the deployment files (just check) start three further utility images: natsio/nats-box:latest, redis:alpine and curlimages/curl. An air-gapped installation mirrors these images too, or runs the equivalent checks in Pre-installation checks. Operator tools, such as kubectl, Helm and a container runtime for the mirroring host, are also brought into the network.
Image list from a staging installation
A staging installation with internet access gives the exact list of images, including tags that the charts derive from the chart versions:
-
List every container image and image volume reference in the namespace:
kubectl get pods -n foundation4ai -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{range .spec.volumes[*]}{.image.reference}{"\n"}{end}{end}' \| grep -v '^$' | sort -u > images.txtcat images.txtExpected result: one line per image, including the NATS, Valkey, Prometheus and Foundation4 images and every model and package image.
-
Pull the images and write one archive:
xargs -n 1 docker pull < images.txtdocker save -o foundation4ai-images.tar $(cat images.txt)Expected result:
foundation4ai-images.tarexists anddocker savereports no error. -
Inside the network, load the archive, then tag and push each image to the internal registry:
docker load -i foundation4ai-images.tardocker tag <source image>:<tag> <registry>/<repository>:<tag>docker push <registry>/<repository>:<tag>Expected result:
docker loadreportsLoaded imagefor each image, and each push reports a digest.
Registry values
The following values point every chart at an internal registry that keeps the source repository paths. global.foundation4aiImageRegistry replaces the registry host of the API server, gRPC service and dashboard images. The NATS chart prefixes global.image.registry to each NATS image, and the Valkey chart prefixes global.imageRegistry.
global:
serverImageTag: "<api server image tag>"
grpcImageTag: "<grpc service image tag>"
dashboardImageTag: "<dashboard image tag>"
foundation4aiImageRegistry: <registry>
imagePullSecrets:
- <pull secret>
image:
registry: <registry>
pullSecretNames:
- <pull secret>
imageRegistry: <registry>
prometheus:
imagePullSecrets:
- name: <pull secret>
server:
image:
repository: <registry>/prometheus/prometheus
configmapReload:
prometheus:
image:
repository: <registry>/prometheus-operator/prometheus-config-reloader
postgres:
image:
registry: <registry>
imagePullSecrets:
- name: <pull secret>
The charts read pull secrets in three shapes. The Foundation4 charts read plain names and name entries. Prometheus and PostgreSQL read a list of name entries. NATS reads global.image.pullSecretNames, and Valkey reads global.imagePullSecrets, both as lists of names. A list of plain names in global.imagePullSecrets therefore serves the Foundation4 charts and Valkey, as described in Charts, images and installation bundle. The helm test pod of Valkey takes no pull secrets, as described in Values reference. A registry inside the network that requires no pull secret needs none of the pull secret keys.