Skip to main content

Models, packages and air-gapped installs

This page describes how Foundation4 delivers embedding models and Python packages, how an operator adds model and package images to a deployment, and which images an air-gapped installation mirrors. The page also states which embedding providers, text splitters and language model features work without network access. Operators who plan an air-gapped installation or add embedding models need this page. Providers and models lists the parameters and network access of every provider.

Model and package delivery​

Foundation4 runs the API server and the workers from the same image. Each API server pod and each worker pod also runs the gRPC service as a sidecar container, a second container in the same Kubernetes pod, which hosts the Python embedding providers and every text splitter.

SourceContentConsumers
API server imageThe Foundation4 binary and the FastEmbed model files listed in Default model shadowing, in /data/fastembedAPI server and worker processes
gRPC service imageThe Python service, the LangChain libraries and the NLTK and tiktoken packages, without the GPT4All, Sentence Transformers and spaCy packagesgRPC service
Package imageOne Python package with dependencies: gpt4all, sentence-transformers or spacygRPC service, mounted at /packages/<name>
Model imageModel files for one provider, such as GPT4All GGUF files, NLTK data or tiktoken encodingsEvery container of the API server and worker pods, mounted at the path in mount

Model and package images use the Kubernetes image volume source: the kubelet pulls an OCI image and mounts the image content read-only, without running the image. The cluster therefore needs image volume support in the API server, the kubelet and the container runtime. The application chart requires Kubernetes 1.31 or later, and the operator confirms that the image volume feature is enabled before adding a model or package image. Deployment overview and requirements lists the cluster requirements. An installation that uses only the models in the API server image, such as the evaluation profile in Install for evaluation, needs no image volumes.

Values for model and package images​

Model and package images are declared under api-server in the values file of the application release:

api-server:
packages:
- name: gpt4all
image:
reference: <registry>/<package image repository>:<tag>
pullPolicy: IfNotPresent
models:
- name: gpt4all
image:
reference: <registry>/<model image repository>:<tag>
pullPolicy: IfNotPresent
mount: /data/gpt4all
KeyEffect
packages[].nameNames the volume and the mount directory /packages/<name> in both gRPC service containers
packages[].imageCopied unchanged into the image volume source: reference and pullPolicy
packages[].imagePullSecretsPlain names or name entries, added to every pod of the api-server chart
models[].nameNames the volume
models[].imageCopied unchanged into the image volume source: reference and pullPolicy
models[].mountThe mount path in every container of the API server and worker pods
models[].subPathOptional. A directory of the image to mount in place of the image root, for an image that keeps the files in a directory
models[].imagePullSecretsPlain names or name entries, added to every pod of the api-server chart

Every pod of the api-server chart, including the worker pods and the hook Jobs, receives global.imagePullSecrets, api-server.imagePullSecrets and the pull secrets of every model and package image. Values reference describes every key of the charts.

A change to packages or models changes the pod templates, so helm upgrade replaces the API server and worker pods without a manual restart.

Container mounts​

Every container of the API server and worker pods mounts each model image the same way, at mount, so a model has the same path in the server, worker and grpc containers:

ContainerPodPackage imagesModel imagesWritable caches
serverAPI serverNot mountedThe image at mount/cache/fastembed, /cache/huggingface
grpcAPI server/packages/<name>The image at mount/cache/huggingface
workerWorkerNot mountedThe image at mount/cache/fastembed, /cache/huggingface
grpcWorker/packages/<name>The image at mount/cache/huggingface

The model images keep the model files at the image root. An image that keeps the files in a directory, such as a model image of an earlier build with the files under data, takes subPath: data in its models entry. Subpath mounts of image volumes need a Kubernetes version that supports subpaths on image volumes.

The writable caches are temporary volumes of each pod, and their content is lost when the pod restarts. The chart omits /cache/huggingface when api-server.huggingfaceCache.enabled is false or when a model image is mounted at /data/huggingface.

Model directories​

The paths configuration keys tell Foundation4 where each provider reads model files. The API server and worker processes read paths.fastembed directly, use paths.huggingface for the tokenizers of BM25 indexes, and send paths.gpt4all, paths.huggingface and paths.nltk to the gRPC service with each request.

KeyChart valueProviderContent
paths.fastembed/data/fastembedFastEmbedEmbeddingsONNX models in the Hugging Face cache layout
paths.gpt4all/data/gpt4allGPT4AllEmbeddingsGGUF model files
paths.huggingface/cache/huggingface, or /data/huggingface when a model image is mounted thereThe Hugging Face tokenizers of BM25 indexes, and, with the huggingface model image at /data/huggingface, HuggingFaceEmbeddings and SentenceTransformersTokenTextSplitterSentence Transformers models and tokenizers in the Hugging Face cache layout
paths.nltk/data/nltkNLTKTextSplitterNLTK data, such as punkt_tab

The mount value of a model image equals the paths value of the provider that reads the files. The chart values match the mount paths that this page uses for the model images, so a deployment changes a path only for a different directory layout. Configuration reference describes how a configs file sets these keys.

In a chart installation, HuggingFaceEmbeddings and SentenceTransformersTokenTextSplitter read their models from the huggingface model image mounted at /data/huggingface. With that image mounted, the chart sets no writable Hugging Face cache, so the image also holds the tokenizer of every BM25 index that uses a Hugging Face tokenizer.

The gRPC service reads two further directories that no paths key names:

  • /data/tiktoken. The tiktoken encodings of TokenTextSplitter, from the tiktoken model image. The variable TIKTOKEN_CACHE_DIR, set through api-server.grpcExtraEnv, names another directory.
  • /data/spacy/<pipeline>. A named spaCy pipeline of SpacyTextSplitter. The default pipeline sentencizer needs no files.

Default model shadowing​

A model image mounted at /data/fastembed replaces the directory of the API server image. The FastEmbed files in the API server image are then hidden in every container that mounts the model image.

The seeded embedding model all-MiniLM-L6-v2 loads Qdrant/all-MiniLM-L6-v2-onnx. With a FastEmbed model image that lacks those files, each process downloads the seeded model into the writable cache /cache/fastembed on first use, which fails without internet access and leaves every pipeline that uses the seeded model unable to embed text. A FastEmbed model image therefore contains every FastEmbed model that the deployment uses, including the seeded model.

The API server image holds the following entries in /data/fastembed:

EntryModel
models--qdrant--all-MiniLM-L6-v2-onnxQdrant/all-MiniLM-L6-v2-onnx, the seeded model
models--Qdrant--all-MiniLM-L6-v2-onnxA link to the previous entry, for the capitalized name
models--Xenova--bge-small-en-v1.5Files of Xenova/bge-small-en-v1.5, which the built-in name BAAI/bge-small-en-v1.5 loads
models--nomic-ai--nomic-embed-text-v1.5nomic-ai/nomic-embed-text-v1.5
models--sentence-transformers--all-MiniLM-L6-v2sentence-transformers/all-MiniLM-L6-v2, which loads as a custom model with mean pooling from onnx/model.onnx

The following command lists the directory of a running API server:

kubectl -n foundation4ai exec deploy/foundation4ai-api-server -c server -- ls -la /data/fastembed

FastEmbed models​

FastEmbed accepts the following kinds of models:

  • Built-in models. Three of the ten built-in models, the seeded model, BAAI/bge-small-en-v1.5 and nomic-ai/nomic-embed-text-v1.5, load from the files in the API server image. FastEmbed downloads the files of the other seven from Hugging Face the first time each process loads the model. When the FastEmbed directory is read-only, such as a model image, the download goes to the writable cache /cache/fastembed of the pod, and each pod downloads again after a restart. Those models fail in an air-gapped deployment unless a FastEmbed model image holds their files.
  • Known Sentence Transformers names. sentence-transformers/all-MiniLM-L6-v2 and sentence-transformers/all-MiniLM-L12-v2, created without parameters, load as custom models with mean pooling from onnx/model.onnx. The API server image holds the files of sentence-transformers/all-MiniLM-L6-v2. FastEmbed never downloads these two models, and no Foundation4 image holds the files of sentence-transformers/all-MiniLM-L12-v2, so that model loads only from files that a FastEmbed model image adds.
  • Custom ONNX models. FastEmbed reads a custom model only from local files in the FastEmbed model directory or in the writable cache, and never downloads the model. A custom model is the supported way to add a FastEmbed model to an air-gapped deployment.

A custom model occupies one directory in the Hugging Face cache layout, where the model name has each / replaced by --:

<paths.fastembed>/models--<model>/snapshots/<snapshot>/

The snapshot directory holds the ONNX file named by the onnx parameter and the files tokenizer.json, config.json, special_tokens_map.json and tokenizer_config.json. The snapshots directory holds exactly one snapshot directory. The loader reads only the ONNX file, so a model that stores weights in separate external data files cannot be loaded, and a data_files parameter that names a file returns HTTP 400. The model is exported as a single ONNX file. Providers and models lists the parameters.

Custom FastEmbed model​

The following procedure builds a FastEmbed model image that contains the files of the API server image and one custom model, example-org/example-embed-onnx. The build runs on a host with Docker and access to the registry that the cluster uses.

  1. Place the model files in the build directory and list the files:

    find models--example-org--example-embed-onnx -type f

    Expected result: five files under models--example-org--example-embed-onnx/snapshots/<snapshot>/: the ONNX file, such as onnx/model.onnx, and the four JSON files.

  2. Create a file named Dockerfile in the build directory. The first stage reads the FastEmbed directory of the API server image, so the new image keeps the seeded model and the other included models:

    FROM <registry>/foundation4ai-api:<api server image tag> AS api

    FROM scratch
    COPY --from=api /data/fastembed/ /
    COPY models--example-org--example-embed-onnx/ /models--example-org--example-embed-onnx/

    Expected result: the build directory holds Dockerfile and the model directory.

  3. Build and push the image:

    docker build -t <registry>/fastembed-models:1.0.0 .
    docker push <registry>/fastembed-models:1.0.0

    Expected result: the push ends with a line that reports the image digest.

  4. Add the image to the values file of the application release:

    api-server:
    models:
    - name: fastembed
    image:
    reference: <registry>/fastembed-models:1.0.0
    pullPolicy: IfNotPresent
    mount: /data/fastembed

    Expected result: models lists the entry once, next to any other model images of the deployment.

  5. Upgrade the application release:

    helm upgrade --install foundation4ai ./charts/foundation4ai \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

    Expected result: Helm reports STATUS: deployed.

  6. Check the pods:

    kubectl -n foundation4ai get pods

    Expected result: new API server and worker pods show 2/2 containers ready and Running.

  7. Compare the FastEmbed directory in the server and worker containers:

    kubectl -n foundation4ai exec deploy/foundation4ai-api-server -c server -- ls /data/fastembed
    kubectl -n foundation4ai exec deploy/foundation4ai-api-server-worker -c worker -- ls /data/fastembed

    Expected result: both listings show the five entries of the API server image and models--example-org--example-embed-onnx. A listing without these entries points to the image layout: an image that keeps the files in a directory needs subPath, as described in Container mounts.

  8. Create the embedding model. The shell variables are described in Authenticate:

    curl -X POST "$FOUNDATION4_URL/embedding-models" \
    -H "x-api-key: $FOUNDATION4_API_KEY" \
    -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
    -H "Content-Type: application/json" \
    -d '{
    "name": "example-embed",
    "description": "Custom ONNX model from the FastEmbed model image",
    "provider": "FastEmbedEmbeddings",
    "model": "example-org/example-embed-onnx",
    "size": 0,
    "parameters": {"onnx": "onnx/model.onnx", "pooling": "mean"}
    }'

    Expected result: status 201 and a body whose size is the dimension of the model. A missing file returns HTTP 400 with Error reading <file> in details.error.

  9. Check that the worker loads every configured model without network access:

    kubectl -n foundation4ai exec deploy/foundation4ai-api-server-worker -c worker -- ./foundation4ai data check-models

    Expected result: a line that starts with - example-embed (FastEmbedEmbeddings, example-org/example-embed-onnx): ok, loads offline from. When every other configured model also loads, the last line is Every configured model loads. Command-line interface describes the output and the failures.

Providers in the gRPC service​

The gRPC service loads Python packages from /packages at startup. For each directory in /packages, the service adds /packages/<name>/lib to the Python path when that directory exists, and /packages/<name> otherwise. A custom package image therefore contains the package directories of a Python 3.13 site-packages directory at the image root, built for the processor architecture of the cluster nodes.

Provider or splitterPackage imageModel image and mount
GPT4AllEmbeddingsgpt4allgpt4all, at /data/gpt4all
HuggingFaceEmbeddingssentence-transformershuggingface, at /data/huggingface
SentenceTransformersTokenTextSplittersentence-transformershuggingface, at /data/huggingface, which holds the default model sentence-transformers/all-mpnet-base-v2
NLTKTextSplitterNone: the gRPC service image includes the nltk packagenltk, at /data/nltk
TokenTextSplitterNone: the gRPC service image includes the tiktoken packagetiktoken, at /data/tiktoken
SpacyTextSplitterspacyNone for the default sentencizer pipeline

The seeded NLTKTextSplitter text splitter exists in every installation and splits text once the nltk model image provides the punkt_tab data.

GPT4All provider​

  1. Add the package and model images to the values file of the application release:

    api-server:
    packages:
    - name: gpt4all
    image:
    reference: <registry>/<gpt4all package image>:<tag>
    pullPolicy: IfNotPresent
    models:
    - name: gpt4all
    image:
    reference: <registry>/<gpt4all model image>:<tag>
    pullPolicy: IfNotPresent
    mount: /data/gpt4all

    Expected result: packages and models each list one gpt4all entry.

  2. Upgrade the application release:

    helm upgrade --install foundation4ai ./charts/foundation4ai \
    -n foundation4ai -f foundation4ai.values.yaml --wait --timeout 10m

    Expected result: Helm reports STATUS: deployed, and the new API server and worker pods show 2/2 containers ready.

  3. Confirm that the gRPC service loaded the package:

    kubectl -n foundation4ai logs deploy/foundation4ai-api-server -c grpc | grep "Added package directory"

    Expected result: INFO:processor:Added package directory to site: /packages/gpt4all or the same line ending in /packages/gpt4all/lib.

  4. Confirm that the model files are present:

    kubectl -n foundation4ai exec deploy/foundation4ai-api-server -c grpc -- ls /data/gpt4all

    Expected result: all-MiniLM-L6-v2.gguf2.f16.gguf and nomic-embed-text-v1.5.f16.gguf.

The gRPC service loads GPT4All models with downloads turned off, so a model file that the model image lacks fails instead of downloading. The Hugging Face provider follows the same steps with the sentence-transformers package image and the huggingface model image at /data/huggingface. The variable HF_HUB_OFFLINE with the value 1 in api-server.grpcExtraEnv keeps the Hugging Face provider from contacting the Hugging Face Hub, so a model that the image lacks returns HTTP 503 instead of downloading. foundation4ai data check-models confirms that every configured model loads, as described in Command-line interface.

Offline capabilities​

The following table states what works without access to the internet. The conditions match Providers and models.

CapabilityWithout internet accessCondition or reason
FastEmbed, the models in the API server imageYesFiles present in the API server image, or in the FastEmbed model image that replaces the directory
FastEmbed, the other built-in modelsNo, unless a FastEmbed model image holds the filesFastEmbed downloads missing files from Hugging Face on first use in each pod
FastEmbed, custom ONNX modelsYesFiles in a FastEmbed model image
OpenAI-compatible embeddingsYesendpoint names an embeddings server inside the network
Hugging Face Sentence TransformersYes, with the imagessentence-transformers package image, the model in the huggingface model image at /data/huggingface, and HF_HUB_OFFLINE=1 in api-server.grpcExtraEnv
GPT4AllYes, with the imagesgpt4all package image and the model file in the gpt4all model image; the gRPC service turns off GPT4All downloads
Hugging Face inference endpointOnly with an internal endpointThe endpoint must be reachable from the gRPC service
Recursive character, character, code and Markdown header splittersYesNone
Token splitterYes, with the tiktoken model imageEncoding files in /data/tiktoken, or in the directory that TIKTOKEN_CACHE_DIR names; otherwise tiktoken downloads the files
NLTK splitterYes, with the nltk model imageThe gRPC service image includes the nltk package, and the punkt_tab data comes from the model image
spaCy splitterYes, with the spacy package imageThe default sentencizer pipeline needs no files; a named pipeline is read from /data/spacy/<pipeline> and never downloaded
Sentence Transformers token splitterYes, with the imagesAs for Hugging Face Sentence Transformers
BM25 index, built-in tokenizer modelsYesThe four models are part of the PostgreSQL tokenizer extension
BM25 index, Hugging Face tokenizerNoThe API server downloads tokenizer.json from huggingface.co when the index is created, as described in BM25 tokenizers
Agents, direct queries and the OpenAI-compatible endpointYes, with an internal model serverThe model server must be reachable from the API server pods
Model check, foundation4ai data check-modelsYesChecks that every configured embedding model and text splitter loads, without downloading FastEmbed files, as described in Command-line interface

An air-gapped deployment therefore works provided its pipelines use offline-capable embedding models and text splitters.

Language models in an air-gapped deployment​

Foundation4 includes no language model and has no default model server. An administrator registers a model server inside the network with POST /llms, as described in LLMs. The API server calls the model server, so the network allows traffic from the API server pods to the model server.

Agents, direct queries and the OpenAI-compatible endpoint call the chat completions API of the model server. A registration without an API key serves a model server that does not check keys. Model servers on private addresses are accepted unless the configuration key app.allow_private_endpoints is set to false.

Network access in an air-gapped deployment​

With the capabilities marked Yes in Offline capabilities, Foundation4 components connect only to services inside the network:

ComponentDestinations
API serverPostgreSQL, the Redis-compatible cache, NATS JetStream, the gRPC service in the same pod, the Prometheus server, registered model servers and embedding servers, and an OpenTelemetry collector when configured
WorkerPostgreSQL, the Redis-compatible cache, NATS JetStream, the gRPC service in the same pod, embedding servers, and an OpenTelemetry collector when configured
gRPC serviceA Hugging Face inference endpoint, when a model uses one
DashboardNone; the browser calls the API server

A model server or embedding server that uses TLS presents a certificate that the API server image trusts. The charts provide no value that adds a certificate authority to the images, so a server certificate from a private certificate authority is an open point to settle with the Foundation4 provider before installation.

Images to mirror​

An air-gapped installation copies every image that the releases use into a registry inside the network and points each values key at the copy. The chart directories include the archives of the PostgreSQL, Valkey, NATS and Prometheus subcharts, so the installation reads no Helm repository.

RoleValues keyImage
API server, workers and hook Jobsapi-server.image.repository or global.foundation4aiImageRegistry, with global.serverImageTagSupplied by the Foundation4 provider
gRPC serviceapi-server.image.repositoryGrpc or global.foundation4aiImageRegistry, with global.grpcImageTagSupplied by the Foundation4 provider
Dashboarddashboard.image.repository or global.foundation4aiImageRegistry, with global.dashboardImageTagSupplied by the Foundation4 provider
Model imagesapi-server.models[].image.referencefastembed, gpt4all, huggingface, nltk and tiktoken, supplied by the Foundation4 provider, or built as in Custom FastEmbed model
Package imagesapi-server.packages[].image.referencegpt4all, sentence-transformers and spacy, supplied by the Foundation4 provider
NATS servernats.container.imagenats, tag 2.12.4-alpine
NATS configuration reloadernats.reloader.imagenatsio/nats-server-config-reloader, tag 0.21.1
NATS utility containernats.natsBox.container.imagenatsio/nats-box, tag 0.19.3
Redis-compatible cache (Valkey)redis.imagevalkey/valkey, the chart default tag 9.0.1
Prometheus serverprometheus.server.imageprometheus/prometheus, the chart default tag v3.9.1
Prometheus configuration reloaderprometheus.configmapReload.prometheus.imageprometheus-operator/prometheus-config-reloader, tag v0.89.0
PostgreSQL, evaluation profile onlypostgres.imagetensorchord/vchord-suite, tag pg18-20260501

The dependency checks of the deployment files (just check) start three further utility images: natsio/nats-box:latest, redis:alpine and curlimages/curl. An air-gapped installation mirrors these images too, or runs the equivalent checks in Pre-installation checks. Operator tools, such as kubectl, Helm and a container runtime for the mirroring host, are also brought into the network.

Image list from a staging installation​

A staging installation with internet access gives the exact list of images, including tags that the charts derive from the chart versions:

  1. List every container image and image volume reference in the namespace:

    kubectl get pods -n foundation4ai -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{range .spec.volumes[*]}{.image.reference}{"\n"}{end}{end}' \
    | grep -v '^$' | sort -u > images.txt
    cat images.txt

    Expected result: one line per image, including the NATS, Valkey, Prometheus and Foundation4 images and every model and package image.

  2. Pull the images and write one archive:

    xargs -n 1 docker pull < images.txt
    docker save -o foundation4ai-images.tar $(cat images.txt)

    Expected result: foundation4ai-images.tar exists and docker save reports no error.

  3. Inside the network, load the archive, then tag and push each image to the internal registry:

    docker load -i foundation4ai-images.tar
    docker tag <source image>:<tag> <registry>/<repository>:<tag>
    docker push <registry>/<repository>:<tag>

    Expected result: docker load reports Loaded image for each image, and each push reports a digest.

Registry values​

The following values point every chart at an internal registry that keeps the source repository paths. global.foundation4aiImageRegistry replaces the registry host of the API server, gRPC service and dashboard images. The NATS chart prefixes global.image.registry to each NATS image, and the Valkey chart prefixes global.imageRegistry.

global:
serverImageTag: "<api server image tag>"
grpcImageTag: "<grpc service image tag>"
dashboardImageTag: "<dashboard image tag>"
foundation4aiImageRegistry: <registry>
imagePullSecrets:
- <pull secret>
image:
registry: <registry>
pullSecretNames:
- <pull secret>
imageRegistry: <registry>

prometheus:
imagePullSecrets:
- name: <pull secret>
server:
image:
repository: <registry>/prometheus/prometheus
configmapReload:
prometheus:
image:
repository: <registry>/prometheus-operator/prometheus-config-reloader

postgres:
image:
registry: <registry>
imagePullSecrets:
- name: <pull secret>

The charts read pull secrets in three shapes. The Foundation4 charts read plain names and name entries. Prometheus and PostgreSQL read a list of name entries. NATS reads global.image.pullSecretNames, and Valkey reads global.imagePullSecrets, both as lists of names. A list of plain names in global.imagePullSecrets therefore serves the Foundation4 charts and Valkey, as described in Charts, images and installation bundle. The helm test pod of Valkey takes no pull secrets, as described in Values reference. A registry inside the network that requires no pull secret needs none of the pull secret keys.