Skip to main content

Providers and models

This page lists every embedding provider and text splitter provider: where each provider runs, the parameters with their defaults, the model names that each provider accepts, the images that carry models and packages, and the network access that each provider needs. Administrators use this page to configure embedding models and text splitters, and operators use this page to plan an air-gapped installation. Embedding models and text splitters explains both object types and shows the create requests.

Where providers run​

Providers run in one of two places:

  • API server and workers. FastEmbedEmbeddings and OpenAIEmbeddings run inside the Foundation4 process: the API server, each worker and the command-line interface (CLI). Each process loads a model on first use and keeps the model in memory.
  • gRPC service. The other embedding providers and every text splitter run in the gRPC service, a Python service that runs as a second container in each API server pod and each worker pod. The gRPC service keeps each loaded model and splitter in memory.

Provider names are case-sensitive. A create request with any other provider name returns HTTP 404, with the message Embedding Provider not found for an embedding model and Text Splitter Provider not found for a text splitter, and the provider name in details.provider.

Embedding providers​

providerRuns inModel sourceNetwork access
FastEmbedEmbeddingsAPI server and workersONNX model files in the FastEmbed model directoryNone for models whose files are present. Other built-in models download from Hugging Face on first use
OpenAIEmbeddingsAPI server and workersA server that implements the OpenAI embeddings APIThe server named in endpoint
GPT4AllEmbeddingsgRPC serviceGGUF model files in the GPT4All model directoryThe GPT4All library can contact the GPT4All service while loading a model
HuggingFaceEmbeddingsgRPC serviceSentence Transformers model files in the Hugging Face model directoryNone for models whose files are in the Hugging Face model directory. A missing model downloads from the Hugging Face Hub, unless HF_HUB_OFFLINE=1 is set on the gRPC service
HuggingFaceEndpointEmbeddingsgRPC serviceA Hugging Face inference endpointThe inference endpoint, on every request

The model directories are set by the paths configuration of the deployment. The defaults are /data/fastembed, /data/gpt4all and /data/huggingface.

Embedding model limits​

  • Dimensions. A vector index stores vectors in a column with the dimension of the embedding model and builds a distance index on the column. The hierarchical navigable small world (HNSW) distance indexes of pgvector support up to 2,000 dimensions, so a pgvector index cannot serve a model with more dimensions. A create request for a model with more than 2,000 dimensions therefore returns HTTP 400 with Invalid parameter, whatever the index type. New vector indexes use VectorChord, as described in Search indexes.
  • Distance. Every model on this page is trained for cosine similarity, so searches use the default cosine distance strategy.
  • Text as given. Foundation4 embeds the text of queries and fragments exactly as given, with no instruction or prefix. Models whose documentation asks for prefixes, such as search_query: and search_document: for nomic-embed-text-v1.5, return vectors of lower quality without the prefixes.

FastEmbed​

FastEmbedEmbeddings runs ONNX models with the FastEmbed library. The model field selects either a built-in model or a custom model:

  • Built-in model. parameters is empty ({}), and model is one of the names in the following table.
  • Custom ONNX model. parameters is not empty, and model names a model directory, as described in Custom ONNX models.
modelDimensionsFiles in the API server image
Qdrant/all-MiniLM-L6-v2-onnx384Yes (seeded model)
BAAI/bge-small-en-v1.5384Yes
nomic-ai/nomic-embed-text-v1.5768Yes
BAAI/bge-small-zh-v1.5512No
Snowflake/snowflake-arctic-embed-xs384No
Xenova/all-MiniLM-L12-v2384No
jinaai/jina-embeddings-v2-base-en768No
mixedbread-ai/mxbai-embed-large-v11024No
intfloat/multilingual-e5-small384No
onnx-community/embeddinggemma-300m-ONNX768No

The API server image carries the files of the models marked Yes in /data/fastembed, and the workers run the same image. FastEmbed downloads the files of any other built-in model from Hugging Face the first time each process loads the model, into /data/fastembed or, when that directory is read-only, into the writable cache that the F4AI_FASTEMBED_CACHE environment variable names. The Helm charts set the cache to /cache/fastembed, a temporary volume of each pod. A process started without the variable uses /tmp/foundation4ai-fastembed. A download fails in an air-gapped deployment, and the files are lost when the pod restarts.

A model image mounted at /data/fastembed replaces the directory, so such an image includes every FastEmbed model that the deployment uses, including the seeded model. The mounted image is read-only, so a built-in model whose files the image does not include downloads into the writable cache, which requires internet access. A model that loads from neither directory returns HTTP 400 with Invalid parameter, and details.error names the model and the failure in each directory.

Two further names are accepted with empty parameters: sentence-transformers/all-MiniLM-L6-v2 and sentence-transformers/all-MiniLM-L12-v2, which Foundation4 loads as custom ONNX models with mean pooling. The API server image carries the files of sentence-transformers/all-MiniLM-L6-v2. Like any custom model, these two names load only from local files and are never downloaded, and no Foundation4 image carries the files of sentence-transformers/all-MiniLM-L12-v2, so creating that model returns HTTP 400 unless a model image adds the files. Any other built-in model name that the table does not list returns HTTP 400 with Unknown FastEmbed embedding model: <name>. intfloat/multilingual-e5-small is the only multilingual built-in model.

Custom ONNX models​

A custom model is any ONNX sentence embedding model stored in the FastEmbed model directory in the Hugging Face cache layout. FastEmbed reads custom models from local files only and never downloads them.

ParameterTypeRequiredValues
onnxStringYesThe path of the ONNX file inside the model snapshot, such as onnx/model.onnx
poolingStringNocls or mean
quantizationStringNostatic or dynamic
data_filesArray of stringsNoNot supported: Foundation4 reads only the ONNX file, so a model that stores weights in separate data files cannot be loaded. A list that names a file returns HTTP 400.

Other parameters return HTTP 400. The model files are read from the following directory, where the model value has each / replaced by --:

<paths.fastembed>/models--<model>/snapshots/<snapshot>/

The snapshot directory holds the ONNX file and the files tokenizer.json, config.json, special_tokens_map.json and tokenizer_config.json. The model directory holds exactly one snapshot. A missing file returns HTTP 400 with Error reading <file> in details.error.

OpenAI-compatible embeddings​

OpenAIEmbeddings calls a server that implements the OpenAI embeddings API, such as an inference server inside the deployment's network.

ParameterTypeDefaultDescription
endpointStringhttps://api.openai.com/v1The base URL of the embeddings server. An air-gapped deployment sets an internal server.
api_keyStringEmptyThe key that the server requires. A server without authentication needs no key.
dimensionsIntegerThe model's native dimensionThe number of dimensions to request from a model that can shorten its vectors, such as text-embedding-3-large, which returns 3,072 dimensions by default. Must be greater than 0.

The model field is the model name that the server expects. Other parameters return HTTP 400.

GPT4All​

GPT4AllEmbeddings runs GGUF embedding models with the GPT4All library. The model field is the name of a model file in the GPT4All model directory.

ArtifactContent
Package image gpt4allThe GPT4All Python package, mounted into the gRPC service
Model image gpt4allall-MiniLM-L6-v2.gguf2.f16.gguf (384 dimensions) and nomic-embed-text-v1.5.f16.gguf (768 dimensions)

The standard gRPC service image does not include the GPT4All package, so the provider requires the package image. Parameters are passed to the LangChain GPT4AllEmbeddings class, such as n_threads and device. Foundation4 sets the model name and the model path, so parameters cannot change them.

Hugging Face​

HuggingFaceEmbeddings runs Sentence Transformers models. The model field is the name of a model in the Hugging Face model directory, such as sentence-transformers/all-mpnet-base-v2.

ArtifactContent
Package image sentence-transformersThe Sentence Transformers package with PyTorch, mounted into the gRPC service
Model image huggingfacesentence-transformers/all-mpnet-base-v2 (768 dimensions)

Parameters are passed to the LangChain HuggingFaceEmbeddings class: model_kwargs (options for the model, such as device), encode_kwargs (options for encoding, such as normalize_embeddings), multi_process and show_progress. Foundation4 sets the model name and the cache directory, so parameters cannot change them. The provider loads the model files from the Hugging Face model directory first, and downloads a missing model from the Hugging Face Hub. With HF_HUB_OFFLINE=1 set on the gRPC service, a missing model is not downloaded: creating the embedding model returns HTTP 500 Internal server error without details, and the gRPC service log shows Provider unavailable with the model name.

HuggingFaceEndpointEmbeddings calls a Hugging Face inference endpoint. The parameter huggingfacehub_api_token is required, and a create request without the token returns HTTP 400 with Invalid parameter, and details.error ends with API token is required for HuggingFaceEndpoint embeddings. The model field names the model that the endpoint serves.

Text splitter providers​

Every text splitter runs in the gRPC service and wraps a text splitter of the LangChain text splitters library.

providerAvailabilityNetwork access
RecursiveCharacterTextSplitterAvailableNone
CharacterTextSplitterAvailableNone
CodeTextSplitterAvailableNone
MarkdownHeaderTextSplitterAvailable. Each fragment begins with the headings of its section and of the parent sections.None
TokenTextSplitterAvailableNone, with the encoding files in /data/tiktoken or in the directory that TIKTOKEN_CACHE_DIR names on the gRPC service. Otherwise the tiktoken library downloads the files.
NLTKTextSplitterAvailableNone, with the NLTK data from the nltk model image
SpacyTextSplitterRequires the spacy package imageNone. The default sentencizer pipeline needs no model files, and a named pipeline is read from /data/spacy/<pipeline>, never downloaded.
SentenceTransformersTokenTextSplitterRequires the sentence-transformers package imageAs for HuggingFaceEmbeddings: none for a model in the Hugging Face model directory
RecursiveJsonSplitterNot available: creation fails with HTTP 400Not applicable

Parameters are passed to the LangChain class as keyword arguments. A parameter that the class does not accept returns HTTP 400 when the splitter is created. A value that the class rejects, such as an invalid language or an overlap larger than the fragment size, returns HTTP 400 with Invalid parameter, and details.error carries the message of the class. A splitter whose package or data files are missing returns HTTP 503 with Provider unavailable, and details.error names what is missing.

Common parameters​

The character, recursive character, code, token, NLTK and spaCy splitters accept the following parameters. The defaults are those of the LangChain text splitters library, version 1.1.1.

ParameterTypeDefaultDescription
chunk_sizeInteger4000The maximum size of a fragment, in characters, or in tokens for TokenTextSplitter
chunk_overlapInteger200The maximum size of the overlap between consecutive fragments. Must not exceed chunk_size.
keep_separatorBoolean, "start" or "end"true for the recursive splitters, false for the othersWhether the separator stays in the fragments, and at which end
strip_whitespaceBooleantrueWhether leading and trailing whitespace is removed from each fragment

Provider parameters​

providerParameters
RecursiveCharacterTextSplitterseparators: the list of separators, tried in order. The default is ["\n\n", "\n", " ", ""]. is_separator_regex: whether the separators are regular expressions.
CharacterTextSplitterseparator: the separator. The default is "\n\n". Text without the separator stays one fragment, whatever the size. is_separator_regex.
CodeTextSplitterlanguage (required): a LangChain language name, such as python, js, java, go, rust, markdown or html. An invalid language returns HTTP 400, and details.error ends with Unspecified or invalid language: <language>.
MarkdownHeaderTextSplitterheaders_to_split_on (required): the headings that start a new fragment, such as [["#", "Header 1"], ["##", "Header 2"]]. Fragment size follows the sections of the document, with no size limit.
TokenTextSplitterencoding_name: the tiktoken encoding. The default is gpt2. model_name: selects the encoding of a model instead.
NLTKTextSplitterseparator: the separator between sentences in a fragment. The default is "\n\n". language: the NLTK language. The default is english.
SpacyTextSplitterpipeline: sentencizer, the rule-based sentence splitter of spaCy, or the name of a spaCy pipeline, such as en_core_web_sm. The default is sentencizer. separator: the separator between sentences in a fragment. The default is "\n\n".
SentenceTransformersTokenTextSplittermodel_name: the Sentence Transformers model whose tokenizer counts the tokens. The default is sentence-transformers/all-mpnet-base-v2. tokens_per_chunk: the fragment size in tokens. The default is the maximum input length of the model. chunk_overlap: the overlap in tokens. The default is 50.

Seeded objects​

Every installation includes the following objects:

ObjectIdentifierProviderConfiguration
Embedding model all-MiniLM-L6-v29ba4a409-8773-415b-aa9d-ae3a2bdbc775FastEmbedEmbeddingsQdrant/all-MiniLM-L6-v2-onnx, 384 dimensions
Text splitter019252e9-b4a0-7713-9a69-d701b4f4a2d1RecursiveCharacterTextSplitterchunk_size 1,000 and chunk_overlap 100. An installation upgraded from an earlier release keeps the default parameters: 4,000 characters, 200 characters of overlap.
Text splitter019252e9-da22-7f12-8c2f-36f4025fb0afCharacterTextSplitterDefault parameters
Text splitter019252e9-f2b1-7482-930f-0cf9400bdd79NLTKTextSplitterDefault parameters. Requires the NLTK data from the nltk model image.

Each seeded text splitter carries a description that names its splitting method.

Model and package images​

Models and Python packages that the standard images do not include are delivered as separate images, which the Helm charts mount into the containers:

ImageContentMounted at
API server imageThe Foundation4 binary and the FastEmbed models marked Yes in the FastEmbed table, in /data/fastembedNot applicable
gRPC service imageThe Python service, the LangChain libraries, the NLTK package and the tiktoken libraryNot applicable
Package image gpt4allThe GPT4All package/packages/gpt4all in the gRPC service
Package image sentence-transformersSentence Transformers and PyTorch/packages/sentence-transformers in the gRPC service
Package image spacyThe spaCy package, without language models/packages/spacy in the gRPC service
Model image gpt4allTwo GPT4All embedding models/data/gpt4all
Model image huggingfacesentence-transformers/all-mpnet-base-v2/data/huggingface
Model image nltkThe NLTK sentence data punkt_tab/data/nltk
Model image tiktokenThe tiktoken encodings gpt2, r50k_base, p50k_base, cl100k_base and o200k_base/data/tiktoken

An air-gapped installation mirrors every image that the deployment uses. The CLI command foundation4ai data check-models checks that every configured embedding model and text splitter loads, without downloading FastEmbed model files, as described in Command-line interface. The API server, worker and gRPC service containers mount each model image at the same path. Models, packages and air-gapped installs describes the image configuration and a check of the files that each container sees.