Search indexes
A search index holds the fragments of a pipeline in the form that one kind of search reads: vectors from one embedding model, or the words of each fragment for text search. This page describes the kinds of search index, the default indexes of a pipeline, the life cycle of an index, distance indexes and the operations that manage indexes. Integrators need this page to design the search of a pipeline, and administrators need this page to change the indexes of a pipeline that already holds content.
Index kinds
A pipeline has any number of search indexes, of the following kinds:
| Kind | type | Contents | Search types | Limit per pipeline |
|---|---|---|---|---|
| Vector index on VectorChord | vchord | One vector per fragment, computed by the index's embedding model | similarity, mmr, hybrid | One per embedding model |
| Vector index on pgvector | pgvector | As vchord | similarity, mmr, hybrid | One per embedding model |
| Full-text index | fts | The words of each fragment, stemmed with PostgreSQL text search for the detected language | search, hybrid | One |
| BM25 index | bm25 | The tokens of each fragment, produced by the index's tokenizer, for ranking with the BM25 formula | search, hybrid | One per tokenizer |
The API names a search index a search indexer: the operations are under /pipelines/{id}/search-indexers, and the identifier of an index is the value that a search request passes to select the index. Full-text and BM25 indexes are both text indexes. The full-text index requires every significant word of the query in a fragment, and a BM25 index ranks every fragment that shares a token with the query. BM25 tokenizers lists the tokenizers of BM25 indexes. Search and retrieval describes how each search type uses an index.
Every index of a pipeline receives every fragment. A worker that processes a document writes the fragments to all the indexes of the pipeline, so the cost of ingestion grows with the number of vector indexes: each vector index calls its embedding model once for each fragment.
A pipeline receives its first indexes at creation. embedding_model_id creates a VectorChord index for that embedding model, and full_text_search creates a full-text index (fts) or a BM25 index (bm25). A pipeline created with neither field has no search index and supports only history search. Pipelines describes the create request.
Default indexes
A pipeline has one default vector index and one default text index. A search that names no index uses the default index of the kind that the search type reads, and a hybrid search that names no index fuses the two defaults. Agents search the default indexes, except that a similarity or MMR placeholder can name an embedding model, as Agents and prompt templates describes.
The pipeline response shows the defaults and every index:
| Field | Content |
|---|---|
default_vector_embedding_index_id | The identifier of the default vector index, or null |
default_text_search_index_id | The identifier of the default text index, a full-text or a BM25 index, or null |
indexers | Each index by identifier, with the kind: {"VectorChord": <embedding model>}, {"PgVector": <embedding model>}, "FullTextSearch" or {"Bm25": <tokenizer>} |
indexer_status | The status of each index by identifier: building, ready or failed |
The indexes created with a pipeline become the defaults. An index added later does not change the defaults, so a pipeline created without a text index has no default text index until a client application makes the added index the default. PUT /pipelines/{id}/search-indexers/{index_id}/default makes a ready index the default of its kind, and searches that name no index use the new default from the next request. A default index cannot be deleted: another index of the same kind becomes the default first.
Index life cycle
An index has one of three statuses:
| Status | Meaning | Searchable |
|---|---|---|
building | The index is being filled with the fragments that the pipeline already holds | No |
ready | The index holds every fragment | Yes |
failed | Filling the index stopped after repeated errors. message holds the reason. | No |
An index created with a pipeline, or added to a pipeline without fragments, is ready at once. An index added to a pipeline that holds fragments starts as building, and the workers fill the index asynchronously, a process called backfill. The workers read the stored fragment text in batches of 100 fragments, so documents are never submitted again. A new vector index whose embedding model already has a ready vector index in the pipeline copies the vectors of that index and calls no embedding model. Documents processed while an index is building are written to the index as well.
backfilled_count reports the number of fragments that the backfill has written. After 10 consecutive failed batches, such as calls to an embedding model that cannot be reached, the index becomes failed. POST /pipelines/{id}/search-indexers/{index_id}/retry restarts the backfill of a failed index from the beginning. A search that names an index that is building or failed returns HTTP 409, and only a ready index can become a default.
Distance indexes
A vector index finds the closest vectors through a database index for each distance strategy, called a distance index. A new vector index has one distance index, for cosine. POST /pipelines/{id}/search-indexers/{index_id}/distance-indexes adds a distance index for euclidean-distance or dot-product, and DELETE on .../distance-indexes/{distance} removes one. The max-inner-product strategy cannot be indexed.
A search that names a distance strategy without a distance index returns HTTP 400 instead of comparing the query with every vector of the pipeline. A search that names no strategy uses the first indexed strategy in the order cosine, euclidean-distance, dot-product. A vector index without any distance index cannot be searched.
Distance indexes are approximate nearest-neighbor indexes: each index trades a small loss of recall for query times that remain short as the pipeline grows. A distance index is built asynchronously with a concurrent build, so searches and document processing continue during the build. distance_indexes in the index's response lists each distance index with valid set to false until the build completes.
VectorChord and pgvector
Foundation4 stores vectors in PostgreSQL with one of two extensions. VectorChord builds vchordrq distance indexes, and pgvector builds hierarchical navigable small world (HNSW) distance indexes. Both extensions compute the same distances, so a fragment receives the same score from either kind of index.
New pipelines receive VectorChord indexes. A pipeline created with an earlier release keeps a pgvector index after an upgrade, with the distance indexes that the pipeline had: cosine, euclidean-distance and dot-product. A pipeline with full-text search before the upgrade receives a full-text index, which becomes the default text index.
A pipeline can hold a VectorChord index and a pgvector index for the same embedding model. A pipeline therefore moves from pgvector to VectorChord in three steps: a VectorChord index for the same model is added and copies the vectors of the pgvector index, the new index becomes the default, and the pgvector index is deleted. Add, rebuild or switch a search index describes the procedure.
Several embedding models
A pipeline can hold vector indexes for several embedding models, one per model and extension. A similarity search reads one vector index, so the embedding model of a search is the model of the index that the search names, or of the default vector index. Vectors from different models are never compared.
A change of embedding model therefore needs no new pipeline and no new ingestion. A client application adds a vector index for the new model, waits for the backfill, which embeds every stored fragment with the new model, makes the new index the default and deletes the old index. During the backfill, searches continue on the old index.
Operations
The search index operations are under /pipelines/{id}/search-indexers. Reading requires read permission on the pipeline, and every change requires write permission.
| Operation | Request | Response |
|---|---|---|
| List the indexes | GET /pipelines/{id}/search-indexers | 200 and an array of indexes, ordered by identifier |
| Add an index | POST /pipelines/{id}/search-indexers | 201 and the new index, building or ready |
| Read an index | GET /pipelines/{id}/search-indexers/{index_id} | 200 and the index |
| Delete an index | DELETE /pipelines/{id}/search-indexers/{index_id} | 204 |
| Make an index the default | PUT /pipelines/{id}/search-indexers/{index_id}/default | 200 and the pipeline |
| Retry a failed backfill | POST /pipelines/{id}/search-indexers/{index_id}/retry | 200 and the index, building |
| Rebuild the database indexes | POST /pipelines/{id}/search-indexers/{index_id}/rebuild | 202 and the names of the database indexes |
| List the distance indexes | GET /pipelines/{id}/search-indexers/{index_id}/distance-indexes | 200 and an array of distance indexes |
| Add a distance index | POST /pipelines/{id}/search-indexers/{index_id}/distance-indexes | 202 and the name of the database index |
| Remove a distance index | DELETE /pipelines/{id}/search-indexers/{index_id}/distance-indexes/{distance} | 204 |
Adding and deleting an index wait for the pipeline to be free of document processing for up to 10 seconds, and then return HTTP 409 with the message that the pipeline is busy. A rebuild runs a concurrent reindex of every database index of the search index, one after the other, while searches and document processing continue. Only one rebuild or distance index build runs on a search index at a time. Errors lists the errors of these operations.