Add, rebuild or switch a search index

This guide changes the search indexes of a pipeline that already holds documents: adding a BM25 index and making the index the default, adding and removing a distance index, rebuilding the database indexes of a search index, retrying a failed index, switching to another embedding model and moving a vector index from pgvector to VectorChord. Integrators and administrators who maintain a pipeline need this guide. Search indexes describes the index kinds, the defaults and the life cycle of an index.

Search indexes of the pipeline

The examples use the support-kb pipeline from First search and the shell variables from that tutorial. The API key requires read permission on the pipeline to list the indexes, and write permission for every change. List the indexes of the pipeline:

curl "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and the two indexes created with the pipeline, ordered by identifier. The value of size_bytes depends on the content:

[
  {
    "id": "5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60",
    "type": "vchord",
    "embedding_model_id": "9ba4a409-8773-415b-aa9d-ae3a2bdbc775",
    "tokenizer": null,
    "status": "ready",
    "backfilled_count": 0,
    "message": null,
    "is_default": true,
    "rebuild": null,
    "distance_indexes": [
      {
        "distance": "cosine",
        "name": "5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60:cosine_idx",
        "valid": true,
        "size_bytes": 16384
      }
    ]
  },
  {
    "id": "8e3b6f1d-2c4a-4d7e-9b05-6a1f2e3d4c5b",
    "type": "fts",
    "embedding_model_id": null,
    "tokenizer": null,
    "status": "ready",
    "backfilled_count": 0,
    "message": null,
    "is_default": true,
    "rebuild": null,
    "distance_indexes": []
  }
]

backfilled_count is 0 because the two indexes were created with the pipeline, before the pipeline held documents. Set two variables from the id fields:

export VECTOR_INDEX_ID=<id of the vchord index>
export TEXT_INDEX_ID=<id of the fts index>

BM25 index

Add a BM25 index with the default tokenizer:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{"type": "bm25"}'

Expected result: status 201 and the new index, with status set to building, because the pipeline already holds fragments, and is_default set to false. tokenizer shows the default tokenizer, {"model": "bert_base_uncased"}. A tokenizer object in the request selects another tokenizer, as described in BM25 tokenizers. Set a variable from the id field:

export BM25_INDEX_ID=<id from the response>

A worker fills the new index from the stored fragment text. Read the index until status is ready:

curl "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers/$BM25_INDEX_ID" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200, status set to ready and backfilled_count set to 3, one for each fragment of the pipeline. A search that names the index while the index is building returns HTTP 409. Search the new index by naming the index in params.index:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "refund reason ticket",
    "type": "search",
    "classification": "internal",
    "params": {"index": "'"$BM25_INDEX_ID"'"}
  }'

Expected result: status 200 and the fragment of the kb-refund-policy document, the only fragment that shares tokens with the query, with a BM25 score greater than 0. The other searches of the pipeline are unchanged, because an added index is not a default.

Default index

Make the BM25 index the pipeline's default text index:

curl -X PUT "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers/$BM25_INDEX_ID/default" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and the pipeline, with default_text_search_index_id set to the identifier of the BM25 index. From the next request, every full-text or hybrid search that names no index, and every full-text or hybrid placeholder of an agent, reads the BM25 index instead of the full-text index. Only a ready index can become the default; a request for an index that is building or failed returns HTTP 409.

A default index cannot be deleted. A delete request for the BM25 index now returns HTTP 409 with details.error set to Search indexer <id> is a default of the pipeline: make another indexer the default first. Restore the full-text index as the default, and delete the BM25 index:

curl -X PUT "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers/$TEXT_INDEX_ID/default" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

curl -X DELETE "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers/$BM25_INDEX_ID" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and the pipeline with the full-text index as the default, then status 204. Deleting an index removes the index's table, and searches that name the deleted index return HTTP 400. Adding or deleting an index waits up to 10 seconds while a worker processes documents of the pipeline, and then returns HTTP 409 with details.error set to The pipeline is busy processing documents, try again shortly. The client application repeats the request in that case.

Distance index

The vector index of support-kb has a distance index for cosine only, so a search with another distance strategy returns HTTP 400. Add a distance index for euclidean-distance:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers/$VECTOR_INDEX_ID/distance-indexes" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{"distance": "euclidean-distance"}'

Expected result: status 202 and the name of the database index under construction:

{
  "id": "5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60",
  "distance": "euclidean-distance",
  "index": "5d1e8c2a-3f4b-4a6c-9e7d-1b2c3d4e5f60:l2_idx"
}

The database builds the index concurrently, so searches and document processing continue. List the distance indexes until the new entry shows valid set to true:

curl "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers/$VECTOR_INDEX_ID/distance-indexes" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and two entries, cosine and euclidean-distance, each with valid set to true. A search with "distance_strategy": "euclidean-distance" now runs on the new index: the similarity search for How do customers get their money back? scores the refund fragment 1.1353 instead of the cosine distance 0.6445. A search without distance_strategy still uses cosine, the first indexed strategy.

Remove the distance index:

curl -X DELETE "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers/$VECTOR_INDEX_ID/distance-indexes/euclidean-distance" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 204. A search with euclidean-distance returns HTTP 400 again. The cosine distance index can be removed in the same way, but a vector index without any distance index cannot be searched.

Rebuild

A rebuild replaces each database index of a search index with a newly built copy, one after the other, while searches and document processing continue on the existing copy. A rebuild reclaims the space that the entries of deleted and replaced fragments occupy in the database indexes. Rebuild the vector index:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers/$VECTOR_INDEX_ID/rebuild" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 202, with the identifier of the search index in id and the names of the database indexes being rebuilt in indexes. While the rebuild runs, the rebuild field of the search index reports the database index, the phase and the progress in blocks and rows. The rebuild is complete when rebuild is null. A second rebuild, or a distance index request, on the same search index during the rebuild returns HTTP 409.

Failed index

An index whose backfill fails 10 times in a row, for example because the embedding model of a vector index cannot be reached, has status set to failed and the reason in message. After the cause is fixed, restart the backfill:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search-indexers/$INDEX_ID/retry" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and the index with status set to building. The backfill starts again from the first fragment. A retry of an index that has not failed returns HTTP 409 with details.error set to Search indexer <id> has not failed. A failed index can also be deleted.

Switch the embedding model

A pipeline moves to another embedding model without a new ingestion. The workers embed the stored text of every fragment with the new model, so the backfill calls the new model once for each fragment of the pipeline.

1. Find or create the new embedding model, as described in Embedding models and text splitters, and set NEW_MODEL_ID to the model's identifier.
2. Add a vector index for the model with POST /pipelines/{id}/search-indexers and the body {"type": "vchord", "embedding_model_id": "<NEW_MODEL_ID>"}. Expected result: status 201 and the index with status set to building.
3. Read the index until status is ready. Searches continue on the old index during the backfill, and documents processed during the backfill are written to both indexes.
4. Add a distance index for each strategy, other than cosine, that the client applications use.
5. Compare the two models with similarity searches that name each index in params.index.
6. Make the new index the default with PUT /pipelines/{id}/search-indexers/{index_id}/default. Expected result: status 200, and searches that name no index use the new model from the next request.
7. Delete the old index. Expected result: status 204.

Agents follow the default vector index. An agent placeholder that names the old model in params.embedding keeps searching the old index until the index is deleted, and then fails. Such placeholders are changed to the new model, or the embedding parameter is removed, before step 7.

Move from pgvector to VectorChord

A pipeline created with an earlier release keeps a pgvector vector index after the upgrade, with distance indexes for cosine, euclidean-distance and dot-product. Both kinds of index return the same distances, so a pipeline can stay on pgvector. A pipeline moves to VectorChord with the following steps:

1. Add a VectorChord index for the same embedding model, with the body {"type": "vchord", "embedding_model_id": "<model id of the pgvector index>"}. The new index copies the vectors of the pgvector index, so the backfill calls no embedding model.
2. Read the index until status is ready.
3. Add the distance indexes for euclidean-distance and dot-product if client applications use those strategies, and wait until each is valid.
4. Make the VectorChord index the default.
5. Delete the pgvector index.