Metadata, filters and taxonomies

Metadata is structured data that a client application attaches to each document, such as a room identifier, an author, a date or a product line. This page describes how a pipeline validates metadata with a schema, how search requests filter fragments by metadata, and how taxonomies extend a filter across a hierarchy of values. Integrators need this page to design the metadata of a pipeline and to scope search results per request.

Metadata

The metadata of a document is a JSON object, submitted in the metadata field of the create request. Foundation4 copies the metadata of a document to every fragment of the document, so every search result carries the metadata of the source document. Metadata is stored unencrypted, because PostgreSQL compares metadata values directly when applying filters. Content that must remain encrypted belongs in the document text, not in metadata.

Metadata schema

Each pipeline has a metadata schema: a JSON Schema document that the metadata of every submitted document must satisfy. A document whose metadata does not satisfy the schema is rejected with HTTP 400 and the message Invalid metadata, and details lists each failing field with the reason. A missing required field is the exception: details reports it under an empty key, such as {"": "\"msg_id\" is a required property"}, and reports only one missing field per request.

The schema also declares the type of each field that a filter can reference. A filter on a field that the schema does not declare returns an error. A pipeline that relies on metadata filters therefore declares every filterable field in the schema.

Foundation4 accepts a subset of JSON Schema:

• Root object. The root of the schema has the type object and a properties object. An empty schema, {}, accepts any metadata and supports no filters. Foundation4 does not check the root type when the pipeline is created, and a schema with another root type, such as {"type": "string"}, rejects the metadata of every document.
• Field names. Field names consist of letters, digits and underscores, and do not start with a digit.
• Field types. Each field declares one type: string, integer, number, boolean or object. A field of type object declares the nested fields in a properties object.
• Arrays. Fields of type array are not supported.

Standard JSON Schema validation keywords, such as required, enum and maxLength, apply as usual. The following schema declares the fields that a chat integration uses to scope and display results:

{
  "type": "object",
  "properties": {
    "room_id": {"type": "string"},
    "msg_id": {"type": "string"},
    "username": {"type": "string"},
    "timestamp": {"type": "string"}
  },
  "required": ["room_id", "msg_id"]
}

The schema can be replaced after the pipeline is created, as described in Pipelines. A replacement schema applies to documents submitted after the change and to every later filter. Metadata already stored is not validated again, so a replacement schema that changes the type of an existing field can cause filters on that field to fail for older documents.

Filters

A filter restricts a search to fragments whose metadata satisfies a condition. Every search mode accepts a filter in the filters field of the request, and PostgreSQL applies the filter in the same query that selects the fragments. Full-text search therefore returns up to k fragments that satisfy the filter, rather than filtering a fixed set of k results after retrieval. Similarity and MMR search use an approximate vector index, so a search with a selective filter can return fewer than k fragments.

A filter is a JSON object that maps a field name to a comparison. The following filter restricts results to two rooms:

"filters": {"room_id": {"$in": ["GENERAL", "6f1c2a"]}}

The comparison operators are the following. The value in each comparison must have the type that the schema declares for the field.

Operator	Meaning	Field types
$eq	Equal to	All
$neq	Not equal to	All
$gt, $ge	Greater than, greater than or equal to	All
$lt, $le	Less than, less than or equal to	All
$like	Matches a SQL LIKE pattern, case-sensitive (% matches any sequence of characters)	String
$ilike	Matches a SQL LIKE pattern, case-insensitive	String
$in	Equal to one of the values in an array	All
$not-in	Equal to none of the values in an array	All
$teq	Equal to a value or to any value beneath that value in a taxonomy	String

Filters apply to top-level metadata fields. Fields of type object cannot be compared directly.

Several fields in one object must all satisfy their comparisons. Each field takes exactly one operator, so a range on one field combines two conditions with $and. The logical operators combine conditions explicitly:

• $and. An array of filters, all of which must be satisfied.
• $or. An array of filters, at least one of which must be satisfied.
• $not. A single filter that must not be satisfied.

The following filter restricts results to two rooms and to messages sent on or after 1 September 2026 by any author other than a system account:

"filters": {
  "$and": [
    {"room_id": {"$in": ["GENERAL", "6f1c2a"]}},
    {"timestamp": {"$ge": "2026-09-01T00:00:00.000Z"}},
    {"$not": {"username": {"$eq": "system"}}}
  ]
}

The $not condition also excludes messages without a username field, because a comparison on a missing field is never satisfied, whatever the operator. $not around an $and is the exception, as Missing values describes. Timestamps stored as ISO 8601 strings in UTC, all in the same format, compare chronologically. Filter operators lists every operator with value types, examples and errors.

Taxonomies

A taxonomy is a hierarchy of metadata values, such as regions that contain countries and countries that contain cities. The $teq operator matches a value together with every value beneath that value in a taxonomy, so a filter for a region matches documents tagged with the region, with any country in the region, or with any city in those countries.

A taxonomy is created with POST /taxonomies and is independent of any pipeline, so one taxonomy can serve several pipelines. Entries are added with POST /taxonomies/{id}/entries. Each entry links a parent field and value to a child field and value:

parent	parent_value	child	child_value
region	Europe	country	France
region	Europe	country	Germany
country	France	city	Paris

The parent field and the child field can differ, so a taxonomy can link values across several metadata fields. The following filter uses the taxonomy above:

"filters": {"region": {"$teq": ["Europe", "0b8d6f3e-5c1a-4e2b-9a7d-3f6e1c2b4a90"]}}

The first element is the value and the second element is the taxonomy identifier. Foundation4 expands the filter through every level of the taxonomy, and matches documents where region is Europe, country is France or Germany, or city is Paris. Every field reached by the expansion must be declared in the pipeline's schema as a string. Changes to taxonomy entries take effect in the next search, without reprocessing any document.

Each link from a parent field and value to a child field and value is stored once across all taxonomies. When a second taxonomy adds a link that another taxonomy already holds, the request returns HTTP 201 without that entry, and a filter that names the second taxonomy does not expand through the link. A link therefore belongs to the first taxonomy that adds the link.