Organize and search documents by path

This guide gives documents a path, lists the paths of a pipeline, searches under a path, lists the documents under a path and moves a document to another path. Integrators follow this guide to scope searches to a folder, a project or a source system within one pipeline. The examples use the support-kb pipeline from First search and the shell variables from Authenticate. Document paths describes the path syntax and the matching rules.

Path layout

A path layout maps the structure of the source content onto labels separated by periods. The following layouts are common:

Source system. The first label names the system the content comes from, such as wiki, tickets or kb, so a search can cover one source or all sources.
Topic tree. The labels follow a topic hierarchy, such as kb.billing.tax.vat, so a search can cover one topic and every topic below the topic.
Owner. A label names a team, a project or a customer, such as projects.atlas.design, so a search can cover the content of one owner.

A layout gives every document of the pipeline a path, or no document a path. A search without path finds only the documents without a path, and a search with a pattern finds only documents with a path, so a pipeline that mixes both needs two searches to cover all content. The support-kb pipeline already holds three documents without a path. The documents of this guide have paths, so the earlier searches of First search return the same results after this guide.

A path addresses content and does not restrict access: any key that can search the pipeline can search any path. Document paths describes how to keep content apart.

Documents with paths

Add three documents, each with a path in the kb source and a topic below kb:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "classification": "internal",
    "external_identifier": "kb-sso-setup",
    "path": "kb.accounts.sso",
    "metadata": {"product": "accounts"},
    "contents": "Single sign-on is configured on the Security page. Administrators upload the metadata of the identity provider and assign users to groups."
  }'

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "classification": "public",
    "external_identifier": "kb-card-update",
    "path": "kb.billing.payment-methods",
    "metadata": {"product": "billing"},
    "contents": "Customers update the card on file from the Billing page. The new card is charged from the next invoice."
  }'

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "classification": "internal",
    "external_identifier": "kb-vat-rates",
    "path": "kb.billing.tax.vat",
    "metadata": {"product": "billing"},
    "contents": "Invoices for customers in the European Union show value-added tax at the rate of the country of the customer."
  }'

Expected result: status 201 for each request, with status set to pending and path set to the path of the request. A path with an uppercase letter, a slash or an empty label returns HTTP 422 and creates no document. Set a variable from the id field of each response:

export SSO_DOC_ID=<id from the first response>
export CARD_DOC_ID=<id from the second response>
export VAT_DOC_ID=<id from the third response>

Retrieve each document until status is success, as in First search.

Path list

List every path of the pipeline:

curl -G "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/paths" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 200 and the three paths in byte order, each with one document. The three documents without a path are not listed:

{
  "data": [
    {"path": "kb.accounts.sso", "documents": 1},
    {"path": "kb.billing.payment-methods", "documents": 1},
    {"path": "kb.billing.tax.vat", "documents": 1}
  ],
  "next": null
}

List the paths exactly one level below kb.billing. The -G and --data-urlencode options of curl encode the pattern as a query parameter:

curl -G "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/paths" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  --data-urlencode "path=kb.billing.*"

Expected result: status 200 and kb.billing.payment-methods only. kb.billing.tax.vat is two levels below kb.billing, so kb.billing.* does not match the path, and kb.billing.> does. A pipeline with many paths returns 100 paths per page by default: first sets the page size, up to 1,000, and after set to the next value of a page returns the following page.

Search under a path

Search the documents under kb.billing, at any depth:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/search" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How do customers change their card?",
    "type": "similarity",
    "classification": "internal",
    "path": "kb.billing.>",
    "params": {"k": 5}
  }'

Expected result: status 200 and two fragments, from kb-card-update and kb-vat-rates, each with path set to the path of its document. The fragment of kb-card-update comes first, because the fragment is closest to the query. kb-sso-setup is outside the pattern, and the three documents without a path are outside every pattern.

The path field accepts every pattern of Document paths. On this pipeline, the patterns return the following documents:

`path`	Documents searched
`kb.billing.tax.vat`	`kb-vat-rates`
`kb.billing.*`	`kb-card-update`
`kb.billing.>`	`kb-card-update`, `kb-vat-rates`
`kb.>` or `>`	The three documents of this guide
Field omitted or `null`	`kb-password-reset`, `kb-invoice-export`, `kb-refund-policy`

The pattern combines with the classifications and the metadata filter of the request, and applies to every search type, including hybrid and history search. An agent request takes the same path field, which applies to every placeholder of the agent, as described in Agents and prompt templates.

Documents under a path

The document list takes a pattern in the path query parameter:

curl -G "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  --data-urlencode "path=kb.billing.>"

Expected result: status 200 and a page with the documents kb-card-update and kb-vat-rates. Without path, the document list returns all six documents of the pipeline, with and without a path. The list follows the paging and filters of Paginate and filter lists.

Moved documents

A new version with a different path moves a document. Every version carries its own path, so a new version always sends path: a version without path has no path. Move kb-vat-rates from kb.billing.tax.vat to kb.billing.vat by submitting a new version with the same external identifier, classification and contents:

curl -X POST "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "classification": "internal",
    "external_identifier": "kb-vat-rates",
    "path": "kb.billing.vat",
    "metadata": {"product": "billing"},
    "contents": "Invoices for customers in the European Union show value-added tax at the rate of the country of the customer."
  }'

Expected result: status 201, the identifier in VAT_DOC_ID, a new version and path set to kb.billing.vat. The previous version expires at submission, so the document leaves searches under kb.billing.tax.vat at once and appears under kb.billing.vat when the new version reaches success. The path list then shows kb.billing.vat in place of kb.billing.tax.vat, and kb.billing.* now matches the document.

Cleanup

Delete the three documents of this guide, so that the document lists of other guides show the three documents of First search:

curl -X DELETE "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents/$SSO_DOC_ID" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

curl -X DELETE "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents/$CARD_DOC_ID" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

curl -X DELETE "$FOUNDATION4_URL/pipelines/$PIPELINE_ID/documents/$VAT_DOC_ID" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET"

Expected result: status 204 for each request. The path list then returns an empty data array.

Path layout​

Documents with paths​

Path list​

Search under a path​

Documents under a path​

Moved documents​

Cleanup​