0005: Full-text search with pluggable providers

Context#

A decision archive is useful in proportion to how fast it answers “did we ever decide anything about X”. The filter page covers titles, tags and descriptions, but the reason behind a decision is in the body. Search over the bodies must work on a static host with no server and no account, because that is how most archives start. Larger organisations already run a search service and want the archive in it, with the same ranking, synonyms and analytics as the rest of their docs. Both cases need the same palette; only the index behind it differs.

Decision#

Inkan has one search entry point: a button in the header and the ⌘K / Ctrl K shortcut open a palette. The palette asks a provider for results. The default provider is Pagefind: inkan build indexes the finished HTML and writes the index to dist/pagefind/, so search runs from static files. Alternatively the search option names Algolia, Meilisearch, Typesense or a custom module; the build then writes dist/search.json, a flat list of records you push into that service, and the palette queries the service from the browser. search: false removes search altogether.

How to use it#

The palette#

Inside the palette ↑ and ↓ move through the results, Enter opens the selected one and Esc or a click outside closes it. With an empty query the palette lists the seven newest records under “Recent decisions”. A query is sent 120 ms after the last keystroke; a newer query aborts the one in flight. Each result shows the status sticker, the title, the project and type, an excerpt with the matches marked and the record reference (SRE-ADR-0007). The footer says how many results there are in all, for example 8 of 23 results.

The palette shows at most maxResults hits (default 8, between 1 and 50). Hits whose url is not an http(s) page are dropped.

Choosing a provider#

1
export default defineConfig({
2
  search: { type: 'pagefind', maxResults: 8 },
3
});

Every provider accepts these two options:

Without type the provider is pagefind. The schema is strict: a key that does not belong to the chosen provider fails the build, for example search.apiKey: unknown option with type: 'pagefind', and search.type: type must be one of pagefind, algolia, meilisearch, typesense, custom for anything else. A plugin may set search in its config hook; the result is validated again after the hooks ran, so a plugin that forgets host fails with search.host in the message.

Pagefind (default)#

1
export default defineConfig({
2
  search: {
3
    type: 'pagefind',
4
    ranking: { termFrequency: 1.0, pageLength: 0.75, termSimilarity: 1.0, termSaturation: 1.4 },
5
  },
6
});

The index is produced by the build, not by the dev server. In inkan dev the palette opens, but a query shows “Search works after inkan build; run inkan preview to try it.” The build log ends with Search index: N pages. Deploy the whole dist/ folder; the page loads pagefind.js from <base>/pagefind/, so a sub-path deployment needs no extra step.

Every record body and every Markdown page is indexed. Navigation, the record header, the right rail, the footer and the pager are excluded. Each record also carries Pagefind metadata and filters: title, ref, project (the project name), type and status (their labels) and one tag entry per tag. A custom page can call the Pagefind JS API at <base>/pagefind/pagefind.js with filters such as { project: 'SRE' }.

Algolia, Meilisearch, Typesense#

1
export default defineConfig({
2
  search: { type: 'meilisearch', host: 'https://search.acme.dev', apiKey: process.env.MEILI_SEARCH_KEY, indexUid: 'decisions' },
3
});

The palette asks the service for maxResults hits with <mark> highlighting on content and description; params is merged into that request last, so it can override anything. A hit is read as a record (see below): url (or href), title, description, content and the project, type, status and ref fields, so an index with another shape works as long as those names match. When the service cannot be reached or answers with an error, the palette says so and names the options to check.

External providers work in inkan dev too, because the browser talks to the service directly.

search.json#

With any provider other than pagefind, the build writes dist/search.json instead of the Pagefind index (in dev it is served at /search.json). The file is a JSON array with one record per built record and per Markdown page:

Push it to the service after every build, for example in the CI job that deploys dist/:

1
# Meilisearch
2
curl -X POST "$MEILI_HOST/indexes/decisions/documents?primaryKey=id" \
3
  -H "Authorization: Bearer $MEILI_ADMIN_KEY" -H 'Content-Type: application/json' --data-binary @dist/search.json
4

5
# Typesense (create the collection with title, description, content and tags fields first)
6
jq -c '.[]' dist/search.json | curl -X POST "$TYPESENSE_HOST/collections/decisions/documents/import?action=upsert" \
7
  -H "X-TYPESENSE-API-KEY: $TYPESENSE_ADMIN_KEY" --data-binary @-
8

9
# Algolia
10
curl -X POST "https://$ALGOLIA_APP_ID.algolia.net/1/indexes/decisions/batch" \
11
  -H "X-Algolia-Application-Id: $ALGOLIA_APP_ID" -H "X-Algolia-API-Key: $ALGOLIA_ADMIN_KEY" \
12
  --data-binary "$(jq '{requests: map({action: "updateObject", body: .})}' dist/search.json)"

Drafts are not built, so they are not in the file. /search.json is a reserved route, like /pagefind/.

A custom provider#

1
export default defineConfig({
2
  search: { type: 'custom', module: './search.ts', options: { endpoint: 'https://search.acme.example' } },
3
});

The module runs in the browser. It gets options and returns an object with a search method and an optional preload, which the palette calls when it opens so the first query is fast. The types come from @tilaven/inkan/search, together with SearchUnavailableError: throw it to show your own message in the palette (text in backticks renders as code) instead of the generic “Search failed. Try again.”

1
import { SearchUnavailableError, type SearchProviderFactory } from '@tilaven/inkan/search';
2

3
const acme: SearchProviderFactory<{ endpoint: string }> = ({ endpoint }) => ({
4
  async search(query, { limit, signal, base }) {
5
    const res = await fetch(`${endpoint}?q=${encodeURIComponent(query)}&limit=${limit}`, { signal });
6
    if (!res.ok) throw new SearchUnavailableError(`Search answered ${res.status}. Check \`endpoint\`.`);
7
    const { hits, total } = await res.json();
8
    return { hits, total }; // hits: SearchHit[], total: matches in the index
9
  },
10
});
11

12
export default acme;

The same types are also exported from the root module @tilaven/inkan (SearchConfig, SearchRecord, SearchHit, SearchProvider, SearchProviderFactory, SearchContext, SearchResults). To ship a provider as a package, let a plugin set search in its config hook; give module an absolute path or a file: URL (new URL('./search.ts', import.meta.url).href), because a relative path is resolved against the site’s config file, not the plugin.

Drafts and hidden pages#

Pages with draft: true are not built in production, so they are in neither index. Files and folders starting with _ are never pages and never indexed.

Consequences#

• The default needs no service, no key and no tracking; the Pagefind index is only available after a build, so reviewing it locally means build then preview.
• Pagefind indexes the rendered HTML, so anything hidden from the DOM is not found; search.json is built from the Markdown source, so an external index sees the text before rendering, without MDX components or diagrams.
• An external provider puts a search key in the page and makes every query a request to a third party; the archive’s text lives in that service as well.
• Keeping an external index current is your CI’s job; the build only writes search.json.
• Custom providers are a browser contract, not a build-time one: the module cannot read the content folder, it can only call something that already holds the records.
• The free-text field on the record index page stays metadata-only, see Filterable record index.