Engineering notes · Import pipeline
We rebuilt the most expensive part of book import. A measured A/B test on the same chapter cut the annotation phase from 106 AI calls to 3 and roughly halved wall-clock time, with no loss in definition quality.
The short version. Importing a philosophy book runs each chapter through an AI pipeline that splits it into sections, finds the hard words, and writes two kinds of definitions. The old design made one AI call per word occurrence — about a hundred calls per chapter — and asked the model to retype the entire text just to add links. We replaced that with one call per small batch of sections and let plain code insert the links. On a real sample chapter the annotation phase dropped from 106 calls / ~12 min to 3 calls / ~3 min, and the definitions read just as well.
A reader on Be-ing sees a philosophy text in the left pane. Hard words are clickable. Click one and the right pane shows two things: a global definition (what the word means in general, shared across every book) and an in-context definition (what the author means by it in that exact sentence). Everything in the import pipeline exists to produce those definitions and the links that point to them.
A book is organized as book → chapters → sections. Sections are bite-sized reading chunks (~150–300 words) that the AI invents during import. The work happens in four CLI commands:
pnpm import-book — download from Project Gutenberg, strip boilerplate, split into chapters.pnpm process-chapter — the expensive one. Split each chapter into sections, identify terms, write in-context definitions, and insert links.pnpm generate-definitions — fill in the global definitions (deliberately deferred so chapter processing stays focused).pnpm review-book — an AI quality report.Nearly all of the cost lived in step 2.
A token is the unit AI usage is billed in — roughly three-quarters of a word. Every call has input tokens (everything you send) and output tokens (everything the model writes back). Three things in the old process-chapter design quietly inflated both.
The old pipeline identified the terms in a batch of sections, then made a separate AI call for every single occurrence of every term to write its in-context definition. Those calls reused one "session" so the model would remember the chapter — but resuming a session works like forwarding an entire email thread every time you ask one new question. Call one sends the section text. Call two sends the section text plus call one's question and answer. By the twelfth call, you're sending ~10,000 tokens of replayed history to get back an 80-token answer.
To add links, the old "annotate" step asked the model to re-emit the entire section text word for word with [term](#anchor) syntax sprinkled in. That is the most expensive kind of output, it can subtly alter the author's words, and it fumbled the JSON wrapper often enough that roughly a third of windows silently shipped with no links at all.
Sections were processed in overlapping windows of five (three to actually process, two as neighboring "context"). Those extra context sections fattened every identification call without much benefit — the in-context definition is about this sentence, and the section it lives in is context enough.
Two focused changes, each independently shippable.
The new module src/lib/ai/annotate-links.ts is a pure function. The model already told us each term, where it appears, and which anchor ID each occurrence gets — so inserting [will to truth](#def-will-to-truth) is a deterministic find-and-replace. It matches whole words case-insensitively (Unicode-aware), prefers compound phrases over their parts (ontological poverty wins over a bare ontological elsewhere), preserves the original casing, and links every appearance. Zero tokens, byte-identical text, and the parse-failure bug is gone by construction. It ships with 13 unit tests.
A new prompt, IDENTIFY_AND_DEFINE, asks for the terms and their in-context definitions in a single JSON response per batch of four sections — no session resume, no context neighbors. It carries the same 25–60 word style contract and the same good/bad examples that keep the old per-occurrence definitions tight. A new provider method annotateAndDefineBatched runs it; the existing pnpm test-definitions tool gained a --pipeline legacy|batched flag so the two approaches can be measured on identical input.
We ran both pipelines on the same sample chapter — the opening of David Bentley Hart's The Experience of God (13,524 characters, split into 10 sections), using the same model (opus). The only variable was the pipeline.
Annotate-phase calls
106 → 3
~35× fewer
Total run time
19.3 → 9.9 min
includes shared global-def phase
In-context defs
102 / 97
equivalent coverage
Def length (median)
36 / 27 words
both inside 25–60 contract
The headline numbers — call counts and wall-clock time — are measured from the two runs. The per-occurrence design's cost also showed up live: the legacy run made 19, 28, 33, and 22 in-context calls across its four windows (102 total, plus 4 identify calls), each replaying the growing session. The batched run did the same work in three calls total.
On token counts. The Claude Agent SDK on the Pro subscription doesn't surface exact token counts in this harness, so the token figures we used while planning (~5.9M input → ~180k per book) are modeled estimates, not measurements. What the A/B did measure — call counts and wall time — confirms the mechanism: removing conversation replay and text re-emission is where the savings come from.
This was the real question. The per-occurrence design originally existed to stop definitions from drifting long and generic. Batching them didn't bring that back — if anything the batched definitions are tighter (median 27 vs 36 words, both well inside the 25–60 word target). Read the same term from each run:
Legacy · "contingency"Hart means the world's quality of not having to exist — its dependence and lack of self-necessity. Everything could just as easily not be; nothing about reality requires or guarantees its own existence.
Batched · "contingency"Hart means that the world need not exist and need not be as it is — it has no inherent necessity, depending entirely on something beyond itself for its being.
Term selection differed only at the margins, and in both directions: the legacy run uniquely caught 22 terms, the batched run uniquely caught 14. Reading the diffs, these are judgment calls rather than misses. Notably, the batched run followed our own compound-phrase guidance better in places — it defined ontological pedigrees and ontological indigence as compounds (the exact examples in our definitions guide) where the legacy run only linked the bare words.
The A/B surfaced two minor issues, both already diagnosed and neither a blocker:
def-contingency can be issued in two different batches. (The legacy path has the same per-window flaw; it's harmless within a section.) A single shared counter across batches fixes it.Switch process-chapter to the batched pipeline. Quality held on every axis we could check, and the cost difference isn't close. The remaining work, in order:
process-chapter (with the two fixes above).generate-definitions too — it's now the largest remaining cost (~410–440s of each run was that shared phase, still one call per term). Batching ~20 terms per call would turn a full chapter into roughly four AI calls end to end.All changes are covered by the test suite (317 passing) and lint/typecheck clean.
The AI-generated definition outputs from both runs are kept alongside this post:
The input chapter and the full annotated-section outputs reproduce the copyrighted source text (Hart's The Experience of God), so they are kept locally under the gitignored books/test/ rather than committed here.