Enrichment configs (the enrich-worker harness)

The corpus is decorated by two independent enrichment harnesses. This page documents the second one — enrich-worker — which the older enrichment.md (the queue / fraud-enricher-v2 system) does not cover.

Read enrichment.md first for the queue pipeline (fraud-publisher → fraud.rows → fraud-enricher-v2). This page is the companion for the config-loop pipeline.

Two harnesses, one corpus

Both write through fraud-db-broker into the single corpus.db on node-eighteen. They are wired completely differently:

Rule of thumb: if the output is a deterministic resolve (CIK lookup, matter grouping), a network call (CourtListener), or a bespoke LLM prompt that needs its own pool/parse/apply shape, it belongs in enrich-worker. If it is a per-document regex/LLM decorate that fits the WorkflowDef contract, it belongs in the queue harness.

Anatomy of an EnrichConfig

Defined in enrich-worker/src/config.ts. Every config implements:

export interface EnrichConfig {
  name: string;          // enrichment_name — log label; not written when apply is set
  promptVersion: number; // bump to re-enrich every row
  pool: string;          // gpumon pool to route the LLM call to
  parse: "text" | "json";// how to interpret the LLM content
  concurrency: number;   // in-flight cap (shares the NIM keypool)
  batchSize: number;     // rows fetched per cycle
  selectorSql: string;   // SELECT that returns rows to enrich — MUST select id
  buildBody(row): Record<string, unknown>;   // OpenAI chat body for one row
  attribution(row): Attribution;             // envelope attribution headers
  apply?: ApplyHook;        // POST result to a broker /apply/* endpoint
  resolveLocal?(row): ...;  // LOCAL resolver — skip gpumon dispatch entirely
}

Three config flavors

The worker (enrich-worker/src/index.ts) branches on which optional hooks are set:

1. Deterministic (resolveLocal) — no GPU, no network LLM. processRow calls cfg.resolveLocal(row) and POSTs the returned {endpoint, apply} to the broker. Return null to mark the row processed without writing. Examples: cik_resolve (flat-file CIK lookup), matters_build (group docs by case_number).

2. Network / non-LLM apply (apply + resolveLocal) — a deterministic call that hits an external service. Example: courtlistener_caption (case_number → caption via the CourtListener API).

3. LLM (pool + buildBody + parse) — dispatched to a gpumon-ingress pool. With apply set, the parsed result is POSTed to /apply/*; without it, the result is upserted into the enrichments table. Examples: scheme_classify, bluebook_citation, matter_title, caption_body_extract.

The selectorSql bind-param contract

runConfigCycle binds parameters based on the config flavor — get this wrong and the worker either re-processes every row forever or selects nothing:

The worker opens a fresh read-only corpus.db handle each loop iteration so it sees the current WAL view of broker writes (cross-container WAL gotcha — a long-lived handle goes stale).

Registered configs

From ENRICH_CONFIGS in enrich-worker/src/config.ts (execution order):

scheme_classify exists in both harnesses. The queue scheme-classify workflow is the primary path; the enrich-worker variant is the older config-loop implementation. Avoid running both against the same rows.

Adding a new enrichment config

1. Create enrich-worker/src/configs/<name>.ts exporting one EnrichConfig. Pick the flavor (resolveLocal / apply / classic LLM) and write a selectorSql that follows the bind-param contract above.
2. If it writes a dedicated column (not enrichments), add a broker /apply/<name> endpoint in broker/src/handlers.ts and whitelist the target columns.
3. Register it: import in enrich-worker/src/config.ts and add to the ENRICH_CONFIGS array in the desired execution slot (deterministic configs first — they are free and unblock LLM configs that depend on their output).
4. Test the bind count (enrich-worker/src/configs/apply-hook.test.ts asserts ?-placeholder counts) and run bun test in enrich-worker/.
5. Migrate any new columns into corpus.db via a broker migration (broker/src/migrations/).
6. Deploy — see below.

Deploy mechanics

enrich-worker runs as fraud_fraud-enrich-worker, pinned to node-eighteen, 1 replica, mem limit 1.5 GB (the CIK flat-file map is memory-heavy — do not lower). There is no Docker registry; after rebuild you must docker save | ssh ... docker load to node-eighteen, then force-update:

docker save fraud-processor:latest | ssh rooot@node-eighteen docker load
ssh rooot@node-eleven 'docker service update --force fraud_fraud-enrich-worker'

Confirm by grepping a marker string from the new code inside the running container — :latest is mutable and a stale image will look successful.

Throughput / gentleness

Per-config concurrency and batchSize are the throttle. LLM configs share the pool-postcrime-triples NIM keypool with the queue harness, so keep concurrency modest (history: concurrency ≤ 8 to avoid broker flooding). Deterministic configs (resolveLocal) cost nothing but broker write bandwidth. When the loop finds zero rows across all configs it sleeps IDLE_SLEEP_MS — so a fully-drained worker idles cheaply at 1 replica.