Enrichment system redesign — proposal

For review before implementation. Goal: a small set of clearly-named services, a clean sub-task taxonomy with pools simplified to context-length tiers, a trigger-based scheduler, a one-file way to define a new task, data read/write on postcrimedb (.181/postcrime), and stats-mining + web-chart caching folded in as scheduled enrichment.

1. What exists today (the sprawl we're replacing)

Naming confusion found: entities-ner is just the entity-extraction stage of the entities workflow; it dispatches to a pool literally named pool-extract-triples (a misnomer — that pool is qwen3.6-27b used for all JSON extraction, not triples specifically). Pools today encode model + context-length + workload all at once (pool-postcrime-qwen-14b, pool-summary-medium, …), so a task's name, its pool's name, and the model are three different vocabularies.

2. Proposed services (4, clear roles)

   crawler/new docs ─┐
   task completions ─┤→  enrich-scheduler ──enqueue──▶  enrich-queue  ──▶  enrich-runner (N replicas)
   failures / skips ─┘     (the brain)        (RabbitMQ, one routing      (the muscle)
                                               key per task type)               │
                                                                                 ▼
                                                                          postcrimedb (.181)

fraud-publisher, publisher-regex, enrich-worker, processor-triples, and the enrich-*.yml one-shots all collapse into the three services above. stats-miner becomes a set of tasks (§6).

3. Sub-task taxonomy (clean names) + pool simplification

Pools → context-length tiers only. A task declares the ctx tier it needs; the pool is just a ctx-routed target that picks/falls-back across models. Model choice leaves the task vocabulary.

Tasks (clean, purpose-named; the runner picks the pool from kind+ctx):

entities-ner → extract-entities (it's entity extraction; "ner" + "extract-triples" pool were the confusing bits).

4. Defining a new task — one file

Extend WorkflowDef into a declarative EnrichmentTask (one file in enrich-tasks/), so adding a task is: write the file, drop it in the dir (auto-registered). The scheduler reads triggers + selects; the runner reads run + writes; both read reportsAs.

export const task: EnrichmentTask = {
  name: "summarize",
  kind: "llm",                 // regex | llm | vision | embed | http | sql
  ctx: "m",                    // → pool-ctx-m (only for kind:llm)
  reportsAs: "summarize",      // /progress + gpumon workflow_stage label
  dependsOn: ["recover-ocr"],  // re-runs when an upstream task changes this doc (DAG)
  triggers: ["new-doc", "backfill", "on-dep-change", "on-failure"],
  // WHAT needs it (Postgres now, not corpus.db):
  selects: (sql, scope) => sql`SELECT id, body, ocr_text FROM postcrime.documents
                               WHERE summary_combined IS NULL ${scope}`,
  // DO it (dispatch helper injects the pool + gpumon attribution):
  run: async (doc, llm) => ({ summary_combined: await llm.chat(doc.body) }),
  // WRITE it (postcrimedb single-writer path):
  writes: { table: "postcrime.documents", key: "id", cols: ["summary_combined"] },
};

5. Scheduling model (triggers)

enrich-scheduler enqueues per task based on declared triggers:

A task_state(task, doc_id) → status, attempts, last_ts, watermark table (in postcrimedb) replaces the scattered workflow_runs/queued_* columns and makes "what's pending/skipped/failed" one query.

6. Stats mining + web-chart caching as enrichment

Each of the 42 stats-miner panels becomes a stats-<panel> task with kind: "sql" and triggers: ["on-data-change", "schedule"]. When enrichment changes the underlying rows (or on a cadence), the scheduler debounce-fires the affected panels; the runner materializes the panel into stats.panels and writes the rendered chart payload to SeaweedFS …/dashboard/stats/cache/. Result: stats stay fresh automatically as enrichment runs, no separate weekly miner.

7. Data read/write → postcrimedb

• Read: selects() queries postcrimedb (.181, schema postcrime) — the canonical store — not SQLite corpus.db.
• Write: a single postcrime-writer path (the runner, or a thin writer the runner calls) owns writes to postcrimedb with upsert discipline, replacing the broker /apply/* → SQLite + dual-write. corpus.db/broker can stay as a transitional fallback during a soak, then retire.

8. Migration path (incremental, low-risk)

1. Land EnrichmentTask + the enrich-tasks/ registry; port existing workflows into it (mechanical).
2. Build enrich-runner (generalize enricher-v2) reading PG, emitting per-task progress; run it alongside the old enricher on a few tasks; verify parity on /progress.
3. Build enrich-scheduler (repurpose enrich-worker) with triggers; cut producers over to it; retire fraud-publisher/publisher-regex/processor-triples/one-shots.
4. Fold stats panels in as stats-* tasks; retire stats-miner.
5. Flip reads/writes fully to postcrimedb; retire the SQLite path.

Open decisions for you

• Pools: confirm the ctx-tier simplification (pool-ctx-s/m/l + vision + embed), vs. keeping workload-named pools.
• PG writer: runner writes postcrimedb directly (single-writer per task) vs. keep a thin broker-style writer service in front.
• Triggers: Postgres LISTEN/NOTIFY for new-doc/dep-change (event-driven) vs. watermark polling (simpler, ~few-sec latency).
• Scope: all-at-once remake vs. the incremental §8 path (recommended).