Enrichment worklist — 2026-06-09
DB: .181/postcrimedb schema postcrime · 33,801 docs · entities 113,534.
Path: crawler → db-broker (PG write) → publisher → enrich.* queues → enrich-scheduler (dispatch) → enrich-runner ×5 (gpumon.dispatch poll) → cells. All on node-eighteen.
Live coverage (% of 33,801 — column COUNT, not ledger)
| Field | Cov | Note |
|---|---|---|
| body / title | 100% | complete |
| ocr_text | 94.4% | 1,111 [OCR_UNRECOVERABLE] + 1,230 pdf-kind no-ocr |
| summary_combined | 95.4% | 1,548 tail |
| scheme / keywords | 99.0 / 99.8% | 328 / 82 tail |
| summary_llm3_qwen14b | 33.8% | hi-tier 14b — partial; design decision needed |
| statute_refs | 60% · court_code 54% · outcome 47% · money 59% | mostly litigation-gated |
| primary_entity | 43% · reference_filing 52% | |
| bluebook_citation | 16% (34% of applicable) | needs court+parties first |
| cik 6.3% · parties 8.6% · case_number 23% | case-gated by design | |
| title_cased | 1.7% | orphan/unrun pass — decide backfill vs drop |
Ledger (task_state) undercounts (legacy processor wrote columns w/o rows) — trust COUNTs. entity_mentions table missing/renamed → verify mention-coverage table name.
Dependency chain (existing data)
OCR text is the root. ocr_text → summary → scheme/keywords → {statute,court,outcome,money,entities} → {references,parties,cik} → bluebook → triples. Re-OCR'd docs MUST re-flow downstream.
Work to create — ordered
- DEPLOY GAP (P0, blocks all prod fixes). Push
51b7ce3(ids-bypass) + redeployenrich-runner+enrich-schedulercarrying51b7ce3+24ea283(junk write-guard). Their deployed checkout lacks both → entity-junk regen + recover-ocr broken prod-side.docker save|loadto node-eighteen (pinned), thenservice update --force; grep marker in container to confirm. - Migrate
enrich-tasks/src/llm.ts:148off HTTP Ingress (/v1/chat/completions, deprecated) →dispatch-client.tspoll mode; flipenrich-worker/src/index.tsdefaultreply→poll(durability). Keep ≤8 LLM in-flight. Lib already present inenrich-worker+processor. - Finish OCR recovery (unblocks downstream on 1,111 docs). (a) 539 image-only (
pdf_pathpresent) → re-runrecover-ocrids-scope on a pool-ocr-reachable host (prod runner/spark, NOT mepstudio). (b) 572 url-only → new derive-pdf-path-from-url pass, then refetch+recover. - Backfill text-derived tails (after #3). summary 1,548 + scheme 328; re-derive money/statute/court/outcome/entities on newly-OCR'd docs (idempotent re-emit via
scripts/reconcile-enrich-queue.ts). - Two design decisions. (a)
title_cased1.7% — live target or dead column? (b)summary_llm3_qwen14b33.8% — enrich-all or opt-in tier. Resolve before backfilling either. - bluebook + case-gated — last; depend on court_code/parties (Phase B: B2 CIK-classify, B4 scheme scoring, B5 CourtListener edges).
New-data workflow (forward pipeline)
Each new doc must auto-traverse #3→#6. Verify: publisher resequences raw→tiered enrich.*; scanned PDFs submit to gpumon-ocr-api with reaper writing ocr_text; scheduler startupReconcile() re-emits PG-queued on empty queues. Every worker/cron MUST POST https://postcrime.atsignhandle.xyz/api/progress (project:postcrime, stable workflow name) — a job that doesn't report is incomplete.