Shell Heuristics · application docs

publisher · enricher-v2 · corpus

Investigation workflow

How to use what's running today to investigate a real target. Worked example: "I'm looking at Cane Clark LLP and want every shell they ever filed for, with the death-spiral promissory notes flagged."

Throughout: node-eighteen is the workload host, node-eleven is the swarm manager.

Worked example — Cane Clark LLP

Step 1 — Find Cane in the opinion-letter signal

The smallest, highest-quality NDJSON is opinion_letter_presence (1.5 MB). Grep it for the firm name.

ssh rooot@node-eighteen \
  "grep -l -i 'cane.clark\\|cane clark' \
     /mnt/oink/docker/edgar-fraud-scan/data/flagged/opinion_letter_presence.ndjson"

(Use jq if you need structured output; ctx_execute_file is fine for large NDJSONs.)

Output: a list of {cik, accession, form, evidence} rows where the opinion letter named Cane.

Step 2 — Pivot from each CIK into the other three NDJSONs

For each cik from step 1, grep:

  • edgarizer_fingerprint.ndjson — was the same shop the filing agent?
  • reg_s_issuance.ndjson — did they issue under Reg S?
  • promissory_note_clauses.ndjson — do they have convertible / death-spiral notes?

GAP: there is no tool that joins these four NDJSONs by CIK. Today this is a grep | jq flow. TODO: a one-liner script scripts/cik-signal-join.ts that takes a CIK and returns hits across all four. Would live alongside cross-corpus-accomplices (Phase F scaffolded).

Step 3 — Pivot from CIK into corpus.db

Cross-reference CIKs from step 2 against corpus.db to see if DOJ/SEC ever filed anything about them.

ssh rooot@node-eighteen 'sqlite3 -readonly /var/lib/fraud-data/corpus.db \
  "SELECT d.id, d.kind, d.title, d.published_at, d.case_number
   FROM documents d
   WHERE d.body LIKE \"%<CIK>%\"
      OR d.title LIKE \"%<TICKER>%\"
   ORDER BY d.published_at DESC LIMIT 50;"'

GAP: documents doesn't have a cik column. We rely on body LIKE which is slow on 99 GB of text. TODO: add a documents.cik_json column populated by an extract-cik workflow. Until then, FTS5 search via /api/search?q= is faster.

Alternative: use search-ui at https://postcrime.atsignhandle.xyz/?q=<cik> — same DB, indexed.

Step 4 — Look at extracted facts

For each candidate documents.id:

ssh rooot@node-eighteen 'sqlite3 -readonly /var/lib/fraud-data/corpus.db \
  "SELECT cf.* FROM case_facts cf WHERE cf.document_id = <id>;"'

Plus: documents.summary_paragraph, documents.scheme, documents.scheme_confidence, and the JSON in documents.reference_filing_json.

Step 5 — Cross-check ciks-banana curated roster

jq '.[] | select(.name | test("Cane"; "i"))' \
  *********/ciks-banana/.vercel/output/static/ciks/roster.json

That static JSON is the 672-entity Cane network curated catalog. It maps name → cik → relationships.

GAP: ciks-banana/cane-network/ referenced in the brief does not exist — the equivalent data is in .vercel/output/static/ciks/. TODO: confirm with tankbottoms.eth which set is canonical going forward.

Starting points

From a CIK

  1. Hit all four NDJSONs (grep | jq) for "cik":"<padded-cik>".
  2. Cross-check ciks-banana/.vercel/output/static/ciks/roster.json.
  3. Search corpus.db for any DOJ/SEC document mentioning the ticker / company name.
  4. If new evidence exists, kick a case-facts + scheme-classify pass for those doc IDs.

From a person (e.g. Travis Stephen Cane)

  1. Look in ciks-banana/.vercel/output/static/ciks/roster.json for an entity entry.
  2. Search corpus.db via /api/search?q="Travis Stephen Cane" for press-release / litigation mentions.
  3. Pull case_facts rows for any matched document_id.
  4. GAP: no /entity or /person page yet — relies on raw search. See frontend.md Goals.

From a filing pattern (e.g. "death-spiral note")

  1. promissory_note_clauses.ndjson filtered by score=1 is the seed set.
  2. Group by evidence[].field='filing_agent_cik_prefix' to find common agents.
  3. Pivot via Step 2 above into edgarizer_fingerprint to find the rest of the agent's roster.
  4. Then back into corpus.db to see who got caught.

From a date window

  1. /stats calendar heatmap (/stats/api/calendar) gives a per-day filing volume by source.
  2. sqlite3 corpus.db "SELECT id, kind, title FROM documents WHERE published_at BETWEEN ? AND ?;".
  3. For each row, look at documents.scheme and case_facts.first_act_date to find clusters.
  4. GAP: no proper timeline view — see frontend.md Goals.

Operational recipes

Republish a stuck workflow row

If a document is stuck queued_workflow != NULL for >15 min:

  • The publisher's MAX_INFLIGHT_MIN=15 already re-publishes (see services/fraud-publisher/index.ts:isPublishable).
  • If that fails: clear by hand via the broker:
ssh rooot@node-eleven 'docker exec -i $(docker ps -q -f name=fraud-db-broker) \
  curl -sX POST localhost:3100/update \
    -H "Content-Type: application/json" \
    -d "{\"id\":<doc_id>,\"queued_at\":null,\"queued_workflow\":null,\"envelope_id\":null}"'

Drain the DLQ

# WARNING: stops the enricher first; the smoke race-condition (Q11) applies.
ssh rooot@node-eleven 'docker service scale fraud_fraud-enricher-v2=0'
# inspect / requeue via the federation-queue-client CLI or rabbitmqctl
ssh rooot@node-eleven 'docker service scale fraud_fraud-enricher-v2=1'

Force a single workflow for one doc

ssh rooot@node-eleven 'docker service update --env-add PUBLISHER_WORKFLOWS=case-facts \
  fraud_fraud-publisher'

Or run the legacy processor/scripts/enrich.ts <workflow> --doc-id=<id> from inside a swarm task.

Check what's in flight

ssh rooot@node-eighteen 'sqlite3 -readonly /var/lib/fraud-data/corpus.db \
  "SELECT queued_workflow, COUNT(*) FROM documents
   WHERE queued_at IS NOT NULL GROUP BY queued_workflow;"'

Check per-workflow progress

ssh rooot@node-eighteen 'sqlite3 -readonly /var/lib/fraud-data/corpus.db \
  "SELECT workflow, status, COUNT(*) FROM workflow_runs
   GROUP BY workflow, status ORDER BY workflow;"'

(Reproduces what processor/scripts/status.ts reports — see commit 7564e56.)

Trigger a fresh EDGAR fingerprint pass

GAP: postcrime-heuristics doesn't own this. The edgar-fraud-scan swarm services (edgar-agent-bypass, edgar-rm-expander, edgar-clusterer, ...) are managed separately on node-eleven. TODO: document the trigger command. Until then, ping the edgar-cik-cli maintainer to refresh flagged/*.ndjson.

Gaps to prioritise for investigation speed

  1. GAP: CIK column on documents — body LIKE is slow. Add documents.cik_json + extractor workflow.
  2. GAP: Per-CIK NDJSON join script — scripts/cik-signal-join.ts.
  3. GAP: /cik/[cik] page — would replace 4 grep commands with 1 URL.
  4. GAP: Curated-vs-flagged reconcile — bridge ciks-banana/roster.json (curated) with flagged/*.ndjson (raw). Some Cane scheme entities never get flagged because their fingerprint is below threshold.
  5. GAP: Evidence-pack export — produce a single ZIP / IPFS bundle for one CIK or one case.
  6. GAP: DLQ + queue depth dashboard — currently invisible to anyone outside swarm shell.
rendered from docs/application/investigation-workflow.md  ·  v0 reference: gpumon-writer hand-coded catalog ↗ (the layout that inspired this style guide)