LLM routing — why enrichment runs on LAN GPUs, not the NIMs

How every enrichment / anomaly LLM call is dispatched to a backend, and why the NVIDIA NIM endpoints carry only a trickle of the traffic even though the pools are configured "NIM-primary". Debugged 2026-06-02 against the live fleet.

Update — 2026-06-02 (later), deployed. Two changes landed after the diagnosis below (see fraud-heuristics/docs/NIM-THROUGHPUT-ANALYSIS-2026-06-02.md and gpumon commit cc35c5e):

1. Lever 1 — enrich tasks repointed off pool-postcrime-*. The heavy LLM tasks now send pool-qwen-large / pool-qwen-14b / pool-gemma directly (enrich-tasks/src/tasks/*), and runner concurrency was raised (PREFETCH 4→6, runner replicas 3→5). Caveat: pool-qwen-large is the LiteLLM qwen3-next-80b path, which dead-airs at NVIDIA (≈15 s) and falls back to slow LAN — so this alone did not lift NIM share. The pending follow-up (Lever 2b) is to repoint those tasks onto the now-fixed pool-postcrime-triples (below) and off the dead qwen3-next path.
2. Lever 2 — pool-postcrime-* are no longer collapsed onto the shared anchor. They now use a dedicated NIM-only &postcrime_nim_text anchor (gpt-oss-120b + maverick + nemotron-nano-vl, no LAN, no vision-in-text waste) with a safe 2-card LAN breaker-fallback (spark-1 gemma + nv-1 — explicitly not the 33 s spark-2 Q6). Deployed as Docker config gpumon_gpumon_pools2_v51 on both gpumon-ingress and gpumon-scheduler (the scheduler does the actual routing when SCHEDULER_URL is set — updating only the ingress is not enough).

Separately, the gpumon dashboard SSE (queues + LLM stream) had stopped surfacing because gpumon-writer was OOM-crash-looping on its unpruned 3.8 GB telemetry DB at a 512 MB limit. Fixed: limit 512M→768M + pruned gpu_samples_5s 1.75M→267K (gpumon/scripts/prune-telemetry.js; needs a retention cron).

The path a call takes

enrich-runner / publisher / anomaly-classify
   → POST gpumon-ingress :4010  (x-gpumon-pool: pool-postcrime-*)
      → SCHEDULER_URL set → hands the whole request to gpumon-scheduler
         → gpumon.dispatch queue → per-key/per-backend admission control
            → NIM (build.nvidia.com)  OR  LAN GPU (spark-1/2, nv-1/2/3/4)

gpumon-ingress does not pick the backend when SCHEDULER_URL is set (it is): it forwards to gpumon-scheduler, which loads pools2.yaml, applies pacing, and dispatches. The in-ingress weighted pick is only the fallback when the scheduler 400s on an unknown pool.

Historically (pre-2026-06-02-later) all pool-postcrime-* aliases (triples, gemma, qwen-14b, qwen-large) resolved to the same tier — _primary_nim_default (NIM weight ~50: maverick, nemotron-nano-vl, gpt-oss-120b)

• 4 LAN GPUs (weight ~40) — so naming tasks onto different postcrime pools bought zero NIM isolation. As of config v51 they instead use the NIM-only _postcrime_nim_text anchor with a 2-card LAN breaker-fallback (see the Update banner above). The analysis of why the LAN GPUs dominated still applies to any pool still on _primary_nim_default (e.g. pool-extract-triples, edgar-*).

Why NIM only gets ~4 % in practice

NIM is throughput-capped by its own latency. Scheduler env:

With 1 in-flight/key and 70 s/call, each key sustains ~1 call/70 s, so 14 keys give a **12 RPM aggregate NIM ceiling**. Enrichment runs at hundreds of RPM, so the scheduler admits ~12 RPM to NIM and sheds the rest to the LAN GPUs by design — that is the intended "NIM = premium trickle, LAN = workhorse" split.

Live confirmation (2026-06-02): of the last 500 LLM calls, 19 hit a NIM (rpm_used ≈ 11, zero 429s, rate_limit_5m = 0); the other 481 ran on LAN GPUs. Working as designed.

Secondary issue — only gpt-oss-120b is serving among the NIMs

maverick and nemotron-nano-vl (higher-weighted) showed zero recent traffic. They share the same 14 keys, so it is not a key-resolution problem. The likely cause is the upstream flakiness pools2.yaml already documents (llama-3.3-70b "dead-airs", qwen3-next / ministral removed for dead-air / 400s): those endpoints return 5xx/dead-air → the ingress circuit breaker trips OPEN (15 s → 120 s backoff) → they are excluded → only gpt-oss stays in rotation. five_xx_1h = 24 is consistent with this.

Confirm with: docker service logs gpumon_gpumon-scheduler — look for NIM backend … api_keys resolved empty — backend disabled (key issue, unlikely here) and breaker-OPEN / 5xx warnings for maverick / nemotron.

Levers (if you want more NIM share)

1. More keys. Throughput is latency-bound; the clean scaling lever is adding NIM API keys beyond 14 (_nim_keys in pools2.yaml). Each key ≈ +0.85 RPM.
2. NIM_PARALLEL_PER_KEY=2 ≈ doubles NIM throughput but risks per-key lockout (NVIDIA rejects concurrent calls on a key) — test before committing.
3. Fix or drop the flaky NIMs. Re-probe maverick / nemotron-nano-vl on build.nvidia.com; if dead-airing, drop them from _primary_nim_default so breaker churn stops and gpt-oss + healthy models keep the share.
4. Accept the split. NIM cannot out-throughput the LAN fleet here; reserve it for large-context / quality-sensitive passes (summaries, scheme classify), and let the LAN GPUs carry the bulk regex/triples volume.

The large-prompt → haiku fast path only triggers above 16 K tokens (≈ 64 K chars); triples prompts are ~4 K chars, so it is not a factor in the NIM under-use here.

Live audit (2026-06-02) — why NIM is stuck at ~13 RPM, not 20–40

Direct NVIDIA probe (bun scripts/nim-model-audit.ts in gpu-federation-monitor) of the in-use NIM models:

So the top-weighted NIM is dead, gpt-oss alone absorbs the NIM share and 429-saturates at ~13 RPM, and the healthy nemotron-nano (weight 20) gets zero because its breaker is stuck OPEN. A pool throughput bench (bun scripts/pool-prompt-bench.ts --ingress http://192.168.1.211:4010 --pools pool-postcrime-triples --sizes 4k --reps 8 --concurrency 8) returns 8/8 → 200, p50 ≈ 3.2 s — the ingress path is healthy; the limiter is which NIM the scheduler will route to.

How to reach a steady 20–40 RPM (in order)

1. Restart the scheduler — the fastest, lowest-risk fix. docker service update --force gpumon_gpumon-scheduler resets every backend breaker, so the healthy NIMs (nemotron-nano, gpt-oss, and nemotron-mini on short pools) rejoin rotation instead of gpt-oss carrying it solo. Dead maverick re-trips on its own; the healthy ones share the load.
2. If maverick stays dead-airing, drop the two *nim_llama4_maverick entries from _primary_nim_default (and swap the large-ctx refs in _primary_nim_text_largein to *nim_gpt_oss) so ~40 % of NIM picks stop hitting a 20 s timeout + breaker churn. Re-probe first — NVIDIA endpoints flap and it may have recovered.
3. Pacing headroom. NIM_MIN_INTERVAL_MS=6000 caps each key at 10 RPM and NIM_PARALLEL_PER_KEY=1. With healthy fast models (nemotron-* at ~170–260 ms) the keys are work-starved, not interval-bound, so step 1 alone should lift aggregate RPM. Only raise NIM_PARALLEL_PER_KEY=2 if still short (it risks per-key lockout — bench it).
4. Verify with the dashboard NIM cards (https://gpu.atsignhandle.xyz/) and re-run pool-prompt-bench — watch model_mix shift onto multiple NIMs and sum rpm_used climb past 20.

These are gpu-federation-monitor changes (services/ingress/pools2.yaml + scheduler env) that require a scheduler redeploy/restart to take effect. Throughput tooling: scripts/nim-model-audit.ts (liveness), scripts/pool-prompt-bench.ts (per-pool NIM/LAN mix through the ingress), scripts/nim-capacity-bench.ts (raw per-model NIM ceiling).

See also: enrichment passes (what each pass does) · anomalies pipeline (anomaly-classify shares these pools).