Prow Labs
Launch Red-Team →
ENGINEERING TEARDOWNS // PROOF-OF-WORK NOTES

Deterministic Failure Forensics for Production LLM & Vision APIs.

Architectural notes and failure-mode analyses from building Prow // Red-Team—covering reasoning-token truncation, multimodal CAD spatial clashes, statutory precedence inversions, and null-safe rubric grading. Zero private suite ground-truth spoilers.

TD-001 // Evaluation Harness Engineering 2026-11-22 · 7 min read

Handling Reasoning-Token Exhaustion & Empty Content Payloads in OpenAI-Compatible Audit Pipelines

Reasoning-enabled LLMs share completion token budgets between hidden chain-of-thought tokens and visible output. Low default token caps trigger silent HTTP 200 OK empty-string responses that naive evaluation scripts misclassify as model refusals.

/v1/chat/completionsfinish_reason: length16,384 Token CeilingEmpty-Reasoning Guard
TD-002 // Multi-Modal Vision Forensics 2026-11-22 · 8 min read

Why Vision LLMs Hallucinate Clearances on Mirrored CAD Blueprints (and How to Catch Spatial Clashes Programmatically)

Multimodal models routinely read callout text accurately while missing physical part collisions, mirrored chirality inversions, and clearance violations on engineering drawings. Here is why vision encoders fail spatial verification and how to test for it.

Multi-Modal Vision2048px Lossless PNGSpatial Clash Detection3-Role Vision Failover
TD-003 // Sycophancy & Policy Precedence 2026-11-22 · 8 min read

Beyond Jailbreaks: Measuring Executive-Pressure Sycophancy and Statutory Precedence Failures in Enterprise Agents

Most enterprise AI incidents are not caused by exotic cipher jailbreaks—they happen when an assistant obeys an urgent manager memo or a misleading 'STATUS: PASS' summary header over a binding federal or safety regulation.

Sycophancy & DeceptionPolicy PrecedenceMulti-File RAGChain-of-Verification