Why Vision LLMs Hallucinate Clearances on Mirrored CAD Blueprints (and How to Catch Spatial Clashes Programmatically)
Multimodal models routinely read callout text accurately while missing physical part collisions, mirrored chirality inversions, and clearance violations on engineering drawings. Here is why vision encoders fail spatial verification and how to test for it.
Standard multimodal benchmarks test whether a vision-language model (VLM) can transcribe a receipt, summarize a bar chart, or identify objects in natural photography. Enterprise engineering, logistics, and compliance pipelines face a harder problem: verifying whether a visual schematic or CAD blueprint physically agrees with the written specification inside the same workspace. When text and geometry conflict, frontier VLMs overwhelmingly trust text annotations and hallucinate spatial compliance.
1. Three Structural Failure Modes in Vision Encoders
Across mechanical assemblies, civil right-of-way diagrams, and operational routing maps, VLM failures cluster into three repeatable pathologies:
If a title block or revision note claims a 12.0 mm clearance while the drawn section view depicts direct flange-to-housing interference, the model parrots the 12.0 mm text label and signs off.
Patch-based ViT encoders are trained with horizontal-flip invariance priors on natural images—causing systematic blindness to mirrored port orientations, reversed helical threads, and swapped left/right datum axes.
Aggressive client-side resizing (512px–1024px) or lossy JPEG quantization blurs leader lines, ±0.05 tolerance symbols, and decimal points before inference even starts.
2. Eliminating Pre-Inference Artifacts: Lossless 2048px PNG Normalization
Before attributing a blueprint failure to a model, the auditing harness must guarantee that fine-line CAD callouts survived serialization. Encoding blueprints as lossless 2048px PNG payloads with explicit MIME signature detection (image/png, image/jpeg, image/webp) preserves sub-millimeter leader arrows, datum targets, and cross-hatching across OpenAI-compatible image_url blocks.
3. Decoupling Text & Vision Routing in Production Architectures
Many production AI stacks pair a fast, text-only reasoning model (target_model) with a dedicated multimodal model (vision_model) for image-bearing requests. If an evaluation harness forces a single model ID across both text and blueprint suites, teams either overpay on text-only workloads or trigger immediate 400 Bad Request errors when an image hits a text-only endpoint.
Role 1: target_model -> Dispatches 33 text & multi-file reasoning suites Role 2: vision_model -> Dispatches 8 image-bearing CAD & spatial hazard suites Role 3: judge_model -> Executes Phase 2 Per-Trap Chain-of-Verification rubric grading Optional Failover (heal_errors=True): If target_model rejects image payloads in Phase 1, automatically fail over blueprint suites to vision_model (600s timeout, lossless 2048px PNG) prior to Phase 2.
Testing spatial reasoning alongside multi-step tolerance math and regulatory precedence exposes the exact failure seam that breaks automated engineering review, insurance underwriting, and logistics dispatch pipelines in production.