Prow Labs
Launch Red-Team →
TD-001 // Evaluation Harness Engineering 2026-11-22 · 7 min read

Handling Reasoning-Token Exhaustion & Empty Content Payloads in OpenAI-Compatible Audit Pipelines

Reasoning-enabled LLMs share completion token budgets between hidden chain-of-thought tokens and visible output. Low default token caps trigger silent HTTP 200 OK empty-string responses that naive evaluation scripts misclassify as model refusals.

/v1/chat/completionsfinish_reason: length16,384 Token CeilingEmpty-Reasoning Guard

When benchmarking standard instruction-tuned LLMs over /v1/chat/completions, setting a conservative max_tokens cap (such as 2048 or 4096) is routine defensive practice. Against modern reasoning models—including OpenAI o-series, gpt-oss-20b, and DeepSeek-R1 derivatives—that exact configuration creates a silent, high-severity evaluation bug: HTTP 200 OK responses carrying an empty string ("") or null in choices[0].message.content.

1. Anatomy of a Silent HTTP 200 Reasoning Truncation

Across OpenAI-compatible inference gateways (OpenAI, Groq, vLLM, Fireworks, and custom enterprise proxies), internal reasoning tokens are billed and metered inside the completion budget prior to emitting the first visible character of message.content. On dense multi-document workspaces—where a model must cross-check a .xlsx ledger, a statutory .pdf clause, and a sequence of .eml overrides—reasoning traces routinely consume 3,500 to 9,000+ internal tokens before synthesis begins.

Observed Wire Payload // HTTP 200 OK With Exhausted Reasoning Budget
{
  "id": "chatcmpl-9f82a7c1e4",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": ""
      },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 11420,
    "completion_tokens": 4096,
    "total_tokens": 15516,
    "completion_tokens_details": {
      "reasoning_tokens": 4096
    }
  }
}

Notice why naive evaluation harnesses corrupt their own scorecards here:

  • Transport layer reports healthy completion: Status code is 200, so retry loops watching for 429 or 5xx never fire.
  • False PASS on negative-constraint traps: If a deterministic rule checks that the model did not leak a restricted token or did not endorse an unsafe override, an empty string trivially satisfies naive negative regex checks unless guarded.
  • False FAIL attributed to model capability: On rubric-graded suites, the Judge LLM receives an empty target transcript and scores a reasoning failure when the actual root cause was harness misconfiguration.

2. Enforcing a 16,384 Output-Token Ceiling + Empty-Reasoning Interlock

A production-grade red-teaming pipeline must separate infrastructure/budget truncation from behavioral model failure. Inside deterministic injection adapters, two invariants eliminate silent empty payloads:

Defensive Adapter Pattern // Completion Ceiling & Empty-Content Guard
REASONING_COMPLETION_CEILING = 16384

def extract_verified_completion(resp_json: dict) -> tuple[str, str | None]:
    choice = (resp_json.get("choices") or [{}])[0]
    msg = choice.get("message") or {}
    content = (msg.get("content") or "").strip()
    finish_reason = choice.get("finish_reason")
    usage = resp_json.get("usage") or {}
    reasoning_toks = (
        (usage.get("completion_tokens_details") or {}).get("reasoning_tokens")
        or (1 if msg.get("reasoning") or msg.get("reasoning_content") else 0)
    )

    if not content:
        if finish_reason == "length" or reasoning_toks:
            return "", (
                f"REASONING_TOKEN_EXHAUSTION: finish_reason={finish_reason!r}, "
                f"completion_tokens={usage.get('completion_tokens')}, "
                f"message.content returned empty."
            )
        return "", f"EMPTY_COMPLETION_PAYLOAD: finish_reason={finish_reason!r}"

    return content, None

3. Distinguishing Terse Abstentions From Truncated Outputs

Once empty truncations are isolated as explicit execution errors (preventing silent PASS inflation), the opposite edge case emerges on hallucination probes: concise, high-discipline abstentions. Well-aligned models increasingly respond to fabricated entity or citation traps with a crisp one-sentence refusal (for example, "Cannot verify: no primary record exists for that docket or statute.") rather than a three-paragraph disclaimer.

Deterministic hallucination graders must verify two conditions simultaneously: (1) finish_reason != "length" with non-empty content, and (2) explicit non-existence / unverifiability semantics without fabricated details—ensuring concise refusals earn a clean PASS while truncated reasoning traces fail closed as explicit ERROR telemetry.

More Prow Labs Engineering Teardowns

View All →