When does evidence justify an AI answer?

01 / The evidence decision

A correct calculation can still support an unjustified conclusion.

Imagine purchases rise from 20 to 30. Has the purchase rate improved? We need the visitor counts: 20 purchases from 1,000 visits is 2%; 30 from 1,000 is 3%, but 30 from 1,500 is still 2%. The arithmetic is easy. Knowing whether the evidence licenses the comparison is the harder part.

We tested a complete evidence policy: withhold a conclusion while decisive support is missing, qualify it when only a weak clue arrives, recover the supported conclusion and action when the missing evidence appears, and ignore similarly formatted but irrelevant information. Always refusing cannot pass because it fails recovery. Cautious wording cannot pass if the selected claim stance or action still commits to an unsupported conclusion.

These are matched task versions, not a single conversation accumulating clues. A separate comparison joins two different matched worlds: an armed case contains the statistical defect; its disarmed control makes the tempting conclusion warranted. A model can respond to that surface difference without following the complete four-condition policy.

02 / Models, tasks and populations

We tested related questions in two experiments.

We generated tasks from explicit data-generating processes, storing private target values and labels for what the visible evidence could identify. Some insufficient-evidence cases have analytic or constructive certificates, such as two observationally equivalent worlds with different target values. The design covers ten statistical mechanisms; they are not all causal-identification problems.

We created four task splits from fixed random seeds: development, public test, private evaluation and a pre-generated rotation. Each has 306 task bundles: 122 structural base views, 172 linked ladder views and 12 cue views, for 1,224 bundles overall. The same structural templates recur across splits, so these are not 1,224 independent problem types. Related versions of a base stay in one split, and we keep model-visible material, evaluation metadata and private truth separate.

The sealed model roster was DeepSeek V3.2, Kimi K2.5, Qwen3-32B, Qwen3-VL-235B-A22B, Llama 4 Maverick, GLM-4.7, GLM-4.7 Flash, Nemotron 3 Super 120B, MiniMax M2, MiniMax M2.1 and MiniMax M2.5. GLM-5 contributed a separate 612-unit exploratory lane and was excluded from the sealed aggregate. The later campaign used the full roster below.

We queried the later model group through provider APIs during one run window. Each result measures the requested model together with its provider, adapter, output budget and response interface; it is not an intrinsic ranking of the model family. We used one natural no-tool prompt condition. Adapters differed by provider; output caps were 8,192 or 16,384 tokens and were fixed after smoke tests. We account for all 612 assignments for every available model, including provider failures and unusable answers. We had not used the rotation split before this campaign; it cannot now serve as an untouched test set.

We verified 47 one-to-one armed/disarmed links across all ten families, but not every armed task has a matched control. The full ladder has 54 anchors from six families, each queried twice for 108 joint trials per model with estimable results. Ten anchors from data leakage and Simpson reversal form a separate 20-trial short ladder. Multiple comparisons and shared seasonality contribute no full-ladder anchors and remain NA for that analysis.

03 / Scoring and uncertainty

An unsupported action and an unusable answer are different failures.

Models returned task-declared decision and action options together with claim-level stances: support, reject, uncertain or not applicable. A deterministic scorer read those structured fields against generator-defined targets. A hedged rationale could not override a definitive claim or cause-specific action. The main endpoints did not use an AI judge.

We report these endpoints separately. Delivery asks whether the interface produced a valid response; it does not establish that the statistical task was solved. The narrow failure composite counts both direct unsupported commitment and unusable output. A model with missing answers therefore cannot appear safe simply because we cannot observe its actions.

We resample the underlying base or verified design link, because repeated queries are not independent observations. The later pairwise comparisons share bootstrap resamples within each endpoint and use Holm adjustment. Each endpoint’s adjustment family includes all 276 planned model pairs; ineligible or unavailable comparisons remain NA. The prespecified roster-class contrasts use their own three-comparison Holm family. We call a pair separated only when it passes the specified adjusted criterion. Models can belong to overlapping groups when we cannot separate them statistically; those groups are neither ranks nor proof of equivalence.

We fixed the task set, response schema and scoring rules before seeing outcomes. We did not follow the written output-budget escalation rule exactly: two runs outside the watchlist received larger budgets after 40% no-response, one at exactly 20% despite a greater-than-20% threshold, and MiniMax M2.5 was labelled escalated at a 4% trigger while already configured at 16,384 tokens. These choices limit a clean comparison of the models alone.

04 / Complete evidence-policy results

The best observed model passed all four conditions 17 times out of 108.

GPT-5.6-sol reached 15.74% joint success. Eight models scored zero. We could not statistically separate any of the 21 models with estimable results: 0 of 210 eligible model-pair comparisons met the adjusted criterion. We could not establish a winner for complete evidence policy.

The fail-closed full-policy rate averages the 54 declared bases equally and retains both assigned repetitions. The complete-case sensitivity averages over represented bases after omitting incomplete trials; its number of complete trials is shown for coverage and is not necessarily the denominator of a pooled success rate. The largest complete-case point estimate was GLM-5.2 at 18.75%, over 32 represented bases and 50 complete trials. The short ladder is a distinct E0 → E2 population; its maximum was 15.00%.

Gemini 3.1 Pro and MiniMax M2.5 completed ordinary task assignments, but their results contain no ladder-rung rows, so full-policy success is NA. Llama 4 Maverick was unavailable. We did not produce the planned pairwise-complete or relaxed-envelope ladder analyses, and separate model summaries cannot reconstruct the missing joint outcomes. We also could not estimate the earlier sealed ladder: that workflow did not assemble the linked rung records and their shared identifiers.

05 / Matched-surface response

Outputs shifted with the surface; that does not establish internal recognition.

All 23 available models had a descriptive armed-minus-disarmed estimate on their observed verified links, ranging from −28.17 to 0.00 percentage points. Only Claude Opus 4.8, GPT-5.6-sol and Gemini 3.6 Flash covered all 47 links and met the coverage requirement for adjusted pairwise comparisons. GPT-5.6-sol had the largest eligible shift, −25.49 points. The only pairwise difference that remained after adjustment favoured sol over Gemini Flash.

The largest descriptive shift was GPT-5.6-terra’s −28.17 points on 46/47 links, with sharp bounds [−29.69, −25.44]. Eleven of 23 model bounds remained below zero. These are missing-link bounds, not confidence intervals; they bracket unobserved link differences rather than quantify sampling uncertainty.

The surface comparison produced two overlapping groups: {GPT-5.6-sol, Claude Opus 4.8} and {Claude Opus 4.8, Gemini 3.6 Flash}. A response shift could reflect statistical understanding, a correlated visual or textual cue, or another change in the rendered problem. Our cue check below means we cannot attribute it cleanly to one mechanism.

06 / Response delivery

Usable delivery separated models much more than complete policy did.

Strict schema-valid first-attempt delivery ranged from 20/612 (3.27%) for GLM-5 to 578/612 (94.44%) for GPT-5.6-sol. We found 11 overlapping model groups, and 189 of 253 eligible model-pair differences remained after adjustment for multiple comparisons. This is interface compliance under the recorded configuration, not a ranking of statistical-task accuracy.

We checked whether each provider request completed, whether a response arrived, whether its structure was valid and whether we could classify its decision. The gaps show where the workflow lost usable answers. Llama 4 Maverick remains NA; we do not count unavailable-model requests as failures in the denominator for available models.

We had prespecified a caveat for GLM-5: an earlier baseline produced structured output under the same contract 33.9% of the time, versus 99.2% for its peer runs. The later 3.27% result still measures the model together with this protocol. Simple deterministic stubs can produce valid structured output by construction, but that alone says nothing about evidence policy.

07 / Narrow avoidance check

The same high failure score can hide two very different remedies.

The exploratory E-R4 check used only eight E0 units per model from four shared-seasonality bases. Three models scored 75%, two scored 87.5% and 18 of 23 scored 100% adverse failure. These small one-family samples cannot establish one model’s superiority or statistically separated groups.

We used these analytic controls to check that the scorer assigns the expected results to known policies. They do not provide a human baseline or validate the whole construct. Always abstaining can pass this E0-only check while failing the full policy’s E2 recovery requirement.

08 / Prespecified roster-class comparison

The two model groups differed on policy and delivery on the same tasks.

We compared the prespecified contemporary and earlier model groups during the same run window; the roster table shows which models belong to each group. We queried all of them on the same task set, with model-specific adapters and output caps. This comparison therefore cannot tell us how performance changed historically from then to now.

The full-policy utility difference was +0.0420 [simultaneous 95% CI +0.0077, +0.0763], and delivery was +0.1252 [+0.1021, +0.1482]; both had Holm-adjusted p = 0.00030. Surface comparison was NA because no earlier-roster model covered all links. For E-R4, utility is the negative of its adverse composite; the −0.0038 contrast and unadjusted descriptive interval do not support an inferential claim. The three confirmatory roster-class comparisons formed a separate adjustment family.

09 / Earlier experiment results

Unsupported commitment remained common even when outputs responded to the surface.

We completed and accounted for all 10,637 planned evaluations in the earlier sealed campaign. The primary insufficient-evidence analysis unexpectedly had only 154 units from four shared-seasonality bases across 11 models: 88 no-tool units and 66 workflow-package units. That limited sample cannot supply a ten-family average, and its failure score includes unusable output as well as commitment.

The model/base-standardized adverse rate was 92.05% [95% CI 88.64%, 95.45%], compared with the pooled 141/154 = 91.56%. We bootstrapped the full 122-base non-mitigation population with 2,000 draws; 1,957 contained a contributing base. Nine of ten intended families contributed no primary observations. We could not estimate the version requiring all 792 model-by-slot cells: only 44 were populated and 748 were empty. We did not substitute a broader average.

The near-universal disarmed endorsement makes a pure inability to understand the question an incomplete explanation. The high armed endorsement still shows insufficient withholding. The negative paired contrast establishes a surface-conditioned output change, without identifying internal recognition.

We compared workflows on identical tool-covered tasks from the same four bases. Execution and full-file access came together, so we could not isolate a pure tool effect. The 2,000-draw bootstrap had 1,914 effective draws, and we could not establish a benefit. The instruction intervention passed its recovery and over-abstention guardrails, but it needed to pass every criterion; those guardrails could not compensate for the missed improvement margin. We do not have complete tool transcripts.

We could not recheck power for the primary proportion because we had not frozen its null, alternative or target-power assumptions. Power calculations for other contrasts cannot establish adequate power for this four-base primary sample.

10 / Exploratory prompt controls

More caution in a prompt did not lower the end-to-end failure estimate.

In an exploratory prompt comparison after unsealing, we accounted for all 364 requests. We used four shared-seasonality insufficient-evidence bases and nine other-family sufficient-evidence guardrail bases, each across DeepSeek V3.2, Kimi K2.5, Llama 4 Maverick, GLM-4.7 Flash, Nemotron 3 Super 120B, MiniMax M2.1 and Qwen3-32B. We report those two samples separately. This comparison does not replace the sealed efficacy test.

We found no suffix with a lower adverse composite point estimate than neutral. Sufficiency-first reduced direct commitment from 8/28 to 7/28, while unusable output rose from 3/28 to 4/28. This is why we report direct commitment and end-to-end failure separately.

We balance the sufficient-evidence rates across tasks, bases, families and models, using classifiable claims. These rates do not measure general task accuracy. Their intervals neither establish nor exclude blanket withholding. The bootstrap direction-stability values describe how often the estimated direction persisted; they are not p-values.

11 / Interface checks and a larger-budget rerun

Getting more usable answers did not establish a safer evidence policy.

We reran all 867 original MiniMax M2.5 units in a separate exploratory experiment, increasing the output cap from 4,096 to 8,192 tokens. We accounted for every request. This new stochastic run produced substantially more responses and classifiable decisions, but the full-cohort adverse rate improved by only three points. It does not replace the sealed result.

Among 104 pairs classifiable in both runs, commitment changed from 90.38% to 92.31%: +1.92 points [95% CI −5.56, +9.74]. That interval does not establish equivalence. Marginal commitment rates used different survivors—186 original and 431 rerun responses—so their 90.86% and 90.02% values are not a paired effect.

When we inspected Nemotron 3 Super 120B parser failures, all 58 responses marked missing in a 318-unit eligible pool contained nonempty raw text. Fifty-seven were valid bare JSON that the frozen fenced-envelope contract rejected. All 58 were outside the frozen insufficient-evidence primary sample. We kept the original scores; accepting a new format after the fact cannot turn those responses into original successes.

This distinction matters for a product: malformed output calls for interface repair; an unsupported but well-formed action calls for a better evidence gate. Additional tokens and parser tolerance address acquisition, while recovery and restraint still need their own evaluation.

12 / Check the evaluation itself

Rendered cues were detectable, limiting the meaning of a response shift.

After unsealing, we trained a linear classifier to predict armed status from the later campaign’s 47 matched rendered surfaces. It reached ROC AUC 0.8959 [95% CI 0.8230, 0.9534], p = 0.00050. It exceeded the inherited 0.65 operational ceiling. We inherited that ceiling from a check on a different object, so it is a design target rather than a validated general threshold.

We found strong signal in the visible scaffolding and data values. The task-ID negative control did not show a material identifier signal. Those correlated cues do not prove that a model used them, but they mean we cannot attribute the surface shift cleanly to statistical recognition.

An additional five-link novel-private sensitivity produced full-rendered AUC 0.9200 [95% CI 0.7600, 1.0000]; it was too small to repair the broader concealment result. The earlier and rotation estimates are not pooled.

After unsealing, we ran 1,212 AI-judge assignments on 404 sampled responses. A parser consistent with the prompt accepted clean fenced or bare JSON: 1,131/1,212 (93.32%) were valid, including 523 fenced and 608 bare responses. An earlier fenced-only parser accepted 523/1,212 (43.15%). We report that lower value as a sensitivity because it arose from a scorer/prompt mismatch; it does not show that the judges failed to follow the requested format.

We used DeepSeek V3.2, Qwen3-32B, GLM-4.7 Flash and Nemotron 3 Super 120B as judges, without treating their majority vote as ground truth. The human review was first-party and candidate-visible. A retrospective attestation covered 266 review records—150 scorer-audit responses, 60 legacy-pilot responses and 56 task-validation items—plus 168 historical audit packets. Migrated records preserve final labels and provenance, but not contemporaneous interaction logs. We did not produce independent blinded double labelling, an independent human agreement statistic, quantitative frozen-scorer-versus-human validation or a human baseline.

13 / Exploratory breadth

High commitment in a family is conditional on the answers that survived.

After unsealing, we pooled classifiable binary commitments across all 23 later models to explore task-family patterns. This includes warranted and unwarranted task states, so it measures commitment frequency, not unsupported action or accuracy. Family coverage varied from 39.90% to 74.41%; most missing answers arose from envelope and token failures, rather than random missingness.

NA means that we could not classify any output in that cell. Low coverage can substantially change which answers remain in a comparison. We use these rounded complete-case summaries to explore patterns, not to identify family effects or rank models by family. The main full-policy, surface and delivery analyses keep their own missingness rules.

14 / Experiments and follow-ups

We also ran exploratory follow-ups to understand the failures.

We used earlier exploratory runs and interface checks to diagnose failures. Their observations are not added to either main campaign’s sample size. The private-task follow-up below preceded the later prospective rotation campaign reported above.

We ran this exploratory follow-up after unsealing. It used 612 assignments per model and an eight-unit shared-seasonality adverse endpoint. We report its model group, sample and unadjusted intervals separately from the sealed primary and the later rotation campaign. We make no pooled or pairwise significance claim across these experiments.

We also analysed previously collected private-split ladder responses across 24 model records. The best fail-closed full-policy estimate was 10.19%, and the short-ladder maximum was 20.00%. This retrospective exploration neither confirms the policy prospectively nor replaces the non-estimable sealed ladder. The Sonnet 5 table includes one repaired repetition record; its original fail-closed composite was unchanged. We used 30% as design context, not a pass/fail threshold, and make no ranking or pairwise significance claim.

We explored stance and action fields in 10,248 available rung-level records. Claim stances included 4,211 uncertain, 2,022 reject, 390 support and 47 not-applicable values; not every rung supplied a usable stance. We did not implement the eligibility analysis for the planned detection-to-action contradiction metric, so this inventory cannot establish a contradiction rate.

15 / Boundaries of the evidence

Our conclusions are limited by the tasks, samples and interfaces we tested.

• The tasks are synthetic. Known data-generating processes identify mechanisms, rather than their prevalence in a real organization. Structural templates recur across splits.
• We verified only 47 design links. Detectable armed/disarmed cues limit our interpretation of surface effects, and only three models had complete link coverage for adjusted surface comparisons.
• The sealed primary and the later narrow failure composite each cover four shared-seasonality bases. Neither is a ten-family avoidance average.
• Family and complete-case ladder summaries use answers that could be classified, and those answers are not a random sample. Missing data cannot be counted as caution or used to reconstruct joint policy success.
• We used different provider adapters and output caps, and made different budget-escalation decisions. The results describe those model configurations in one run window; they are not timeless properties of a model family.
• We ran each main campaign once. Querying each later task twice gives repeated measurements, not an independent replication of the whole campaign.
• Our human review was first-party and candidate-visible. Agreement among AI judges does not replace independent human validation.
• We can trace the aggregates and selected checks, but the available materials lack the source code, execution runner, private tasks and raw provider responses needed to rerun the experiment independently from start to finish. We elicited probability fields, but did not report a frozen calibration analysis.
• These benchmark measurements do not validate Opscien’s production accuracy, restraint or performance.

16 / What we take back to the product

Put evidence sufficiency between an answer and an action.

Three questions need separate tests: can a model react when the problem changes; can it follow the complete withhold–qualify–recover–ignore policy; and can it deliver a usable response under the actual interface? In the later campaign, delivery separated models much more sharply than complete evidence policy. A directional response alone was not a reliable decision gate.

For an analytical product, we would test the evidence required for each claim and action, preserve the tool and response traces, measure direct unsupported decisions separately from acquisition failures, and include recovery when decisive support arrives. The next practical test is whether that gate remains reliable on real workflow data under a complete cost and latency budget.

We share these findings because they sharpen a practical question: what evidence should an analytical AI require before it acts? The experiments give us some answers, and show where we still need better tests.