Reasoning with data
How models interpret evidence, handle numerical questions and arrive at an answer.
Research notes
Opscien Labs
We do research and publish the findings we believe are interesting and genuine.
We use these notes to explain what we tested, what we found and what we still need to understand. The methods, results and limitations are included in each note.
From the notebook
Research notes
01 / AI efficiency & numerical reasoning
When is another AI step worth the compute?
A second attempt can repair an answer or introduce a new error. We tested retries, calculator use and ways to choose the next step across five local models and seven hosted models.
Read the full note
02 / Statistical reasoning & evidence
When does evidence justify an AI answer?
We tested whether models withhold unsupported conclusions and resume when the missing evidence arrives. The results cover 23 models in the later experiment and eleven in an earlier study, with separate measurements of reasoning and response failures.
Read the full note
What we explore
Models, tools and methods.
We study how models reason with data, how tools change their answers and how to measure both reliably.
Tools and workflows
What changes when models use calculators, retry a task or choose a different next step.
Benchmarks and measurement
How scoring rules, comparisons and simple baselines shape the conclusions we can draw.
Failures and uncertainty
Where answers go wrong, what confidence signals reveal and what the evidence leaves unresolved.
How we share findings
How we report our results.
Each note starts with a question we investigated. We explain the experiment, show the results and discuss what they mean.
We include the model names, methods and full results so readers can follow our reasoning and judge the conclusions for themselves.
The question
What we wanted to understand and why it was worth investigating.
The experiment
The setup, data, comparisons and scoring behind the result.
The findings
What changed, what did not and which explanations the evidence supports.
The boundaries
What the experiment cannot establish and what we need to test next.