How a review is made, what our two evidence labels mean, and the claims we refuse to make.
01
Five steps
1
We read the paper and write down what it claims. Each main claim is quoted with its page, section or table. We separate what the researchers measured from how they interpret it.
2
We decide what else to examine: appendices, released code and data, earlier versions of the paper. A review names everything we examined, and only what we examined.
3
Models from different AI labs raise objections. Each model receives the paper and our account of it, and is asked what is wrong with both. Our own account is under review too.
4
We check each objection against the sources. An objection held if the sources support it. It was mistaken if the sources show it is wrong, and we quote where. It stays open if what we examined cannot settle it. No objection is settled by counting how many models agreed with it.
5
A named editor decides what we publish and is responsible for it.
02
Two labels for our evidence
Every claim we make about a paper carries one label. The label describes that claim, not the paper as a whole.
Read: supported by reading the paper and the materials we name, without running anything.
Rerun: we ran that part of the experiment ourselves. We say exactly what we ran, which version of the code and data, what we changed if anything, and what we got, including where our numbers differ from the paper’s.
We never present the first as the second. Rerunning the authors’ code with their data is not an independent reproduction, and we do not call it one.
03
What we refuse to claim
01
We do not treat agreement between AI models as proof. Models often make the same mistake: when two are wrong, they tend to be wrong in the same way, and models from different companies are not automatically independent of each other (Kim et al., ICML 2025). A shared answer can be a shared error.
02
We do not let a fluent objection outrank the source. Model output tells us where to look. The paper, its materials and our reruns decide.
03
We do not drop an objection because most models disagreed with it. An objection is closed only by checking it against the sources. If they cannot settle it, it stays open and we say so. This is stricter than CounterProof’s rule for code, where a finding also closes when the reviewer who raised it withdraws it: here, a withdrawal alone does not close an objection.
04
We do not report “we found no problem” as “the paper has no problems”. A review says what was examined, not what is absent.
05
We do not grade papers. There is no score. A review says what the evidence supports and where it stops.
04
What the method cannot do
- It is only as good as the materials examined. When code or data are not released, we say so and explain what, if anything, we could rerun with the materials available.
- Models from different labs may share blind spots, so a valid objection may be one that none of them raised. Readers who see one can tell us: see Corrections.
- The same AI review run twice may produce different objections, and so may a newer model.
- The method is new. Whether it consistently delivers what this page describes is something our published reviews must show. Each review keeps its full record so that you can judge.
05
The same method as CounterProof
This is the method CounterProof uses to review security-critical code, applied to research papers. Its rules for code are published at What we will not claim. The main difference is the evidence: code can be tested at a fixed revision. For a paper, we say which experiments, if any, we reran and which claims we assessed by reading the sources.
CounterProof Research is a practice of Clavestra Capital Limited.