CounterResearch helps readers answer one question: what can I reasonably conclude from this AI research?
01
Who it is for
People who need to understand new AI work but cannot spend hours examining every paper: engineers choosing methods, researchers following neighbouring fields, and decision-makers assessing claims about capability, reliability and security.
02
The problem we work on
A paper announces a promising result. Its abstract sounds convincing, and commentary repeats the headline. A reader still needs to know:
- What did the researchers actually test?
- Does the evidence support the headline?
- Under what conditions does the result hold?
- What remains unknown?
- Should this change anything in my work?
We do that reading and explain the answers in plain language.
03
What we do differently
We publish the assessment and how we reached it.
Models from different AI labs examine each paper and raise objections. We check each objection against the paper and any other materials we examined. A named editor takes responsibility for the published interpretation.
We publish the objections that turned out to be mistaken, not only the ones that held. An AI reviewer can misread a paper just as an author can overstate a result. The record shows that our scrutiny applies to our reviewers too.
Using models from several labs is part of the method. Their number does not show that a review is good: models often make the same mistake. How we work explains what decides instead.
04
Who publishes it
CounterResearch is a publication of CounterProof Research. CounterProof is an adversarial review practice of Clavestra Capital Limited (Malta). It reviews security-critical code with the help of AI models, independently of the people who wrote the code, at counterproof.io.
CounterResearch applies the same method to AI research papers, in public. It is also a way to show how that method works. That gives us a reason to publish careful reviews, and gives you a reason to check them: every review carries its record so that you can.
Editor: Luuk Soons. The editor is responsible for every published review.
The thinking behind this method is set out in the editor’s essay Counterpoint in music, CounterProof in method: when agreement between reviewers is evidence and when it only repeats a shared mistake, and why criticism belongs inside the work of building something others depend on.
05
Our interest, stated
CounterProof sells independent reviews of security-critical code, including code written with AI. Research on AI-written code, AI security, AI evaluation and AI code review bears on that business in both directions. A paper that finds AI-written code unsafe supports the case for what we sell. A paper that finds AI review unreliable may count against our own method, which uses AI models. Every review where this applies says so at the top. We check an objection that favours our business the same way as any other, and we publish results that go against our interest.
No one with a stake in how a paper is received pays for its published review: not its authors, their employers, or vendors whose products it concerns. The editor chooses which papers we review. If we review a paper privately for a client, the client has no say in our conclusions. We publish such a review only if the client has no stake in how the paper is received, and then we name who paid for it.
06
How we use AI
Models from several labs raise objections, and an AI assistant helps us draft. Each review names the models used and what each one did. Every quotation and every number is checked against its source: by an automated check where the text can be matched exactly, and by the editor where it cannot (a number read from a chart, for instance). The editor reads each review before it is published.
07
Contact
Corrections and comments: counterresearch@nym.hush.com. See Corrections for what to send and what happens next.