CounterResearch

A note on the Agentic Stemmatics paper

Note not a review 14 October 2026

This is a note, not a review. The editor of CounterResearch, Luuk Soons, is the author of the paper it describes. CounterResearch has not reviewed the paper, and a review here would not be independent.

01

What the paper is

Agentic Stemmatics: Collation, Provenance, and Conservation for an Emerging Machine Textual Tradition, by Luuk Joseph Soons, is a preprint, not peer reviewed, that calls itself “A field report and a research programme”. It is published on the CounterProof Research site and deposited on Zenodo under a CC BY 4.0 licence. The version of record is v1.35 of 25 September 2026, DOI 10.5281/zenodo.22959463; the concept DOI 10.5281/zenodo.22030516 resolves to the latest deposit. The live page, as read on 14 October 2026, gives its version as v1.37, a revision of v1.35 that corrects the record of the paper’s review; no claim in the body changes, and one sentence was added to a search note in §7.1. Appendix A keeps the full revision history. Its competing-interest statement says the author directs the practice that sells review using this method, and that the paper “was drafted with an AI assistant of the Claude lineage (Anthropic)” under the author’s direction and responsibility. The same is true of this note.

02

What it argues

  • Model outputs are testimony, not proof (§1, §3). “We therefore treat model outputs as testimony.” The witness is the model lineage, sampled repeatedly; one verdict is one draw, to be checked against the artefact.
  • Agreement is weakest where it is most relied on (§1, §4). “Agreement is being counted as evidence exactly where it is weakest.” Cited measurements show that two models that are both wrong often give the same wrong answer, and that more capable models show more correlated errors.
  • Similarity is not descent (§4). Two models can reach the same mistake without copying; “a feature useful for recognising a model need not reveal its ancestry”. Its proposed rule: a shared feature counts as evidence of common ancestry only where independent origin would be improbable. The rule is called proposed, not established, and insufficient on its own.
  • When agreement can establish something (§3, §5). “Majority voting and consensus are unsafe as truth procedures when reviewer independence is unmeasured or failing.” Agreement can strengthen a claim once the relevant dependence has been assessed. Its alternative, adjudication, has stated costs: a veto for a stubborn reviewer, no bound on false positives, a narrow operating region.
  • The proposed investigations (§6, §7). Outputs return to training corpora through four channels (§6). Proposed: reconstructing model relationships from shared features scored by improbability under independent origin, checked against documented lineages, with PhyloLM as baseline (§7.1); a measure of reviewer independence, not yet built (§7.2); sealed model vintages, with rules for when panels become less independent (§7.3).

03

What it does not claim

In its own words: “We do not claim that the method produces better code.” The comparison meant to test it was discontinued before any run (§9, §13). Nor better reviews: “Its effectiveness has not been demonstrated.” (Abstract), and surviving its own review is not offered as evidence of its truth (§13). The central thesis can fail: §11.14 names three outcomes that would refute it, among them reconstruction from shared idiosyncrasies not improving against known model phylogenies. The field record is selected and internal: “The observations come from one commercial practice examining itself.” (§11.10). The method “cannot recover a defect that no witness raises” (§11.7). The correction record has no denominator: “We do not know how many unwarranted claims remain.” (§11.12).

04

How CounterResearch uses it

Our review method is drawn from the practice the paper describes: models from different labs raise objections, we check each against the sources, no objection is settled by counting models, and a named editor decides. How we work gives the steps and one rule stricter than the paper’s: an objection closes only against the sources, not when the model that raised it withdraws it. The paper does not show that this method works, and neither do we. A series of published reviews cannot show it either: the paper says such a record cannot establish cause or a detection rate (§9). The paper also warns about two things this site relies on: using models from different vendors does not by itself make their errors independent (§3), and the error rate of the person who decides is unmeasured (§11.4). Each review we publish will have to stand on its own evidence.

05

Where to read it