Research / Language models
Language models that check papers
Refereeing a scientific paper means recomputing its numbers and testing its derivations. I am training an open language model of about 32 billion parameters to do that, rewarded only when code confirms its check, along with a second model, the cleaner, that removes the habits of machine-written prose from a draft without changing a number. The starting point is low: given two of my own papers and the 51 problems an outside technical review found in them, a 235-billion-parameter model with tools caught 3 for the reviewer’s reason and wrongly flagged 3 of 51 correct passages (4 October 2026). Two studies of model architecture, approved the same day, come next.
- 3 of 51
- review findings caught for the reviewer’s reason by the second loop2026-10-04
- 1 of 51
- caught by the first loop, on the same test2026-10-03
- 3 of 51
- correct passages wrongly flagged, in each loop2026-10-04
- 1.5
- node-hours used on the supercomputer so far2026-10-08
The numbers behind Figure 1, by kind of finding
| Kind of finding | Findings | Caught, first loop | Caught, second loop | Never checked, first | Never checked, second |
|---|---|---|---|---|---|
| Recompute | 6 | 0 | 2 | 5 | 3 |
| Derivation | 7 | 0 | 0 | 4 | 4 |
| Definition | 4 | 1 | 0 | 2 | 3 |
| Statistics | 7 | 0 | 0 | 3 | 2 |
| Consistency | 11 | 0 | 1 | 6 | 5 |
| Overreach | 16 | 0 | 0 | 9 | 8 |
| All | 51 | 1 | 3 | 29 | 25 |
How the referee is tested
On 2 October 2026 an outside technical review of papers A and B, my two orbit papers, listed 51 problems: wrong recomputed numbers, derivations that do not follow, terms used for the wrong quantity, statistics that do not support the claim, inconsistencies between sections, and conclusions that reach past the data. Those 51 findings are the answer key.
The referee is a loop around the model. It lists the claims in a passage, states for each what would make it false, and checks it with tools: look up the paper, recompute a number, run the orbit code, test a derivation with a symbolic algebra engine, or run a statistics routine. For each finding it reads the 81 lines around it. A finding counts as caught only when the referee flags it for the reviewer’s reason, judged by hand. The same loop also reads 51 passages that the review found correct, to count false alarms. The model is Qwen3-235B, served on two DGX Spark units in my house.
Results so far
The first loop caught 1 of 51 findings, and on the two whole papers it caught none. The second loop, redesigned from the first one’s failures, caught 3 of 51 in their windows and 1 on the whole papers, a statistics finding: a 90 percent interval of 0.632 to 0.687 that straddles the paper’s margin of two thirds. Both loops flagged 3 of the 51 correct passages.
Most findings were never picked up for checking: 29 in the first loop and 25 in the second, because the loop checks 10 claims per window and its choice is not the reviewer’s. When the right sentence was checked, the model accepted the paper’s own wording as support, invented tool inputs, or reported its own wrong physics. Neither loop caught any of the 16 findings of overreach, or any of the 7 statistics findings inside a window.
Two smaller tests on 4 October: on five documents with a planted error the second loop found 3 of 5, with a precision of 0.571, and on ten practice claims with answer keys it agreed with the key on 2 and returned “cannot verify” on 8. The second loop used 6.2 million prompt tokens for the 51 findings, counted from the model’s own usage reports.
Training the referee
Prompting improved the mechanics: more claims stated in words, a check before every verdict, and recomputation that runs by name. It did not improve which claims the model chose to check or how it judged the result. The plan is to train that judgment into a smaller open model that fits on one node. Four open base models of about 32 billion parameters will first be scored on 200 documents with planted faults, 99 clean papers and 600 practice claims with answer keys; the best becomes the base for supervised training and then reinforcement learning.
The hypothesis I want to publish: reinforcement learning with a reward that pays only when code confirms the check raises the share of claims correctly verified by at least 20 points over the same training with a reward for a plausible-sounding review. The referee’s timing test, a short job that loads the model on one node and measures its speed, passed at 07:17 UTC on 8 October 2026. The baseline of the four models has not started: it waits for the second release of its test data, whose sets of claims and planted faults must first pass their checks (status as of 8 Oct 2026, 10:56 UTC).
The cleaner
The cleaner is a low-rank fine-tune of a 32-billion-parameter base that rewrites a draft to remove the habits of machine-written prose, such as narrating its own honesty or closing on a slogan, and leaves every number, name and claim as it was. Its timing test passed at 07:12 UTC on 8 October 2026. Its first fine-tune on the supercomputer has been running since 09:30 UTC on 8 October 2026 (status as of 8 Oct 2026, 10:56 UTC).
Next: two studies of model architecture
The token study found no gain from changing how a model reads its input. These two studies, approved on 4 October 2026 and not yet run, move the question into the block itself.
Which activation shape does a network choose?Running
Each layer of a transformer bends its signal once, in the feed-forward layer, through an activation function, and published models use one shape everywhere. In this study each channel learns its own shape, how sharp its gate is and how far it dips below zero, in models of about 34 million parameters trained on 1 billion tokens of an open corpus of web text, mathematics and code, with three seeds. It asks whether the learned shapes change with depth, and whether a learned or restructured shape matches SwiGLU, the gated unit most current models use, at equal size and compute. grid being written on nodes with four H100 cards, about 6 node-hours; the pass lines are written. My prior: one in two for a clear picture by depth, one in five for beating SwiGLU.
Should context reshape the feed-forward layer?Running
In a standard block the feed-forward weights a token meets do not depend on the rest of the sentence. This study gives them that dependence three ways at equal size: a gated layer, the same gate fed from the attention output, and a small weight change set by a summary of the context. Each is compared with the plain model on the token study’s tests of unseen operation and argument pairs and unseen combinations, with three seeds, and each has a frozen copy and a copy with the context shuffled between examples as checks. grid being written, about 3 node-hours. My prior: one in three for a gain beyond seed noise.
Both will be tracked on the token study’s live status page when they run.