sean.theinformed.org / training / dcr / where the research stands

Where the research stands

8 October 2026. The program began as one study of a new way to build a transformer's inputs. That study failed on its own pass lines in its first week, and the question behind it moved inside the model. Eight lines have run since 3 October, about 1,500 training runs; seven are closed and one is running tonight. This page says what each asked, what came back, and what has held up. The status page has the live numbers and the replay; the guide explains the first study from the ground up; the notebook holds every entry.

The question

A small transformer learns a family of programs: a position on a 7 by 7 grid, seven operations that move it, and a target it must print. Training shows it every operation and most pairs of them. The held-out tests ask for what it never saw: pairs of operations in a new order (combinations), an argument in a role it never filled (role binding), and a name bound to an operation inside the prompt (rebinding). On these the plain model scores about 0.6 on combinations and far less on binding, and the gap between what it has seen and what it can put together is the program's subject. Each line changes one place in the model, at matched size and compute, and asks whether that place is where the missing ability lives. A second set of runs trains the same designs as 30-million-parameter language models on a billion tokens of open text, so a design that helps the programs can be priced on prose.

one decoder block: 6 in size small, 12 in the corpus model, width 384 token input constructor 1 residual stream norm attention MLP norm up 384 → 1,536 fold down 1,536 → 384 norm, head next-token scores context from the attention output rank-16 change gate 3 2 5 4 6 7 cut, solve or sample 8 1 Input constructor DCR The token's first vector built from a sparse code over a shared dictionary, in place of the table. 2 Fold FOLD The pointwise shape in the MLP: SiLU, squared ReLU, SwiGLU, a learned fold, a sign-split pair. 3 Context in the MLP CTX A gate fed from the attention output, or a rank-16 change to the up-projection set by a context summary. 4 Rotated frames VIEW One pair of MLP matrices read through four random rotations, the four outputs averaged. 5 The 45-degree pair MUL A second stream turned 45 degrees before a squared fold, so a channel can hold a product of two features. 6 Transplant TRANS The trained MLP matrices of every block moved into a fresh model and frozen there. 7 Width FFN The MLP from none, a sixteenth and a quarter of its width to four times, and a frozen random one. 8 Cuts CUT After the up-projection, an operation that cuts, solves or samples, each with a smooth twin.
The one block of a small transformer, with the place each of the eight lines changed.

The first line, and how it failed

The first study replaced the token table with a constructor: a shared dictionary of 128 learned vectors, from which each position picks four by reading its context, against a dense constructor of the same cost and the plain table. The hypothesis was that sharing parts across tokens would help the model combine what it had learned. The first grid of 270 runs showed one lead, the sparse constructor at 0.565 on role binding against 0.200 for the dense one over three seeds, and that lead was the reason for the second grid.

plain table A sparse code over a shared dictionary D dense control C token id one row per token first vector token identity width 128 causal context attention, ids so far network, hidden 256 code k = 4 of K = 128 kept shared dictionary one column per atom Σ weighted by the code first vector token identity + causal context network, hidden 256 no dictionary first vector, dense Also run: A+, the table with one more block; B, a sparse code from the token alone, no context; D₀, D with a fixed random dictionary; E and F, C and D with J = 2 shared refinement steps on the vector or the code; Q, the token vector plus one hard shared code.
The constructor the first line tested: context picks four atoms from a shared dictionary; the dense control and the plain table beside it.

On five new seeds the two constructors sat together, 0.372 against 0.380, and both below the plain table at 0.575. Each seed landed high or low on its own: the role task has two modes that identical runs fall into either side of, which is why three seeds had misled. At the larger size the sparse constructor led the dense one on both held-out designs by about 0.22, but the table sat between them, so the dictionary bought nothing over a lookup. Rebinding could not be read, the public combinations test showed no gain, and the published reference method, run from its own code, reached 0.76 at best on the control its paper puts at 0.996. In 447 runs and about 22 node-hours, every pass line failed or was unreadable. The line closed on 5 October as a negative result; a short paper carries both readings side by side, the original and the corrected.

Seven lines inside the block

If building the input did not help, the next places to look were inside the block: the fold (the nonlinearity in the feed-forward layer), the context's reach into that layer, how the layer's matrices are read, whether their shape travels, and how much of the layer is needed at all. Each line's pass lines were written before its first run.

The fold the network chooses

Each channel of a 30-million-parameter language model was allowed to learn its own activation shape, how sharp the gate is and how far it dips below zero, starting from SiLU; sign-split channels, a second fold inside the block, squared ReLU, SiLU and SwiGLU trained beside it at matched size on a billion tokens, three seeds each. The control held: SwiGLU beat SiLU by 0.069 nats, as published. The learned fold lost to SwiGLU by 0.018 to 0.021 nats on every source, and the sign-split and second-fold layouts lost by more. The shapes do change with depth, softer gates and deeper lobes in late blocks, in the same direction from three starting shapes, but how far they move follows where they start, so the depth reading is recorded as drift. Two side readings survive as leads: the same fold started sharper ends 0.003 nats ahead of SwiGLU, and squared ReLU beats SwiGLU by 0.011 on web text. 50 runs, 14.5 node-hours, closed 5 October.

SiLU S −4 3 0 squared ReLU R2 −4 3 0 sign-split SS −4 3 0 SiLU(−z) SiLU(z) input z, output f(z) z² above zero, 0 below two halves, two columns SwiGLU SG −4 3 0 SiLU(gate) × value value 1 ½ −½ learned fold f(z) = z · G(z) L −9 2 0 −0.896 −0.278 −0.103 L: β = g = w = 1, which is SiLU Lx: β = 2, g = ½ Lx2: β = ½, g = 2 In the learned fold each channel learns three numbers: β, gate sharpness: how fast the gate turns on around zero; g, far-tail gain: how deep the lobe below zero goes; w, lobe width: how far the lobe stretches along the negative side. All start at 1. β stays within 1/16 to 16, g and w within ¼ to 4. The gate never goes negative, so the lobe returns to zero.
The folds the line compared, drawn from their formulas.

Context inside the map

Three ways of letting the context reach inside the feed-forward layer, a gated layer, the same gate fed from the attention output, and a rank-16 weight change set by a context summary, were compared with the plain model on role binding and combinations. None beat it (135 runs). An 81-run check settled two odd readings from the first batch: a combinations gain in the shuffled-context arm was seed luck (0.648 against 0.601 on nine seeds, p 0.38), and two arms that cannot use the feed-forward layer, a shut gate at 0.134 and no layer at all at 0.115, get a few more held-out multiplier programs right than the full model at 0.071 on nine seeds (paired p 0.005). The no-layer gain of 0.044 sits under the 0.05 line, so the effect stays provisional until the programs are read by hand and by the per-binding reader, which is the one thing this line still owes. 216 runs, 4.1 node-hours; every run is in, and the line stays open for that read.

plain P the baseline parameters against P: +0 gated G gate and value from z parameters against P: +3,072 context-gated C the gate reads the context parameters against P: +7,680 context-rotated W the change starts at zero parameters against P: +3,888 hidden state z up-projection width 1,536 GELU down-projection hidden state z gate 1,024 value 1,024 GELU × down-projection attention output c z gate 1,024 value 1,024 GELU × down-projection z running mean of attention outputs map, 64 a: 16 values up-projection + U diag(a) Vᵀ, rank 16 GELU width 1,464 down-projection G and C hold the plain layer's 1,179,648 MLP weights per block. W narrows to 1,464 channels to pay for its rank-16 parts and the small map that sets a.
Plain, gated, context-gated and context-rotated feed-forward layers.

The multiplication primitive

A feed-forward layer whose hidden channels are read twice, once as they are and once after every pair is turned by 45 degrees, gives each pair a product of two features when the fold is a square. The line asked whether that product lets the model apply a multiplier it never saw in that role. It did not: 0.063 against 0.072 for the plain layer, with a small cost on combinations. On text the pair beat the compute-matched wide layer on every source, but the same two streams without the rotation did as well, so the gain belongs to the square fold and the turn adds nothing. 147 runs, about 7 node-hours, closed 5 October.

One matrix read from several angles, and the transplant

Four parallel streams read one set of feed-forward weights, each through its own fixed random rotation, and the results are averaged. On the program cells the four reads sat with the plain layer; on text they beat it by about 0.025 nats, well short of the compute-matched wide layer, which is the known compression result. Scored through rotations it never trained with, a trained model kept nothing: 0.002 on combinations against 0.610 through its own frames, the same as reading through the identity. The line's own sanity check fell short at nine seeds (the un-rotated four-stream model scored 0.581 against the plain model's 0.601), so the reading carries that caveat. 96 runs, closed negative on 7 October. The transplant line moved those trained matrices, frozen, into a one-frame model and the reverse, with everything else trained fresh around them. The test failed its own null: a model built around frozen trained matrices did far worse than one trained from scratch (0.373 against 0.601 on combinations), and frozen random matrices did better than frozen trained ones (0.546). Inside the batch, a four-frame host did much better with four-frame matrices than with one-frame ones, 0.501 against 0.335 (p 0.005); that is the stored-shape signal the line looked for, recorded as provisional. 90 runs, 1.7 node-hours, closed unresolved 5 October.

Four rotated frames over one matrix VIEW, RS4 z turn into frame k W1 one matrix fold fold fold fold turn the hidden units turn back mean W2 y Same weights as the plain layer; four times its multiply-adds. Frames drawn at random once, then fixed. A control runs with every frame the identity. The 45-degree pair MUL, PR2 u = W1 z stream 1 x² stream 2 turn 45° x² turn back mean W2 One channel pair (a, b) in stream 2: turned and squared: (a − b)²/2 and (a + b)²/2 turned back: (a² + b²)/√2 and √2·ab After the mean, one channel holds b(b + √2·a)/2, a product of two features; plain x² holds one squared. The transplant TRANS four frames attention MLP attention MLP attention MLP one frame attention MLP attention MLP attention MLP RS4 P P ← RS4 RS4 ← P Every block's trained MLP matrices move into the other model and stay frozen; attention, norms, table and head train from a fresh start. Controls: P ← P, RS4 ← RS4, and P ← its own random start.
One matrix read through four frames; the 45-degree pair with a squared fold; the transplant.

How much feed-forward a small transformer needs

The layer was removed, cut to a quarter and a sixteenth of its width, frozen at its random start, and widened four times, nine seeds each on the three program cells with shuffled-label copies beside them. Nothing beat the plain layer: on combinations every reduced arm trailed it, by 0.005 (no layer) to 0.096 (frozen random), and the no-layer and frozen arms trailed it on role addition by 0.10 and 0.07. Every arm beat the layer at four times the width, by 0.12 to 0.21, each at corrected p below 0.04; the plain layer scored 0.76 on combinations and the wide one 0.47. The line closed negative on 8 October, and the finding that survives is the earlier one: at this size the layer earns its place, and width past it costs. Of the 12 corpus runs, 9 finished and the 3 four-times-wide runs failed for memory on 40 GB cards; the losses go in as a note and cannot change the verdict. 216 program runs in 3.0 node-hours; the corpus runs 5.1 so far.

hidden units per block, to scale none OFF attention and residual only a sixteenth N16 96 units a quarter N4 384 units plain P 1,536 units, the reference four times W4 6,144 units frozen random R 1,536 units, never trained Weights and multiply-adds scale with the width. Program cells at size small, nine seeds; the corpus model at three seeds for none, a quarter, random and four times.
The feed-forward width arms, with the plain layer as the reference.

The line running tonight

Seven lines varied the fold and the layer around it, and none beat the plain block. The linear map and the pointwise fold are both continuous, and together they already span every smooth function, so a different fold could not add anything the block could not already represent. The eighth line steps off that plane. It tests operations of a third kind: a cut, which keeps the largest values, looks up a code, samples, sorts or routes; a quotient, which wraps a value onto a circle or reads the itinerary of a chaotic map; and a solve, which returns a fixed point or an energy minimum. Each carries a twin with the discontinuity smoothed away, so a gain is credited to the cut and not to the parts around it. The pass lines ask for a gain of 0.05 over the plain block and over the twin on role addition or combinations, with a shuffled-label null on both, and a price on text of at most 0.05 nats. The plan has no line of the form "no worse", which the feed-forward result showed the null cannot test.

stretch the linear maps fold the pointwise shape a third kind cut, solve or sample + + up-projection → the operation below → down-projection grid 1 twin keep k K softmax weights Ks lookup L attention to codes Ls sample B probabilities Bs grid 2 twin sort S sort within pairs Ss route E weighted mixture Es wrap Q saturating fold Qs tent map I −++ −+− settling map Is grid 3 control fixed point F control: block twice energy min. G control: plain P A twin keeps the parts around the cut and smooths the cut away. Keep k keeps an eighth of the hidden values; lookup snaps to the nearest of 256 learned codes; the fixed point stops within 12 steps; the energy minimum takes 4 gradient steps.
The cut, quotient and solve blocks of the open line, each above its smooth twin.

Grid 1, 252 runs in 4.3 node-hours, was read on 8 October: keep-the-largest, lookup and the sampled fold all fail both lines. Keep-the-largest trails the plain block by 0.16 on role addition and its own twin by 0.15; lookup gains 0.02 on combinations and loses 0.08 on role addition; the sampled fold loses 0.18 on combinations. The shuffled-label copies fail as they should. Two readings are kept for the next grids: on the multiplier cell, where the plain block scores 0.07, lookup and the sampled fold and their twins all sit about 0.05 above it; and lookup's spread across seeds is 0.05 against the plain block's 0.18. Grids 2 to 7 follow, about 2,100 runs across sort, route, the wrap onto a circle, the chaotic itinerary, the two solves, and the spectral, chiral and entropy variants from the theory note; an eighth grid of paired designs is proposed and not yet confirmed. Tonight grid 2's 324 runs are home and being read, grids 4, 5 and 6 are running on nodes with four A100 cards, grid 3's timing run is queued, and the rest wait for a job. The first three grids were budgeted at about 22 node-hours; the timing estimates for the grids with a job so far come to about 80. One corpus grid on nodes with four H100 cards waits for any arm that passes.

What has held up

Across about 1,300 runs, four readings stand on their own numbers, and two more are provisional.

The method readings matter as much as the model readings. A pass line of the form "no worse than the plain model" is passed by construction when both arms sit at chance, so the open line has none. A pooled score over three seeds can be seed luck; the shuffled-label copy beside every arm is what tells.

Runs and cost

LineStateRunsNode-hoursVerdict
Building inputs from shared partsclosed 5 Oct447about 22negative; short paper
The fold the network choosesclosed 5 Oct5014.5negative; two leads
Context inside the mapopen for one read2164.1negative on its lines; one provisional reading owed a read
The multiplication primitiveclosed 5 Oct147about 7negative; the square fold is the text gain
One matrix read from several anglesclosed 7 Oct969.5negative
The transplantclosed 5 Oct901.7unresolved; the test failed its null
How much feed-forward is neededclosed 8 Oct216 + 12 corpus8.1 so farnegative; width past the baseline costs
A third kind of operationrunning576 of about 2,4004.3 so fargrid 1 negative; grid 2 home and being read; grids 4 to 6 running

Program cells run on nodes with four A100 cards of 40 GB; corpus arms on nodes with four H100 cards of 80 GB. Node-hours are the totals the status files recorded; where a line's plan and its record differ, the sealed notebook entry is the figure used.

What comes next