Research / Language models
Building each word’s first vector from shared parts
A language model begins by turning each token, a word or a piece of one, into a list of a few hundred numbers read from a lookup table, one private entry per token. I tested whether building that list from a small shared set of parts, mixed according to the words around it, helps a model handle combinations it never saw in training, which people do without thinking and models still do poorly. In 270 training runs on a supercomputer no alternative beat the plain table on any test, and 177 more runs found that the one lead, on pairing operations with arguments they were never seen with, did not hold on new seeds. The study closed on 5 October 2026 as a negative result, and a short paper on it is in draft.
- 270
- training runs in the first two rounds, each a small model trained from scratch2026-10-03
- 1 of 4
- tests where the sparse front end beat the dense one by the required margin; 2 were needed2026-10-03
- 0.565 vs 0.200
- the one lead: unseen operation and argument pairs, sparse against dense2026-10-03
- 0 of 177
- follow-up runs finished on nodes with four H100 cards2026-10-12
The numbers behind Figure 1
| Test | Plain table | Sparse, from context | Sparse, refined | Dense, from context | Dense, refined |
|---|---|---|---|---|---|
| Unseen pairs of operations | 0.577 (0.52, 0.59, 0.62) | 0.341 (0.23, 0.32, 0.47) | 0.381 (0.45, 0.35, 0.34) | 0.479 (0.30, 0.49, 0.65) | 0.233 (0.09, 0.26, 0.35) |
| Unseen operation and argument pairs | 0.496 (0.61, 0.28, 0.60) | 0.565 (0.56, 0.54, 0.60) | 0.575 (0.56, 0.55, 0.62) | 0.200 (0.15, 0.18, 0.27) | 0.383 (0.19, 0.37, 0.59) |
| The same, from 20,000 programs | 0.578 (0.60, 0.52, 0.61) | 0.397 (0.48, 0.37, 0.34) | 0.576 (0.47, 0.71, 0.56) | 0.474 (0.49, 0.33, 0.60) | 0.167 (0.09, 0.17, 0.23) |
| New names, from 20,000 programs | 0.998 (1.00, 1.00, 1.00) | 0.999 (1.00, 1.00, 1.00) | 0.995 (1.00, 1.00, 0.99) | 0.997 (1.00, 1.00, 1.00) | 0.998 (1.00, 1.00, 1.00) |
The question and the five front ends
Every model in the study has the same body: a six-layer transformer, 384 numbers wide, with about 10.7 million parameters, trained from a blank start on the same data with the same settings and three random seeds. Only the front end changes, the piece that turns each token into its first vector.
- Plain table
- One learned vector per token, the standard design and the baseline.
- Sparse, from context
- A small reader looks at the tokens before this one and picks 4 parts from a shared kit of 128, with their weights. This is the design under test.
- Dense, from context
- The same reader writes the vector directly, with no kit and no limit, at the same cost. It is the control for the sparse version.
- Refined
- Either reader followed by the same small adjustment rule applied twice more.
- Static and frozen kits
- Checks on the design: a fixed mix of parts per token that ignores context, and the sparse reader with a random kit that never learns.
The main hypothesis was that the sparse front end built from context beats the dense one on at least 2 of 4 held-out tests while matching it within 2 points on the ordinary ones. A difference counts when it exceeds twice the pooled standard deviation across the three seeds. These lines were written into the research plan before the first job ran.
The tests
The main tests live in a small invented world with known answers. A program is a starting point on a seven-by-seven grid and a short list of operations such as shift, scale, rotate a quarter turn and flip. The model writes out where the point lands after each step, and code checks every answer.
Specific combinations never appear in training. In one test, eight of the 49 ordered pairs of operations are held out. In another, six operation and argument pairs, such as shift right by 3, are held out although each half appears elsewhere. In a third, an invented word is defined by three worked examples and then used, and in the test it stands for an operation it never stood for before. The study also uses SCAN, a public benchmark from 2018 that asks for commands like “jump twice” after training on “jump” alone. A score is the share of held-out examples answered exactly right, on the whole held-out set, at the end of training.
What failed
The sparse front end beat the dense one on 1 of the 4 tests named in the plan, so the main hypothesis failed. On unseen pairs of operations the sparse version scored 0.341 against 0.479 for the dense one. With 20,000 training programs it scored 0.397 against 0.474. On new names every version sat at the ceiling, 0.998.
The refinement steps did not help the sparse version more than the dense one. Going from one adjustment to four lowered the sparse version by 0.072 on unseen pairs while the dense version gained 0.012. On new names with 20,000 programs the dense version gained 0.253 and the sparse one 0.006.
A stand-in for a published method of snapping each token to a shared set of class entries scored 0.001 on SCAN’s held-out “jump” split against a reference line of 0.90. That line was set from a published 99 percent that was measured with an augmented training set; on this split the published method itself reports 2 percent. In 62 runs the shared kit collapsed to a few parts in use, under a quarter of the 128 or fewer than about 8 in real use. Across all 270 runs no front end beat the plain table on any test.
The one lead
On unseen operation and argument pairs, the sparse front end built from context scored 0.565 against 0.200 for the dense one, a gap well above the required margin of 0.097. The refined versions split the same way, 0.575 sparse against 0.383 dense. The plain table scored 0.496, with seeds from 0.28 to 0.61, so the sparse versions match the table and the dense ones fall well below it. On five new seeds the gap did not hold (see the follow-up below), and no front end beat the table.
The follow-up
The follow-up of 177 runs on the H100 nodes reran the role test with five new seeds, ran it at a larger model size on two other choices of held-out pairs, added a rebinding test with 5,000 programs, and added COGS, a public test of new combinations in English sentences.
On the five new seeds the sparse front end scored 0.372 against 0.380 for the dense one, with the plain table at 0.575. Each seed landed high or low, so this test has two modes, and three seeds had split them unevenly. At the larger size the sparse version scored above the dense one on both choices of held-out pairs, 0.451 against 0.233 and 0.486 against 0.253, by less than the required margin, with the table at 0.382 and 0.247. The rebinding test could not be read, because the plain table reached only 0.54 on familiar cases. On COGS the sparse version scored 0.177 against 0.221 for the dense one. The study is closed as a negative result, and a short paper on it is in draft.
0 of 177 runs had finished as of 12 Oct 2026, 05:52 UTC, for 14.7 node-hours. The live status page keeps every run, step by step.
The published baseline that did not reproduce
To place these results beside published work, I ran the authors’ public code for the SQ-Transformer (Jiang, Zhou and Bansal, ACL 2024), which reported 0.9963 on SCAN’s “around right” split. In our environment the best checkpoints of three seeds scored 0.93, 0.52 and 0.90, a mean of 0.78, and the last checkpoints 0.24 on average, with the optimizer setting from the paper (4 October 2026). The difference is still unexplained, so its score on the “jump” split is withheld. Whether to stop here or make one run on the authors’ own processed data is my decision to make.
Where it ran and what it cost
The first 270 runs used 7.0 node-hours (900 core-hours) on nodes of 128 CPU cores and four NVIDIA A100 graphics processors with 40 GB each, four runs at a time, a few minutes per run. The follow-up ran on nodes of 256 cores and four NVIDIA H100 processors with 80 GB each.
The studies that grew out of this result look inside the transformer block instead, at the feed-forward layer. They are described under New lines below.