Live status / Machine learning
Machine learning
Small transformers trained from scratch, to learn how a model comes to combine operations it has seen into combinations it has never seen. The current line asks whether writing out the intermediate steps in some training examples is what makes that work.
-
Regime 5: both pass lines met
ReadWhen half the training examples show their intermediate steps, the base-size model trained three times as long gets 0.750 of held-out combinations exactly right; trained on final answers alone, it gets about 0.20. It beat both controls on all nine seeds. Trained that long, the base model now matches the small model (0.768) as closely as this comparison can tell, so the larger size buys nothing on this task.
Figure 1. Share of held-out combinations answered exactly, for each of nine seeds (dots) and their mean (bar). "Half" means correct steps were written in half the training examples; the percentages are smaller shares. 1× is 12,000 training steps and 3× is 36,000. Final answer only shows its mean. The share rows come from the third, fourth and fifth grids. Read 12 October. The tests behind the reading
Paired over nine seeds, training with written steps beat final-answer training by 0.550 and training with random intermediate states by 0.556 (Holm-corrected p of 6×10−9 and 3×10−9); the pass lines asked for at least 0.05. Three times the training raised the base model by 0.264, from 0.486 to 0.750, with all nine seeds above 0.5 instead of three.
The size comparison came out at −0.018 (p 0.57). The smallest gain this comparison could detect is 0.096, so it cannot show the two sizes equal within 0.05; it shows no benefit from size.
Final-answer training kept fitting its examples better (training loss down to 0.503 nats) while its held-out score stayed near 0.20. With steps in 12 percent of examples the mean is 0.392 and no seed passes 0.5; at 15 percent, six of nine seeds do.
-
Regime 6: longer programs, next
NextMixed training has never written an exact trace past the one to four operations it trained on. Regime 6 asks whether training on longer programs reaches longer programs: 36 small runs of about five card-minutes each, on one four-card node in a free queue, after a 20-minute timing check.
-
Earlier lines
ClosedThe study began with the token front end: building each word's first vector from shared parts. That line closed with a negative result after 270 runs, and the architecture lines that followed are in the longer write-up, with every run browsable step by step.