Forecast writing

Where it stands, 21 October 2026

Two open models carry the work. Gemma 3 27B is the training line: a LoRA fine-tune that learns to write a full discussion, up to 8,192 tokens, in training run 102. Mistral Small 3.1 24B is the generation line: it writes development forecasts for the scorer to read, so the scoring can be built and checked while the training runs.

Training. Run 102 planned 10,590 optimizer steps and has saved all of them. It runs in pieces, and each piece so far ended on a checkpoint that was reviewed before the next one started. The pieces ran on the free background lane of one GPU cluster until the wait for a slot passed 25 hours. On 20 October the run moved to the uncharged back-fill lane of a second cluster, as three 12-hour pieces. They then sat behind the account's job limit, which another project's shards had filled; early on 21 October that project freed two slots, but the limit never cleared. That evening the three pieces were canceled and the run moved a second time, to a third cluster whose free lane stood idle: the base model, the training image and the reviewed checkpoint were copied to its home disk, a census and a tidy removed a duplicated copy of the base on both machines, and the first of three four-hour pieces started at 20:42 UTC on 21 October, from step 3,498. That piece ran its full 3 h 45, trained 1,096 steps with finite losses, and saved at step 4,594 in eleven seconds. A check that ran after the save compared the new checkpoint with the record of the old one and stopped the piece with an error, so no closing record was written and the two queued pieces were canceled. Early on 22 October the save's four files were hashed on the cluster and reviewed at home, the cause was found in the metadata rather than the weights, and a repaired kit that verifies those hashes on the cluster before it submits queued three more four-hour pieces, the first from step 4,594 with no dependency on the failed job. Its first attempt stopped within minutes: it had validated the save and retired the older one, but read a disk-quota figure that had not yet caught up with that deletion and refused to load the optimizer. A revision that waits for the quota to refresh was rehearsed and queued again before 03:00 UTC, behind another project's work on the same lane. It started at 02:45 UTC, trained from step 4,594 with the optimizer loaded, held finite losses throughout, and saved at step 5,685 in eleven seconds at its budget stop. Its closing record was written cleanly, and the second piece started from that save at 06:33 UTC. That piece reviewed the new save, retired the 4,594 save, waited for the quota figure to catch up before loading the optimizer, trained 1,087 steps and saved at step 6,772 at 10:15 UTC. The third piece started from that save at 10:21 UTC. About 3,800 steps remain, and four more four-hour pieces were queued behind it at 11:10 UTC, each waiting on the one before, which at this pace completes the run. The third piece repeated the pattern and saved at step 7,857 at 14:03 UTC; the first of the queued pieces started from it at 14:09 UTC, with about 2,700 steps left. That piece saved at step 8,942 at 17:51 UTC and the next at step 10,025 at 21:39 UTC, each after retiring the save two back and waiting for the quota figure to refresh, each with finite losses throughout. The piece after them started at 21:41 UTC with 565 steps left, trained them with finite losses, and saved the finished adapter at step 10,590 at 23:45 UTC. The same job then loaded the base model with that adapter and began writing the 4,595 development forecasts, at about one and a half minutes each; it had written seven when a long forecast passed the fifteen minutes its watchdog allows between progress lines, and the job was stopped at 00:18 UTC on 23 October. The last queued piece was dropped with it, as the dependency rule intends. The seven forecasts and the adapter are saved. A one-hour trial of a faster writer that produces several forecasts at once finished early on 23 October: it was faster, but its forecasts diverged from the plain writer's after a handful of tokens, so it was set aside and its twelve forecasts quarantined pending a numerical check. At 03:15 UTC on 23 October the plain writer resumed from the seven saved forecasts, changed only to report progress every minute, as a chain of 36 four-hour pieces, each waiting on the one before. The first piece set the quarantine aside and verified the finished adapter by hash. Its saved census contains 112 discussions: the original seven plus 105 new ones. Its stdout lost the last twelve June discussions, whose saved files passed review. The second piece added 85 by 10:15 UTC, bringing the saved total to 197. The third started at 10:47 UTC on longer forecasts from a second office and returned another successful budget stop with 260 saved discussions, an increase of 63. Its delayed log is truncated at 61 printed completions with no closing result; the accepted count comes from its closing census. At that pace the 36 pieces reach about 3,300 of the 4,595, so a further 15 pieces were queued behind them at 13:30 UTC and ten more at 14:00 UTC, 61 in all as a ceiling; any piece that finds the set complete stops at once, and the last ten count completion against the exact list of 4,595 cases and record every case they write. A one-hour numerical check run alongside found that the plain writer itself, run twice on the same prompt, diverges after a few dozen tokens: greedy decoding on this hardware yields one draw from a narrow distribution rather than a unique text, which is why the forecasts are scored in bulk with confidence intervals.

Writing forecasts. Generation finished on 19 October in round 107: all 4,595 development cases in 19.5 minutes. The six rounds before it crawled because the jobs miscounted their storage; once that was fixed the whole set ran in one sitting.

Scoring. The first scores use rules only (round 111, 20 October, 6.5 minutes on CPUs). For the chance of rain at a point over 24 hours, the written forecasts score a Brier error of 0.1660 on 20,701 matched checks, against 0.1759 for climatology: a skill of 0.0563. 374 forecasts ran into their length limit. Office by office the tables are mixed, and claims worded as hedges often score below climatology. Temperature skill and the consistency count are not measured yet.

Sentence extraction finished at 14:41 UTC on 24 October: all 12,376 protocols for the blind 300-discussion sample. Scoring now includes 2,000 discussion-bootstrap resamples within 24 office-month groups. Numeric point rain-probability skill against matched climatology is 0.0512 (95% interval -0.0277 to 0.1234). Daytime temperature skill is 0.2015 (0.0667 to 0.3042); overnight temperature skill is 0.1489 (-0.0116 to 0.2839). Rain-chance skill is small and its interval includes zero. Daytime temperature error is about a fifth lower than matched climatology, with an interval clear of zero improvement; the overnight comparison is borderline. Per-office results and paired error intervals are available. These provisional results describe the earlier saved generation. The current completed-training generation is still writing its development discussions. Independent claim labels are absent for the generated sample, and previous-issuance, human, field-blind and consistency comparisons remain unmeasured.

Housekeeping. Files on the clusters' work disks that no job reads are being moved to the archive.

What this project is

A numerical weather model already holds a forecast: millions of numbers for every place and hour. A human forecaster's value is reading those numbers well, correcting them where the model is known to go wrong, saying it plainly, and applying physics to the question in front of them: will my radio reach that ship, where is the person who fell overboard, will that storm drop large hail. We are training open language models to do that job from the same inputs a forecaster reads (the issued products, the model guidance, the observations), and measuring the result the way forecasters are measured: against what the weather did, next to the human product, climatology and persistence.