Teaching a language model to forecast, and proving it can

The question and why it matters

Can a language model, given the output of a weather model and the observations a forecaster would have, write a forecast that verifies better than the raw model output read naively, and approach what a professional forecaster writes? The published systems are scored by how much their text resembles a human's; none is scored against observations, and none is tested on whether it will say the model run looks wrong. Either answer changes something: a yes gives a forecaster a colleague that reads every field and every sounding; a no, measured properly, says what a language model cannot do here and where the physics must stay in tools.

The basics, built for this problem

A numerical weather model is a physics program that turns today's measured atmosphere into tomorrow's: millions of numbers (temperature, wind, humidity, pressure) on a grid of points and heights, for every hour ahead. A forecaster reads those grids, knows where the model tends to be wrong, looks at what is happening now, and writes a discussion: a page of plain prose about the next week. A language model is software that writes text one word at a time; training it means showing it many examples and nudging its internal numbers toward writing what the examples wrote. Fine-tuning starts from a model that already writes well and teaches it this job. A forecast is skilled when it beats a reference (persistence, climatology, the raw model) on what happened; the score is computed by extracting claims from the text (a high of 84, a 60 percent chance of storms) and checking each against the station record. Generalize means doing it for forecast offices and a year the model never saw in training.

The data

Every training example is four things aligned on the clock: the professional discussion, the model runs that were available when it was written (never a later one), the observations up to that hour, and what happened afterward. The text is public: about a million discussions from every United States forecast office since the late 1990s, plus marine, tropical and severe-weather products, and public forecasts from other countries for the transfer test. The fields come from the open archives of the global and regional models, cut to each office's area. The held-out set is the most recent year and a fixed set of offices. The specialist domains (search and rescue, radio propagation, thermal contrast, rapid intensification, radar, soundings, sea state) are built from physics tools, with the answer from the tool and the dialogue written around it.

The model

An open base model, fine-tuned, surrounded by tools. The small tier is a model of about 24 to 27 billion parameters that runs on one desktop graphics card; the large tier is 70 to 120 billion for datacenter hardware. A briefing layer computes what a forecaster would (soundings and their parameters, refractivity profiles, ensemble spread, sea state) and hands it to the model as text. Tools fetch data, parse files, run the physics and classify radar patterns. Pictures reach the model's vision path for pattern only. A learned encoder of the whole field, feeding the model directly, is a controlled experiment against the briefing, not an assumption. The data-flow diagram and the parameter table go here once the first fine-tune has run.

The training schema

Three stages. Imitation: fine-tune on the paired corpus with four input representations (raw tables, the computed briefing, both, rendered charts) and the same text targets. Skill: the model writes several candidates per case, code scores them against what happened and against the input, and the better ones win; a proper scoring rule for probabilities, a coverage term and a consistency count keep it from hedging, from saying nothing and from inventing numbers. Conversation: dialogues with correction and deliberate wrong assertions. Then the same data up a size ladder, and the large model teaches the small one. Each run is one node for a few hours, resumable from checkpoints; schedule, batches and seeds are recorded here once set.

Evaluation, and what a null result would mean

Three scores, all reported together: verification against observations, consistency with the input fields, and a blind read by forecasters. Pass lines are written in the research notebook before each run. Two nulls go through the same scorer every time: the same training with each text paired with another day's fields (if skill does not fall, the model is not reading the fields), and a climatology text. A null result on the main question would mean the model's text skill comes from style and climatology, and that a forecast assistant should present tool output rather than write forecasts.

How to read the status page and the replay

The status page lists each stage with its state and a one-line outcome, and the node-hours charged. The replay, once runs exist, walks through training step by step: loss curves per run and condition, each evaluation against its pass line, the clock and the cost moving alongside. Nothing on the page updates on a clock; it updates when a job ends or someone presses Sync.

Glossary

Forecast discussion

The prose a forecast office writes several times a day explaining its reasoning for the coming week.

Verification

Scoring a forecast against what was observed.

Climatology

What usually happens at a place and time of year; the cheapest forecast, and the one to beat.

Brier score

A score for probability forecasts that rewards being right and confident and punishes being wrong and confident.

Sounding

A vertical profile of the atmosphere, from a balloon or a model; its parameters (CAPE, shear) say how likely severe storms are.

Duct

A layer of the lower atmosphere that bends radio waves back toward the surface, carrying signals far beyond the horizon.

Held out

Data the model never trains on, used only to test it.

Node-hour

One cluster node, with its graphics processors, for one hour.