2026-10-03
Neural weather models train on A100 and H100 systems
The study ran sharpening and colorization models on satellite data using A100 and H100 nodes, with some jobs failing and others completing successfully.
What ran
- sharpening benchmark, building 20: built 20 image tiles on a node with 128 CPU cores and 256 GB memory.
- sharpening benchmark, fetching 20: fetched the first 20 tiles via a transfer node with 1 CPU core.
- guided sharpening, 6 runs: executed six training runs on a node with 128 CPU cores, 4 NVIDIA A100 graphics processors (40 GB each), and 512 GB memory.
- red 4x sharpening, 5 runs: began five training runs on a node with 64 CPU cores, still running.
- red 4x sharpening, timing test: completed a timing run on a node with 256 CPU cores, 4 NVIDIA H100 graphics processors (80 GB each), and 600 GB memory.
What finished
- sharpening benchmark, building 20: finished in 5 minutes.
- sharpening benchmark, fetching 20: completed after 37 minutes.
- guided sharpening, 6 runs: completed in 1 hour 38 minutes.
- true colour, training: finished in 41 minutes.
- red 4x sharpening, timing test: completed in 12 minutes.
Numbers
- 12 jobs ran, 9 finished, 3 failed.
- Total node-hours: 3.75. Core-hours: 467.1.
- 1069 status pulls occurred, 1057 succeeded, 12 failed.
- Failed pulls: 3 the records do not say why, 9 the records do not say why.
- red 4x target gain: 0.94 dB over bicubic. Previous model: 0.634 dB.
Failures and fixes
- guided sharpening, all 6 runs: failed with exit code 1:0 after 1 hour 1 minute. The records do not say why. The records list no fixes.
- true colour, training: failed with exit code 4:0 in under a minute. The records do not say why. The records list no fixes.
- red 4x sharpening, main run: was cancelled before starting. The records list no fixes.
What was learned
- sr-v4-fullres-b02 achieved +14.20 dB over bicubic, final loss 0.002002.
- sr-v4-fullres-vis05 reached +17.84 dB, final loss 0.00142.
- Control models scored lower: sr-v4-fullres-b02-ctl at +2.35 dB, sr-v4-fullres-vis05-ctl at +6.20 dB.
- Timing tests for red 4x models scored between +0.37 and +0.45 dB.
- seed42 (true color v3) completed training with +10.13 dB and final loss 10.2463.
- Training logs show consistent error counts (1 per run) but no resumptions for guided sharpening runs.
- rcan-l2-d4 reached 0.705 dB by step 70,000; swinir-l2-d4 reached 0.689 dB by step 20,000.
What was decided
- Weather training page: publish every run's full record, figures and sync.json; notebook quotes the full record.
- Red 4x notebook: H100 timing result and the five-run submission (job 7066).
- Add author and purpose headers to the weather status scripts and units.
- hpc status: pull every cluster, merge into one page, publish.
- hpc status: publish history.json and status.txt beside status.json (replay data, one-screen summary).
- Red 4x: Weather study notebook: entries for the full-resolution, red 4x, TC v3, global radar pilot, benchmark and coincidence rows, 2026-10-02 to 2026-10-03.
- Weather study records: charge column taken from the jobs list (transfer-node jobs bill one core, not a node).
- Weather study notebook: job accounting and development-split records behind the 2026-10-03 entries.
- training status: home moves to (hub folder), weather copy kept; page assets as files for the hub CSP; About link follows.
- hpc status: third rsync fallback for openrsync.
- The guided sharpening at full resolution package file list was updated.
- The guided sharpening at full resolution package notes were updated.
- The red 4x sharpening package file list was updated.
- The red 4x sharpening package notes were updated.
- The satellite radar global pilot package file list was updated.
- The satellite radar global pilot package notes were updated.
- The sharpening benchmark package file list was updated.
- The sharpening benchmark package notes were updated.
- The training status monitor package notes were updated.
- The true color v3 package file list was updated.
- The true color v3 package notes were updated.
What is next
- guided sharpening: score the six returned runs against Sentinel-2 on terrain-placed scenes once the independent-truth collection closes, about 10 October
- red 4x sharpening: runs 1 to 5 are training on H100 nodes (about 15 hours for the longest); then choose the best network and loss; runs 6 and 7 follow
- true color v3: a new plan for a day and low-sun hybrid before any further training; the standard recipe stays on the site
- satellite radar global pilot: the four 72-hour training runs wait for a free A100 node (timing says 8 to 9 hours each); scoring on the cluster follows; the Europe test waits for 90 days of radar, about 17 December
- sharpening benchmark: the remaining 880 tiles are being fetched and built (about 110 processor-hours), then scored and released
- coincidence framework: build the one-year census package, about 4 node-hours on CPU nodes
- waiting in the queue: sharpening benchmark, building (allowed 14 h)
- waiting in the queue: sharpening benchmark, scoring (allowed 3 h)
- waiting in the queue: sharpening benchmark, release (allowed 4 h)
Narrative written by the house model from the day's records and checked by code; decisions and next steps copied from the records. Back to the study page.