Neural sharpening jobs run amid failures

What ran

• A sharpening benchmark fetched data using a transfer node with 1 CPU core, running for 24 h 21 min.
• Red 4x sharpening, pack swinir l 1 d 4 ran on a node with 256 CPU cores and 4 NVIDIA H100 graphics processors, completing in 7 h 26 min.
• Red 4x sharpening saved results via a transfer node with 2 CPU cores, lasting 4 h 46 min.
• Guided sharpening executed 24 runs on a node with 128 CPU cores and 4 NVIDIA A100 graphics processors (40 GB each), finishing in 1 h 49 min.
• Red 4x sharpening, 5 runs continued on a 64 CPU core node for 11 h 50 min.

What finished

None of the jobs marked as finished completed on this day. The records show no jobs in a completed state by end of day.

Numbers

• 9 jobs ran during the day.
• 0 jobs finished.
• 4 jobs failed.
• Jobs that ended consumed 3.86 node-hours and 988.82 core-hours.
• 3 status checks were performed, all successful.
• Three models under red 4x sharpening (edsr-l1-d4, rcan-l1-d4, rcan-l2-d4) completed training by step 375,000 and scored 0.64, 0.67, and 0.73 dB over bicubic respectively.
• SwinIR models reached step 290,000 with scores of 0.67 and 0.74 dB over bicubic.

Failures and fixes

• Red 4x sharpening, pack swinir l 1 d 4 failed twice on a 256 CPU core, 600 GB memory node, each time ending within a minute with exit code 1.
• Red 4x sharpening, scoring failed after 5 minutes on the same node type with exit code 1.
• Red 4x sharpening, 5 runs failed after 15 h 20 min on the same node with exit code 1.

The records do not say why these jobs failed. The records list no fixes.

What was learned

• edsr-l1-d4 completed with a held-out score of +0.64 dB over bicubic and final training loss of 0.012202.
• rcan-l1-d4 completed with +0.67 dB over bicubic and loss of 0.010798.
• rcan-l2-d4 completed with +0.73 dB over bicubic and loss of 0.000472.
• swinir-l1-d4 reached step 290,000 with a best score of 0.677 dB at step 230,000.
• swinir-l2-d4 reached the same step with a best score of 0.744 dB at step 290,000.
• Multiple historical performance points were recorded for rcan and swinir models across training steps.

What was decided

• Red 4x notebook: decision, exam days scored on the cluster.
• Sharpening benchmark: Notebook: benchmark first tiles timed and the 880 submitted; radar pilot timing result.
• The red 4x sharpening package file list was updated.
• The red 4x sharpening package notes were updated.
• The satellite radar global pilot package file list was updated.
• The satellite radar global pilot package notes were updated.
• The training status monitor package notes were updated.

What is next

• guided sharpening: score the six returned runs against Sentinel-2 on terrain-placed scenes once the independent-truth collection closes, about 10 October
• red 4x sharpening: runs 1 to 5 are training on H100 nodes (about 15 hours for the longest); then choose the best network and loss; runs 6 and 7 follow
• true color v3: a new plan for a day and low-sun hybrid before any further training; the standard recipe stays on the site
• satellite radar global pilot: the four 72-hour training runs wait for a free A100 node (timing says 8 to 9 hours each); scoring on the cluster follows; the Europe test waits for 90 days of radar, about 17 December
• sharpening benchmark: the remaining 880 tiles are being fetched and built (about 110 processor-hours), then scored and released
• coincidence framework: build the one-year census package, about 4 node-hours on CPU nodes
• waiting in the queue: sharpening benchmark, building (allowed 14 h)
• waiting in the queue: sharpening benchmark, scoring (allowed 3 h)
• waiting in the queue: sharpening benchmark, release (allowed 4 h)