Part V · Ch. 27 — Super-Resolution, 2×

Part V · Chapter 27 of 29

Super-Resolution, 2×

A residual network that starts as bicubic and learns only what bicubic gets wrong


The problem

A cloud edge looks blocky when its source pixels cover a large patch of ground. Enlarging the picture fills more display pixels and creates no new measurements. Super-resolution learns a finer image from a coarser one. What does it add over an interpolation method that needs no training at all?

The same schematic cloud edge under 1 km and 0.5 km grids.
The same schematic cloud edge under 1 km and 0.5 km grids.

The visible-band task here brings coarser bands toward the red band’s finer sampling. Earlier runs worked at 4×, so each score below comes with its scale and band. Source §2.

The data

A training example degrades an observed band and asks the model to restore it. No measured finer blue-band truth exists, so this is a proxy task. For the blue band, the observed 1 km image is the high-resolution target (HR) and its box-averaged 2 km version is the low-resolution input (LR). Source §7. The target is the image we withheld from the model’s input, not a sharpened version of its own answer. The scale-2 tensor pair is (1, 64, 64) → (1, 128, 128). Earlier scale-4 evaluation used 64→256 spatial dimensions. Source §2.

The Himawari red training run had 41,400 training and 3,600 validation sequences across 6 held-out scan days. Its parameter count belongs to a scale-4 run: 1,515,265. The scale-2 blue model has 1,367,553 parameters. Source §2.

The model

Bicubic interpolation estimates intermediate values by a fixed local calculation. Residual learning trains a correction on top of that baseline. The baseline already covers most of the smooth structure, so the network works on the differences. The shared local filters come straight from the convolution chapter.

EDSRLite: convolution head, 16 residual blocks, pixel shuffle, and addition to bicubic. The final convolution starts at zero.
EDSRLite: convolution head, 16 residual blocks, pixel shuffle, and addition to bicubic. The final convolution starts at zero.

Pixel shuffle moves channel entries into finer spatial positions. It rearranges learned features; it looks up no missing observations. With the final convolution zero-initialized, the correction starts at zero, so the first output equals bicubic. Later updates can help or hurt. Source §2.

Where this comes from

The architecture is not novel: Lim et al.’s EDSR (2017) follows Dong et al.’s SRCNN (2016, TPAMI); Zhang et al.’s RCAN (2018) adds channel attention. Satellite-band super-resolution is already explored: White et al. (2024, AIES) use degrade-and-restore training on GOES-16 ABI bands. We found no dedicated peer-reviewed Himawari AHI super-resolution paper. Ours is the Himawari band corpus, loss-weight sweep, and blind pairwise judging protocol. Source §6.

The loss and why that loss

The same constructed edge crop at bicubic, at exactly +1 dB relative to bicubic, and at the target. The middle image scales the bicubic error toward the target; it is an illustration of the unit, not a trained output.
The same constructed edge crop at bicubic, at exactly +1 dB relative to bicubic, and at the target. The middle image scales the bicubic error toward the target; it is an illustration of the unit, not a trained output.

PSNR measures squared pixel disagreement. For peak value P and mean squared error M, $\mathrm{PSNR}=10\log_{10}(P^2/M)$ dB. With normalized, bounded pixels it runs from zero to infinity; an exact match has infinite score. Blue’s 39.758 dB versus bicubic 38.664 dB means smaller squared error, a gain of 1.094 dB. We did not measure an interval, so we don’t know how much that gain would move on new scans. PSNR rewards blur; we also track structural similarity and blind human choices. Source §2.

Worked example

For exactly +1 dB, $M_{\mathrm{model}}/M_{\mathrm{bicubic}}=10^{-1/10}\approx 0.7943$. Starting with error 1 gives error about 0.7943: a 20.6 percent reduction. The inverse ratio is about 1.259, not a 26 percent error reduction. Source §7.

A target edge, a blurred edge, and a shifted edge: local brightness alone cannot describe how their structures differ. These are constructed examples, not measured images.
A target edge, a blurred edge, and a shifted edge: local brightness alone cannot describe how their structures differ. These are constructed examples, not measured images.

SSIM compares brightness, contrast, and how local patterns vary together: $\mathrm{SSIM}=\mathrm{brightness\ agreement}\times\mathrm{contrast\ agreement}\times\mathrm{structure\ agreement}$, with range −1 to 1 and 1 an identical match. One selected validation tile scores 0.9660 against bicubic’s 0.9500: closer local structural agreement, not 96.6 percent correct pixels. One tile can’t settle which method is better in general. SSIM depends on its local windows and can miss convincing but false detail, so we keep the target comparison and blind judging beside it. Source §7.

$$L=L_1+0.5\,(1-\operatorname{SSIM})$$

In words: add absolute pixel error to half the structural-similarity shortfall.

The masked loss excludes invalid night and limb pixels. Structural comparisons use valid windows. Both model and bicubic must see the same target, mask, and scale. Source §2.

Training as it ran

The Himawari red run used batch size 17, constant learning rate 2e-4, half-precision autocasting, and 8 epochs. The run took 3.25 hours from start to finish; each training epoch took about 6.8 minutes of that. Source §2.

Per-epoch model and bicubic PSNR for the red, blue, and green runs, one panel per run.
Per-epoch model and bicubic PSNR for the red, blue, and green runs, one panel per run.

At the selected epochs, red reaches 34.902 versus 34.362 dB at epoch 6; blue reaches 39.758 versus 38.664 at epoch 14; green reaches 38.838 versus 37.752 at epoch 15. The gains are +0.541, +1.094, and +1.086 dB. They come from the unrounded scores, so the red gain differs from the subtraction in the last digit. Source §2.

Outputs

A brightness stretch changes appearance without changing stored reflectance. Stretch each panel separately and darker model output can look as bright as the target. The shared stretch stops a display adjustment from covering a prediction error.

Blue run, one validation tile, best epoch 14: stored LR input · bicubic · model · HR target. The model scores 37.99 dB PSNR and 0.9660 SSIM on this tile against bicubic’s 36.60 dB and 0.9500. That +1.39 dB gain is larger than the +1.094 dB average over the validation set because this tile has a lot of texture.
Blue run, one validation tile, best epoch 14: stored LR input · bicubic · model · HR target. The model scores 37.99 dB PSNR and 0.9660 SSIM on this tile against bicubic’s 36.60 dB and 0.9500. That +1.39 dB gain is larger than the +1.094 dB average over the validation set because this tile has a lot of texture.
Detail of the same tile, cropped where the squared error falls most. The printed scores describe the whole tile, not this crop.
Detail of the same tile, cropped where the squared error falls most. The printed scores describe the whole tile, not this crop.

Use the magnifier on a cloud edge, then switch between bicubic, the learned correction, and the observed proxy target. The flipbook below shows the same blue-band tile at every epoch. Its PSNR label comes from the held-out score table, not from a fresh calculation on the visible tile.

Screenshot of the blind pairwise judging page: two anonymous crops side by side.
The blind pairwise judging page.

The picture asks a judge to choose between anonymous crops. The Bradley–Terry model turns those choices into relative strengths: $p(A\text{ beats }B)=e^{s_A}/(e^{s_A}+e^{s_B})$, between zero and one. The strength scores s have no absolute scale; only their differences mean anything. We have not collected enough human verdicts to score the model or bicubic this way. Preferences can reward attractive false texture, so the observed target, PSNR, and SSIM stay beside the blind test. Bradley and Terry (1952) Source §2.

A learned correction on top of bicubic, aiming from one-kilometer pixels toward half a kilometer. And why more pixels are never more measurements.

What it is not

An interpolation baseline is also a debugging tool: the untrained residual network should reproduce it on the same input, to the last digit.

A paired evaluation holds the reconstruction method to observations instead of visual plausibility. Bicubic stays useful after training: it is the starting prediction, the comparison baseline, and the thing the residual gets added to. Keep those three roles apart. When the learned correction produces an artifact, the baseline is the concrete reference for saying what the network changed.

Where this shows up when you train

Read the task score and the kind of error a user notices. The reported gains are modest, and the model can’t invent sub-pixel truth. Night pixels are masked out of this evaluation, so a daytime score says nothing about an all-hours product. Source §2.

Run this yourself

trainkit upscale samples/tile.png --output upscale-output

The command writes run.json and model.json. Add --dry-run to preview the replacement targets. Keep your output away from the bundled book artifacts. The run records the data shapes, split, objective, baseline, model score, and elapsed time; the model export holds the learned weights themselves.

Worked example

The synthetic run scores 44.55 dB PSNR against 38.95 dB for the baseline, after about 29 seconds of training. Source §6.

The toy gain is $44.55-38.95\approx 5.60$ dB. This synthetic image is easier than the satellite task; its gain does not predict a satellite gain. We ran it once, so there is no interval on that gain. Source §6.

Repeat the run before you change the recipe. Then change one input or model choice and compare on the same held-out examples. A different random split moves a score even when the learning rule is untouched. Keeping the original baseline is what separates an improvement in the learner from an easier evaluation problem.

What you now know

  • Residual learning starts at bicubic and learns a correction.
  • Degrading an observed band supplies proxy targets, not measured finer blue-band truth.
  • PSNR and SSIM compare against bicubic; neither establishes human preference.
  • The blind judging interface exists, but a pooled human verdict does not.

Where we’re headed

The same questions carry into Nowcasting the Radar.