Part I · Ch. 4 — Why a GPU, Not a CPU

Part I · Chapter 4 of 29

Why a GPU, Not a CPU

Follow the arithmetic to find the parallel work


Which answer has to wait?

Training jobs run on graphics processors because of a property of the arithmetic, not the hardware. It shows up inside a dot product. Draw which answers depend on which earlier answers, and you can see which calculations run together and which have to wait.

Use our output matrix [[1,−1,0],[0,1,1]] on input (1,1,0). The two output sums are 0 and 1. Output one doesn’t need output two’s answer, and output two doesn’t need output one’s. That makes the outputs parallel work: independent calculations can run at the same time when there is hardware free to run them.

A shared 2 by 3 weight matrix and input (1,1,0) feed two separate multiply-and-add chains. Their output sums are 0 and 1, with no dependency edge between the chains.
Output cells are independent, while each dot product still combines its terms.

A data dependency is a calculation that needs a result from an earlier one. Each output sum still depends on its own products. The products can be calculated independently and then combined. Independence between output cells doesn’t remove the need to combine the terms inside each dot product.

Mark one output cell and trace only its incoming edges. Its three products belong to that output. Mark a second output and trace its edges: the input data are shared, but the accumulated result is separate. Shared inputs and independent outputs coexist as long as those inputs stay available and unchanged during the calculation.

Count the arithmetic before timing it

Two outputs and three inputs need 2×3=6 products and 2×(3−1)=4 additions before bias. A FLOP is one floating-point arithmetic operation, and a multiply-add is usually counted as two. For batch size B, an approximate layer count is 2×n_out×n_in×B. It is an operation count, not a number of seconds.

The approximation treats each output contribution as a multiply-add. The addition count is a little different, because the first product has nothing to be added to and a bias adds one more operation. Over large products those differences are small, but the convention still has to be stated when comparing rates. Hardware may run one instruction that multiplies and adds, while the FLOP count still charges it two.

Some work remains a chain

Other computations have a very different dependency picture. Take a loop that keeps a running total. It starts at zero, reads an entry, updates its state, then reads the next entry. Each iteration needs the state the previous one produced. Adding hardware lanes doesn’t remove that edge from the program.

Adding 1, 2, and 3 sequentially produces states 0,1,3,6; a prefix tree groups the associative sums. A separate sign-dependent chain 0 → 1 → −2 → −4 still requires each preceding state.
A serial loop may hide parallel work, but dependencies must be checked before rewriting it.

For entries [1,2,3], the states are 0→1→3→6. The third state uses the second. Addition has a useful property: you can group the terms differently and get the same sum in exact arithmetic. A parallel prefix algorithm uses that to compute all the partial sums on a different graph. Floating-point regrouping can change small rounding details, so equivalence needs a stated tolerance.

A branch that depends on the previous state blocks the same rearrangement. If the sign of the accumulated value decides whether to add or subtract the next input, the later choice isn’t known until the earlier state is. Before calling a loop parallel, check the mathematical dependency, not just the loop syntax on the screen.

Latency is the elapsed time to finish one operation. Throughput is how much work finishes per unit time. A processor can have high throughput and still take longer on one tiny request, especially when setup is involved. A kernel is a device program launched across many work items. Launching it costs something even when the arithmetic inside is short.

Memory bandwidth is bytes moved per unit time. VRAM means video random-access memory, the storage on a graphics device. Inputs have to be where the calculation runs. If they start in CPU memory, copying them to the device can dominate a small calculation. If a training loop keeps data and parameters on the device, later operations reuse them there.

Read the measurement, including copying

Now read a real benchmark. We timed matrix-vector products and square matrix products at a series of sizes. We did not time a neural-network batch of sixty-four examples or a complete training epoch. Its horizontal axis is the side length of a square matrix.

Log-log charts plot square side length n against vector and matrix elapsed seconds for two NumPy thread settings and CUDA with and without transfers. The matrix transfer-inclusive crossover is marked at n = 1024; the vector chart has none measured.
Counting transfers, the square matrix product first beats both CPU modes at size 1024; the vector product never does at the sizes we tested.

Each panel shows four recorded timing modes: NumPy with one configured thread, NumPy with eight configured threads, CUDA compute, and CUDA with transfers. Both NumPy modes were restricted to four CPU cores. CUDA is the programming platform used for these graphics-device measurements. Its compute timing and its transfer-inclusive timing answer different questions about where the operation starts and stops.

The transfer-inclusive square matrix product first beats both measured CPU modes at size 1024. No tested matrix-vector size beats both CPU modes after transfer. Whether a larger one would, we didn’t test.

The axes are logarithmic so both small and large sizes stay visible. Equal spacing means equal multiplicative ratios, not equal additions.

These results hold for the shapes, dtype, hardware, thread settings, and timing boundary we used. They don’t mean that any operation over 1024 entries belongs on a graphics processor.

Change the amount of independent work

The simulator below counts scheduling steps, not seconds. Start with one lane and the fixed two-by-three layer. Step through the products and each output’s additions. Then add a second lane and watch the two output reductions advance on their own. It shows the dependencies; it doesn’t model how a real GPU schedules its work.

Parallelism follows data dependencies, and copying changes the timing result.

See it move

Change the batch size and keep the shared weight table. More examples mean more independent output cells, each with its own reduction. Raise the lane count and watch for the point where extra lanes stop helping this small workload. With fewer independent jobs than lanes, some lanes have nothing to do.

Open the measured pane on its own and pick one of the benchmark’s square sizes. Those values are seconds; the scheduler’s values are ticks. We didn’t time a rectangular layer with an added bias, so the pane has no exact match for it.

Where this shows up when you train

A training program usually gains from keeping weights, intermediate values, and batches on the device across many operations. That spreads the initial transfer cost over more useful work. It still has to feed the device, respect dependencies between layers, and avoid building arrays it doesn’t need.

The next step is changing how the work is written down. Some programs hide independent calculations inside a serial-looking loop. Three small examples come next, rewritten so their array structure is in the open and a numerical library can schedule the work.

What you now know

  • Independent outputs can be computed concurrently.
  • Transfers and launch overhead can dominate small operations.
  • A measured crossover belongs to a specified operation and machine.

Where we’re headed

Next: look inside an ordinary loop and separate the mathematical dependencies from the order the program imposed on them. Continue to the next chapter.