A neural network is mostly multiply-add

Strip away the names and a vision model like ResNet or YOLO is mostly one operation repeated billions of times: multiply two numbers and add the result to a running total. A single image through ResNet-50 is about 4 billion of them, or 8 billion operations if you count the multiply and the add separately. A batch of 32 is over 130 billion.

Every one of those operations is independent of most of the others. That's the whole reason GPUs exist for this work: if the work can be split up, you can do it all at once.

What's actually on the card

Take the cooler off a workstation GPU and you find a few things that matter:

  • The die, split into streaming multiprocessors (SMs). A modern card has over a hundred, each with many small cores and its own fast scratch memory.
  • Memory chips around the die, connected by a very wide bus: hundreds of GB/s to over a TB/s. Feeding the cores is often the real limit, not the maths.
  • Power stages and chokes, because all those cores switching at once draw a lot of current.
  • The PCIe connector, the comparatively narrow road from the rest of the computer.

Tiles and waves

Each layer of the network becomes a large matrix multiply. The software cuts it into tiles, say 128 by 128 outputs, and hands one tile to each SM. When an SM finishes, it takes the next. With a few thousand tiles and around 140 SMs, the work runs in a dozen or so waves.

Press run in the figure above and you can watch it: every SM lights at once, wave after wave, while data streams from memory to the die and results stream back. Then press the one-core key and watch a single core work through the same tiles one at a time. It's the same maths, around 800 times slower.

Why it matters in production

This is why the same model can run at 5 frames per second on a laptop CPU and hundreds on a GPU. It's also why batching matters: a GPU running one image at a time leaves most of its SMs idle. In a multi-camera system, batching frames from several streams together is often the cheapest speed-up there is.

It's also why memory, not compute, is often the bottleneck. Moving data is expensive, so production pipelines keep work on the GPU from decode to inference to post-processing instead of copying frames back and forth.

What I tune when it's too slow

  • Batch size, to fill the SMs without blowing the latency budget.
  • Precision: FP16 or INT8 with TensorRT is often 2-4x faster than FP32 with little accuracy loss, checked against a test set.
  • Keeping frames on the GPU from decode to output, with tools like DeepStream.
  • The model itself: input size, architecture, and pruning, measured on the real hardware, not the spec sheet.