Roofline, Arithmetic Intensity & Alternatives

Rodolfo Azevedo

Institute of Computing, University of Campinas (UNICAMP), Brazil

rodolfo.azevedo@unicamp.br

http://www.ic.unicamp.br/~rodolfo/mo801

Goal of this class

Module 5, Class 4: understanding your speedup and what production systems do instead.

Your accelerator is measured. Now the question is: does the number make sense? The roofline model gives a principled answer — and reveals whether spending more effort on the datapath would help or whether the memory bottleneck is the real limit.

At the end of this class, you should be able to:

• Apply the roofline model to your measured accelerator performance and locate your design on the roofline chart.
• Compute arithmetic intensity (MACs per byte) for the KWS workload and identify the limiting ceiling.
• Explain whether your design is compute-bound or memory-bound and what architectural change would improve it.
• Describe what the RISC-V "V" extension (RVV) and fixed-function NPUs offer compared to your FPGA design.

The roofline model

Two ceilings limit performance:

• Peak compute (MACs/cycle): determined by the number of MAC units.
• Peak bandwidth (bytes/cycle): determined by the memory interface width.
• Arithmetic intensity (AI) (MACs/byte): a property of the algorithm and data movement.

Computing your accelerator's arithmetic intensity

For the Conv2D inner loop (K×K×C_in MACs per output, C_out output channels, 4-wide MAC array):

Bytes read from DMEM per spatial position:

• Input activation window: KH×KW×C_in bytes (read once per spatial position)
• Weights: loaded into BRAM by CPU before computation — cost amortized over all H_out×W_out positions. Per spatial position: 0 bytes from DMEM (weights come from local BRAM).

Effective AI (per spatial position, all C_out output channels computed):

Compare to software (no local buffer, weights re-fetched each time):

The local BRAM buffer increases AI by C_out× (e.g., 64× if C_out=64). The MAC array is now compute-bound (operating near the compute roof), not memory-bound.

Where your design sits on the roofline

The key insight: adding more MAC units without a local buffer would have given zero speedup. The BRAM buffer was the architectural decision that mattered most.

The latency-bound region: a third category

The classic roofline model has two regions: memory-bound (left of the knee) and compute-bound (right). In practice, a third region exists:

Latency-bound: the accelerator is neither saturating memory bandwidth nor compute units — it is stalled waiting for a fixed-latency operation (e.g., a single BRAM read that takes 2 cycles, a bus round-trip that takes 4 cycles) even though neither bandwidth nor FLOP rate is the bottleneck.

Performance (MACs/cycle)
    │
  4 │ ───────────────────────────── ← compute roof
    │                              /
    │        latency-bound zone   /  ← accelerator stalls on bus round-trip
  1 │ ── ── ── ── ●              /    even though bandwidth is not saturated
    │             ↑             /
    │        "should be here"  /
    │                         /
    └──────────────────────────────→  AI (MACs/byte)

How to identify it: if your measured performance is below both the memory bandwidth line AND the compute roof, but increasing data reuse (buffering) doesn't help, the bottleneck is latency, not bandwidth.

In the course reference implementation (docs/stage9-writeup.md): the first accelerator version was latency-bound because each MAC required a bus round-trip (write rs1, write rs2, read result). Adding a local BRAM buffer moved it to the compute-bound region.

Mitigation: pipeline the bus interface, add a request queue, or redesign the memory interface to amortize latency over a burst of operations.

Amdahl's Law: why system speedup < layer speedup

Accelerating Conv2D to 4× gives ~3.1× system speedup (assuming Conv2D is 90% of runtime):

System = Accelerated part + Non-accelerated part
       = 90% at 4×        + 10% unchanged

Speedup = 1 / (0.10 + 0.90/4) = 1 / 0.325 ≈ 3.1×

The fixed 10% (FC layer, bus overhead, requantization) sets an asymptotic ceiling of 10× regardless of how fast Conv2D gets. With actual measured percentages from your profiling, substitute the real numbers.

This is the honest engineering answer: 4× layer speedup, ~3× system speedup. More MAC units would help only if the layer stays compute-bound (which depends on the weight buffer being large enough).

Off-the-shelf alternative: Arm Ethos-U55

The Ethos-U55 is 148× faster with 20× less power. But it required a full design team, a silicon process, and years of engineering. Your design was built in one semester from first principles.

RISC-V Vector Extension (RVV): the ISA alternative

RVV integrates SIMD into the CPU pipeline — no separate peripheral:

For this course, the memory-mapped approach was the right pedagogical choice — it teaches the full stack (hardware interface design, software drivers, bus integration) in a way that RVV shortcuts. In production, RVV wins on almost every metric.

What you would do differently with more time

Honest retrospective on the design choices:

1. Wider MAC array (16-wide): the device budget allows it; the main constraint was implementation time.
2. DMA for weight loading: currently the CPU writes weights to BRAM one byte at a time (slow). A DMA engine would burst-transfer weights from DMEM in 1/32 the time.
3. Accelerate multiple layers: with tiny_conv's simple structure, once Conv2D is accelerated the FC layer becomes the remaining bottleneck — though its share is small, the next improvement is pipeline overlap between layers.
4. Double-buffering: overlap LOAD (next activation) with COMPUTE (current) — doubles throughput with no extra hardware cost.
5. Fixed-point requantization multiplier from BRAM: currently a constant; making it configurable per-layer would allow accelerating multiple different models.

The slides that follow cover security, energy, and reproducibility. Security corners are required reading. Energy and reproducibility corners are optional enrichment.

Security corner: accelerator with unrestricted memory access

Your accelerator sits on the memory-mapped bus and, from the CPU's perspective, is just another peripheral. But what if the accelerator had DMA (Direct Memory Access) capability?

A DMA-capable accelerator can read and write any memory address independently of the CPU:

• Data exfiltration: a compromised accelerator firmware/configuration could read DMEM and send sensitive data to an output peripheral.
• Code injection: a DMA write to IMEM is equivalent to the bootloader attack from M03 — but now it bypasses even PMP, because PMP only governs CPU-initiated accesses.

The missing defense: IOMMU

An IOMMU (I/O Memory Management Unit) does for peripherals what PMP/MMU does for the CPU:

• Each bus master gets its own set of permitted address ranges.
• A DMA request from the accelerator to IMEM would be blocked unless explicitly allowed.

Your Project 3 accelerator does not have DMA — the CPU moves all data via load/store. But production accelerators (GPU, NPU, network cards) always need an IOMMU.

Security corner: power side-channel attacks

Your 4-wide MAC array consumes different amounts of power depending on the input data and weights:

• Multiplying 127 × 127 toggles more transistor gates than 0 × 0.
• An attacker with a current probe on the power supply can record power traces during inference and use Differential Power Analysis (DPA) to recover the model weights.

Why this matters

• Model weights are intellectual property — a company that spent millions training a model does not want it extracted by measuring power consumption of the edge device.
• Input privacy — power traces can also reveal what audio sample was being classified.

Mitigations

These are not theoretical — power side-channel extraction of neural network weights has been demonstrated on FPGAs in published research (Batina et al., CHES 2019).

Project 3 security question: "Your accelerator has direct access to the bus. Describe a scenario where this could be exploited if DMA were added, and one hardware mechanism that would restrict it."

Energy corner: the energy roofline (Optional)

Just as the performance roofline plots throughput vs arithmetic intensity, an energy roofline plots energy efficiency (inferences/J) vs arithmetic intensity:

• Memory-bound region: energy dominated by memory access (each byte read costs ~10 pJ in SRAM, ~1 nJ in DRAM).
• Compute-bound region: energy dominated by MAC operations (~0.5 pJ per int8 MAC in 55 nm).

Your accelerator's BRAM buffer moves the design from "fetching weights from DMEM every cycle" (memory-bound) to "reading weights from local BRAM" (compute-bound):

The voltage scaling opportunity

Recall . Your FPGA runs at 1.2V — an ASIC at 0.6V would use 4× less dynamic power for the same logic.

Combined with the 10× process advantage (55 nm → 8 nm), this explains why production NPUs achieve ~1000× better energy efficiency than your FPGA design — and why your FPGA design still beats cloud inference by ~400×.

Energy corner: total system energy — what the numbers mean

Putting it all together for one KWS inference at 27 MHz:

Static power is 39% of total — and you cannot reduce it without a different FPGA or voltage scaling. This is a fundamental limit of the platform.

Reproducibility note: your final project archive (Optional)

Your Project 3 repository should allow the instructor to reproduce your results from a single make command:

Required structure

├── rtl/                    # all SystemVerilog source
│   ├── cpu/                # Project 1 core
│   ├── bus/                # Project 2 bus + peripherals
│   └── accel/              # Project 3 accelerator
├── tb/                     # all Verilator testbenches
│   ├── Makefile            # `make test` runs all tests
│   └── test_*.cpp
├── sw/                     # C programs
│   ├── bootloader/
│   ├── kws/
│   └── Makefile            # `make` cross-compiles all
├── constraints/            # pin constraints (.cst)
├── Makefile                # top-level: `make synth`, `make load`
├── results/                # synthesis reports, cycle counts
│   ├── utilization.txt
│   ├── timing.txt
│   └── speedup.csv
└── README.md               # memory map, tool versions, how to reproduce

The reproducibility checklist

• [ ] make test passes all Verilator tests on a clean clone
• [ ] make synth produces a bitstream with no timing violations
• [ ] make load programs the Tang Nano 9K
• [ ] results/speedup.csv matches the numbers in your presentation
• [ ] README lists exact OSS CAD Suite version and GCC version
• [ ] No binary blobs committed (all generated files in .gitignore)

Next module

Module 6 — Closing: off-the-shelf alternatives in depth, final presentations, and the full retrospective — from gates to AI inference in one semester.