AI Accelerators: Architecture Overview

Rodolfo Azevedo

Institute of Computing, University of Campinas (UNICAMP), Brazil

rodolfo.azevedo@unicamp.br

http://www.ic.unicamp.br/~rodolfo/mo801

Goal of this class

Module 5, Class 1: from profile to architecture — what production accelerators look like and where your design fits.

You have a spec (M04A04) and a profiling baseline (M04A03). This class puts both in context: what does a real ML accelerator look like, and which ideas are directly applicable at the Tang Nano 9K scale?

At the end of this class, you should be able to:

• Describe the ML hardware taxonomy (CPU, FPGA, DSP, ASIC NPU) and position each on the flexibility–efficiency trade-off.
• Explain the design loop: profile → architecture → implement → measure → iterate.
• Apply the roofline model to predict the attainable performance of a given accelerator design.
• Compare your accelerator spec against representative TinyML solutions such as Ethos-U, GAP9, and MAX78000.

From profile to architecture: the design loop

Every step feeds back: if the measured speedup does not match the prediction, revisit the profiling data and the memory model.

The ML hardware zoo

Your design (4-wide SIMD, 27 MHz) targets the FPGA row — a proof of concept, not production. But the architecture ideas (MAC array, local buffers, tiling) are the same ideas inside the Ethos-U55.

Common architectural patterns in ML accelerators

1. MAC array: multipliers operating in parallel on pairs of values. Your design is a 1D array of 4 int8 MACs.

2. Systolic array: a 2D grid of MAC units where data flows between neighbors — each unit passes its partial result to the right and its input downward. Used in Google's TPU (256×256 int8 systolic array). Eliminates the need for a central memory read at every step.

3. SIMD / vector unit: a single instruction operates on multiple data lanes simultaneously. RISC-V "V" (RVV) extension adds up to 512-bit vector registers to an existing CPU pipeline — no separate peripheral needed.

4. Dataflow / streaming: activations stream through a pipeline of operators (conv → BN → ReLU) without writing intermediate results to DRAM. Minimizes memory traffic at the cost of fixed pipeline topology.

Your accelerator is closest to pattern 1 — a simple MAC array with local buffering — which is a good starting point before the complexity of patterns 2–4.

The systolic array: data flow without a central controller

A systolic array passes data between neighboring cells — each cell does one MAC and hands the result to its neighbor, without reading from a shared memory on every step:

Weight row 0: [w00] [w01] [w02] [w03]   ← loaded once, stationary
                 ↑     ↑     ↑     ↑
Activation:  [a0] → MAC → MAC → MAC → MAC → partial sum out
             [a1] → MAC → MAC → MAC → MAC → partial sum out
             [a2] → MAC → MAC → MAC → MAC → partial sum out

• Each cell: acc += a_in * w; passes a_in to its right neighbor, passes acc down.
• After N cycles, the rightmost column accumulates one full dot product per row.
• No central memory read per cycle — activations flow through, weights are stationary. This is why Google's TPU (256×256 systolic array) is so efficient: the memory bandwidth requirement drops dramatically.

Your design vs. systolic:

• Your 4-wide MAC array reads 4 activations from DMEM and 4 weights from BRAM every cycle — one memory read per MAC. Simple, but bandwidth-limited.
• A systolic array reads weights once per layer and streams activations through — reuse factor = number of columns.

The systolic idea is directly applicable to your Project 3: you could load one column of weights into each MAC unit and stream activations from left to right — a mini systolic array, 4 cells wide.

The Arm Ethos-U55: a real TinyML NPU

The Ethos-U55 is Arm's production TinyML accelerator, targeting Cortex-M systems:

• Interface: AHB bus (same concept as your Project 2 bus — memory-mapped registers + DMA descriptor).
• Datapath: 32–512 int8 MAC units (configurable at tape-out time).
• Memory model: a dedicated SRAM (SRAM0) for activation scratch, plus access to system flash for weights.
• Supported ops: Conv2D, DepthwiseConv, FC, Pooling, Elementwise — enough for tiny_conv and many other model topologies.
• Programming model: the CPU writes a command stream to Ethos-U55's command queue; the NPU fetches it and executes, then raises an interrupt when done.

What is the same as your design:

• Memory-mapped control interface.
• Separate weight buffer (NPU's SRAM0 ≈ your local BRAM).
• The CPU is the master; the NPU is the slave.

What is different:

• Ethos-U55 uses a DMA engine to self-fetch weights from flash — you require the CPU to explicitly write the weight buffer.
• Ethos-U55 supports a full operator graph (compiled by Vela); yours supports one operation at a time.

The RISC-V "V" (RVV) extension: ISA-level SIMD

Instead of a separate peripheral, RVV adds vector registers to the CPU:

# RVV assembly for a length-64 int8 dot product
vsetvli t0, a2, e8, m1, ta, ma  # set vector length (up to VLEN/8 elements)
vle8.v  v0, (a0)                 # load 8-element vector from a[]
vle8.v  v1, (a1)                 # load 8-element vector from b[]
vmul.vv v2, v0, v1               # element-wise multiply
vredsum.vs v3, v2, v3            # reduce: v3[0] += sum(v2)

A single vmul.vv instruction multiplies 8 (or 16 or 32) pairs in one cycle. The vector unit lives inside the CPU pipeline — no bus transactions, no polling.

Why we did not build RVV for this course: implementing a vector unit requires changes to the register file, decode, and datapath that would double the complexity of Project 1. A separate memory-mapped accelerator decouples the hardware design from the ISA — easier to implement, easier to reason about, and directly teachable.

Schaumont's interface design framework

From A Practical Introduction to Hardware/Software Codesign (Ch. 11–12):

Step 1 — Programmer's model: what does software see? (register map, semantics of each register, timing of CONTROL→STATUS→result)

Step 2 — Hardware interface: what signals cross the boundary? (bus protocol, address decode, ready/valid handshake)

Step 3 — Datapath: what computation happens inside? (MAC array, accumulator, saturation)

Step 4 — Control: what FSM drives the datapath? (IDLE → LOAD → COMPUTE → DONE)

This order matters: define the software interface before the hardware, so you know what you are building toward. In M04A04 we did Step 1 (register map). Today and M05A02 we do Steps 2–4.

Schaumont's framework applied — KWS accelerator

Walking through all 4 steps for the KWS Conv2D accelerator:

Step 1 — Programmer's model (done in M04A04):

• Software writes INPUT_ADDR, LENGTH, then sets CONTROL[0] (start).
• Software polls STATUS[0] (done). Reads result from output buffer.

Step 2 — Hardware interface:

• Bus interface: memory-mapped, sel_accel from address decoder.
• DMEM interface: accelerator drives dmem_addr and dmem_re; receives dmem_rdata and dmem_ready.
• No DMA — the CPU stalls while the accelerator fetches activations.

Step 3 — Datapath:

• 4 parallel int8 MAC units, each with a 32-bit accumulator.
• Local 64-entry int8 BRAM for weights; 4-wide read port (32 bits/cycle).
• requant module: bias + (acc × mult) >> shift + zero_point, clamped to int8.

Step 4 — Control FSM:

• IDLE → LOAD → COMPUTE → DONE
• LOAD: fetch one 32-bit word from DMEM per cycle (4 activations at once).
• COMPUTE: feed 4 activations + 4 weights to MAC array; advance weight address.

One key property of this framework: Steps 1–2 are the contract between hardware and software. Once they are agreed upon, Steps 3–4 can be implemented independently — and the testbench (Step 2 simulator) is what validates Step 3–4.

In-class exercise — design trade-off

Your team has budget for one architectural improvement to Project 3. Which do you choose?

Option A: Double the MAC array width from 4 to 8 lanes.

• Cost: 4 more DSP blocks (you have 15 remaining after Project 2's 5).
• Benefit: 2× throughput if weight BRAM can supply 8 values/cycle.

Option B: Add a DMA engine so the CPU does not stall during activation fetch.

• Cost: ~200 LUTs, 1 BRAM, design complexity.
• Benefit: CPU can run the FC layer while the accelerator fetches Conv2D activations.

Option C: Add a second weight BRAM row (double buffer): load row while computing row .

• Cost: 1 more BRAM.
• Benefit: eliminate LOAD latency — overlap LOAD and COMPUTE.

Questions (10 min):

1. Which option gives the largest single-number speedup on the Conv2D layer?
2. Which option gives the largest system-wide speedup (considering Amdahl's Law)?
3. Which option is most important if the design is currently memory-bandwidth-bound?

Expected: (1) A, if BRAM bandwidth matches. (2) B or C, since they reduce overhead that Amdahl's law can't hide. (3) C — double buffering eliminates the LOAD stall, which is a memory-bandwidth bottleneck.

What "verified against the C kernel" means

Before integration, verify the accelerator in isolation:

// In the Verilator testbench:
// 1. Write the same input vector and weight matrix as the C test vector
// 2. Assert CONTROL[0]
// 3. Wait for STATUS[0] == 1
// 4. Read the output
// 5. Compare bit-for-bit against the C kernel output

if (hw_output[i] != c_kernel_output[i]) begin
    $error("Mismatch at output[%0d]: got %0d, expected %0d",
            i, hw_output[i], c_kernel_output[i]);
end

Bit-exact match is the requirement. If the C kernel outputs 42 for a given input, your accelerator must output 42 — not 41, not 43. Quantization is brittle: a 1-ULP error in the requantization step propagates and changes the classification.

Next class

Implementing the Accelerator: the int8 MAC unit, accumulator, local BRAM buffer, and the IDLE→LOAD→COMPUTE→DONE control FSM — all in SystemVerilog, with a Verilator testbench that verifies against the C kernel.