Off-the-Shelf Alternatives & Industry Context

Rodolfo Azevedo

Institute of Computing, University of Campinas (UNICAMP), Brazil

rodolfo.azevedo@unicamp.br

http://www.ic.unicamp.br/~rodolfo/mo801

Goal of this class

Module 6, Class 1: situating your design in the broader ML hardware landscape.

You built something real. Now we step back and ask: where does it fit? What exists commercially that solves the same problem, and how does it differ from what you built — in interface, architecture, and design methodology?

At the end of this class, you should be able to:

• Describe the key commercial TinyML solutions (Ethos-U, GAP9, MAX78000, dedicated NPUs) and their architectural trade-offs.
• Explain the RISC-V "V" (RVV) extension and how it achieves SIMD-style ML acceleration without custom hardware.
• Position your Project 3 accelerator within the broader flexibility–efficiency design space.
• Evaluate when an FPGA prototype, an ASIC tape-out, or an off-the-shelf NPU is the right engineering choice.

The ML hardware taxonomy

The trade-off is always flexibility vs. efficiency:

• A CPU can run any program — your RV32I runs KWS, but also bubble sort and Fibonacci.
• An ASIC NPU runs only ML inference — but does so 100× more efficiently.
• An FPGA sits in between: reprogrammable (so you can tape-out Project 3 in a week), but with 10–50× overhead versus a custom ASIC.

For TinyML at < 1 mW, the market has converged on small fixed-function NPUs (Ethos-U, GAP9, MAX78000) rather than general-purpose CPUs.

RISC-V "V" extension (RVV): ISA-level SIMD

RVV adds a configurable-width vector register file to any RISC-V CPU. The vector length VLEN (typically 128–512 bits) determines how many elements fit in one register.

# Compute 64-element int8 dot product with RVV
vsetvli  t0, a2, e8, m1, ta, ma   # set vector length for int8
vle8.v   v0, (a0)                  # load 64 input values
vle8.v   v1, (a1)                  # load 64 weight values
vwmul.vv v2, v0, v1                # widening multiply: int8 → int16
vwredsum.vs v3, v2, v4             # reduce to scalar

For VLEN=512: one vwmul.vv multiplies 64 pairs simultaneously — an entire inner-loop iteration of our Conv2D in 2 instructions.

Why we did not build RVV: it requires changes to the register file (8 vector registers of VLEN bits each = 512 bytes at VLEN=512), the decode logic, and the datapath. Far more complex than a separate peripheral. But for production hardware, RVV eliminates the bus-overhead penalty that memory-mapped peripherals pay.

Arm Ethos-U: production TinyML NPU

The Ethos-U series (U55, U65) is Arm's answer to TinyML on Cortex-M systems:

Architecture:

• 32–512 MAC units (configurable at IP integration time)
• Internal SRAM (32–2048 KB) for activation scratch
• A command-stream interface: the CPU writes a compiled operator graph descriptor; Ethos-U fetches and executes it autonomously

Compilation toolchain (Vela):

vela --accelerator-config ethos-u55-256 model.tflite
# Produces: command_stream.bin (fed to Ethos-U at runtime)

Key difference from your design: Ethos-U accelerates the entire inference graph with one command. Your accelerator replaces one layer at a time, with the CPU orchestrating each call. The Ethos-U approach amortizes bus overhead across the whole model — your approach pays it per-layer.

GAP9: a RISC-V cluster for TinyML

GreenWaves' GAP9 takes a different approach: instead of a fixed-function NPU, it uses a cluster of 9 RV32IMC cores sharing a scratchpad memory.

For Conv2D: 8 cores each handle 1/8 of the output channels in parallel. No custom hardware — just 8 general-purpose cores.

Lesson: at constrained area budgets, many small CPUs can compete with fixed-function accelerators. The RISC-V ecosystem makes this practical — you can tape out a cluster of RV32I cores for the cost of the IP licenses.

MAX78000: CNN accelerator + microcontroller

Maxim's MAX78000 integrates a CNN hardware accelerator alongside an Arm Cortex-M4:

• 64 parallel MACs for convolutions
• On-chip weight memory (442 KB) — the entire tiny_conv model fits (~19 KB)
• Only supports a fixed set of CNN topologies (no RNNs, no attention)
• Power: < 300 µW for KWS inference

What it gets right that your design does not:

1. Weights stored on-chip permanently — no BRAM loading overhead per inference.
2. The CNN accelerator pipelines all layers, not just one type.
3. The accelerator connects to its own SRAM, not a shared bus with the CPU.

What your design teaches that the MAX78000 does not:

1. You understand every gate, every cycle.
2. You chose the interface and debugged the timing.
3. You measured the bottleneck and made a principled design decision.

The "not-invented-here" lesson

When should you build from scratch vs. use an existing solution?

For this course: building from scratch was the right choice — not because the result is better than Ethos-U, but because you now understand what Ethos-U is doing and why it makes the design decisions it does. That understanding is not available from documentation alone.

Your design on the roofline — vs. production

Situating all designs on the same compute/bandwidth plane (27 MHz clock, int8):

Key observation: all production designs place their weights in on-chip SRAM, making them compute-bound. Your design with the BRAM weight buffer does the same — at a much smaller scale.

The gap between your design and Ethos-U55 is quantitative, not qualitative. The same architectural ideas (parallel MACs + local weight storage) scale from 4 MACs to 256 MACs. The difference is chip area, not conceptual innovation.

Open-source tools for going further

The path from this course to a real chip: Chisel → OpenLane2 → efabless tapeout. The tools exist; the barrier is now knowledge, not money.

The path from this course to a chip

The tools for a free ASIC tapeout exist today. What you would need beyond this course:

Project 3 SV  →  Chisel (or keep SV)  →  OpenLane2 synthesis
                                           ↓
                                    GDS on Sky130 (130 nm)
                                           ↓
                                  efabless chipIgnite submission
                                           ↓
                              Free fabrication via Google-sponsored run
                                           ↓
                              Real silicon in ~6 months

What would change at tape-out:

• Your BRAM becomes SRAM macros (different timing, need Liberty files).
• Your DSP blocks become standard cells (multipliers are just gates).
• Timing closure at 100 MHz is harder — critical path analysis becomes central.
• The bus interface stays the same: AXI4-Lite is the industry standard equivalent of your valid/ready protocol.

What stays the same:

• The architecture: 4-wide int8 MAC array, IDLE→LOAD→COMPUTE→DONE FSM, register map.
• The C software driver: volatile pointer access to a fixed memory-mapped address.
• The verification approach: Verilator testbench comparing against the C kernel.

The gap between "working on FPGA" and "working on ASIC" is significant, but the design knowledge is transferable. You already have the architecture and the verification methodology — ASIC tape-out is mostly engineering execution from there.

In-class discussion — if you could start over

You have now built a complete TinyML SoC, from RV32I core to hardware accelerator. If you could start Project 3 over with 4 weeks (instead of the actual schedule), what would you change?

Consider these axes:

• Interface style: would you use DMA instead of memory-mapped polling? Under what conditions would that help?
• Datapath width: is 4 lanes the right choice for your specific layer dimensions? What would 8 lanes cost vs. gain?
• Weight loading: the current design requires the CPU to write weights to the BRAM one byte at a time. What mechanism would eliminate this bottleneck?
• Layer coverage: the accelerator only handles Conv2D. Would extending it to DepthwiseConv or FC change the system speedup significantly?

Discuss in pairs (10 min), then each pair reports one change and quantifies the expected benefit using Amdahl's Law.

There are no wrong answers here — the point is to apply the tools from the course (roofline, Amdahl, arithmetic intensity) to a design decision you understand from the inside.

Next class

Final Presentations (Part 1): each pair presents their Project 3 results — design choices, speedup numbers, roofline position, and demo on the Tang Nano 9K.