Skip to content

Design Space: This Course's Path Is One Point Among Many

This is a "Topics" course, and the path through Project 1 -> Project 2 -> Project 3 is one concrete way to do hardware/software codesign - chosen because it is buildable from scratch in a semester on a Tang Nano 9K. At several points, real systems make different choices. This page collects those choice points, so we can come back to them after each project: "here's what we did, here's what else is done in practice, and why."

These are also good seeds for seminar topics in Weeks 9-15.

1. How the accelerator talks to the CPU

In Project 2/3, the accelerator is a memory-mapped peripheral: the CPU writes operands/config to registers, triggers the computation, and polls/reads results - the same pattern as the UART or timer. This is simple and matches Schaumont's "hardware interface" model (Ch. 11-12).

flowchart TB
    subgraph A["A: Memory-mapped registers (our choice)"]
        C1[CPU] -- "store/load to fixed addresses" --- P1[Accelerator regs]
    end
    subgraph B["B: Shared/unified memory + DMA"]
        C2[CPU] -- "writes descriptor" --- D[DMA engine]
        D -- "reads/writes" --- M[(Shared RAM)]
        ACC[Accelerator] -- "reads/writes" --- M
    end
    subgraph CC["C: Custom instruction (tightly coupled)"]
        C3[CPU pipeline] -- "operands in registers" --- FU[Accelerator as a functional unit]
    end

Other points in this space:

  • ISA extension / tightly-coupled functional unit - what we did for Zmmul and CMAC in Project 2. Lowest latency, but the new functional unit must fit the CPU's timing and register-file ports; hard to scale to large operand sets (an int8 conv layer's weights don't fit in registers).
  • Shared/unified memory with DMA - CPU and accelerator both address the same RAM; the CPU just hands over a pointer + size, and a DMA engine streams data to/from the accelerator. This avoids copying data through CPU registers, but adds the complexity of a DMA controller and (in multi-master systems) coherence/arbitration.
  • Streaming/FIFO interface - accelerator consumes/produces a stream (common for DSP-like pipelines: filters, FFTs). Great when data is processed once, in order; less convenient for data reused across many output computations (like conv weight reuse).

Why we chose memory-mapped registers: it is the simplest model to design, verify, and debug with the tools available (a few STORE/LOAD instructions and a logic analyzer/waveform are enough), and it directly extends the bus and peripherals already built in Project 2. The trade-off - data movement through CPU loads/stores - is real, and is the topic of the next section.

2. Unified memory vs. explicit data transfer

Our accelerator's inputs (a tile of the MFCC feature map and the relevant weights) must get from the CPU's data memory into the accelerator's registers/buffers via explicit STORE instructions, and results come back via LOAD. This is explicit data transfer.

Many real systems instead give the accelerator a port directly into the same memory the CPU uses (unified/shared memory), so "transferring" data is just handing over an address. This removes the CPU-mediated copy, but:

  • it requires an arbiter (or a multi-ported memory) so CPU and accelerator can both access RAM without corrupting each other's accesses;
  • it raises the question of coherence - if the CPU writes data, then the accelerator reads it, both need to agree on memory ordering;
  • on our Tang Nano 9K, the natural "shared memory" candidate is the on-board PSRAM, which has its own latency/bandwidth characteristics (see next section).

Discuss in Project 3: if your accelerator processes more than one tile per invocation, is the overhead of STORE-ing each input value (one instruction per value) becoming a bottleneck itself? If so, you're looking at exactly this trade-off in miniature.

3. The memory bottleneck

A useful way to reason about whether an operation is compute-bound or memory-bound is the arithmetic intensity: MACs performed per byte moved from/to memory. The tiny_conv Conv2D layer reuses weights across all output spatial positions but still needs to stream activations in and out.

  • If your accelerator can do many MACs per cycle but the bus/PSRAM can only deliver a few bytes per cycle, the accelerator will sit idle waiting for data - the speedup you measure in Project 3 will be capped by memory bandwidth, not by your datapath's peak throughput.
  • This is exactly Schaumont's distinction between computation-constrained and communication-constrained design (Ch. 9): for a communication-constrained kernel, optimizing the datapath further gives no speedup - you must reduce data movement (e.g., buffer/reuse data on-chip, increase burst sizes) instead.

Discuss in Project 3: when you measure your speedup, also estimate how many bytes per MAC your accelerator moves across the bus. If your speedup is much lower than your datapath's theoretical parallelism would suggest, memory bandwidth is very likely why.

4. "Just use a processor with built-in acceleration"

Building a custom accelerator from scratch is the point of this course - but it is worth being explicit that production systems usually don't. Alternatives that exist "off the shelf":

  • Vector extensions (e.g., RISC-V "V" - RVV): the ISA itself gains instructions that operate on vectors of data, amortizing instruction fetch/decode over many MACs, without a separate memory-mapped accelerator.
  • Dedicated NPUs/DSPs integrated next to the CPU (e.g., Arm Ethos-U, various microcontroller vendors' "AI accelerators"), typically with their own local memory and a fixed instruction set for common ML ops (conv, pooling, activation).
  • GPUs / many-core arrays for larger devices - not relevant at our scale, but the same compute-vs-communication trade-offs apply at larger granularity.

Discuss after Project 3: now that you've built one accelerator by hand, compare your interface and datapath choices to one of these real designs (a good seminar topic for Weeks 12-15). What did they do differently, and why might that make sense at their scale?

5. Does it all fit on the FPGA?

The Tang Nano 9K (GW1NR-9C) has roughly 8,640 LUTs, 6,480 flip-flops, and ~468 Kbit of block RAM, plus 64 Mbit of external PSRAM. Across the three projects, the design grows to include:

  • the RV32I core (Project 1),
  • the bus, UART, timer/GPIO, Zicsr/Zicntr, Zmmul, and CMAC (Project 2),
  • instruction/data memory for the inference kernel and its ~20K int8 weights,
  • the accelerator's datapath and any local buffers (Project 3).

This is exactly why a full reference implementation must exist before the projects are assigned (see below) - to confirm the complete system (baseline and accelerated) fits within these resources, and to know ahead of time which parts are tight (most likely: BRAM for weights/buffers, and LUTs if the accelerator datapath is wide).

Reference implementation (instructor task, before the semester)

Before assigning Projects 1-3, build and validate two complete, working systems on real Tang Nano 9K hardware:

  1. System v0 (baseline): RV32I core + bus + UART + timer/GPIO + Zicsr + Zicntr + Zmmul + CMAC + the KWS inference kernel running entirely in software. Record: resource utilization (LUTs/FFs/BRAM) after synthesis, and baseline cycle counts per layer.
  2. System v1 (accelerated): System v0 plus one accelerator for the dominant layer, integrated via the Project 2 memory-mapped interface. Record: resource utilization, end-to-end cycle counts, and speedup.

This reference implementation serves three purposes:

  • Feasibility: confirms everything (including v1) fits on the Tang Nano 9K with realistic headroom - if it doesn't, the scope of Project 2/3 needs to shrink (e.g., a smaller accelerator tile, fewer buffered weights) before students start.
  • Calibration: gives realistic baseline numbers (cycles, resource usage, achievable speedup) to set expectations and grading rubrics - e.g., "a working accelerator with >2x speedup" is a meaningful bar only if we know what's actually achievable.
  • Toolchain validation: surfaces OSS CAD Suite / Tang Nano 9K issues (synthesis quirks, timing closure, PSRAM controller integration) ahead of time, rather than during office hours in Week 13.

This reference implementation is instructor-only material (it would give away the projects' solutions) and should be kept in a private repository, not in this course site. The instructor's copy of this repository has a REFERENCE-IMPLEMENTATION.md with the stage-by-stage build checklist, effort estimates, and a results table to fill in.