TinyML on the Edge

Rodolfo Azevedo

Institute of Computing, University of Campinas (UNICAMP), Brazil

rodolfo.azevedo@unicamp.br

http://www.ic.unicamp.br/~rodolfo/mo801

Goal of this class

Module 4, Class 1: why run AI on constrained hardware, what our model does, and the int8 quantization story.

You now have a processor with peripherals. The question this module answers is: what AI workload runs on it, how was it trained, and why does it need special hardware support?

At the end of this class, you should be able to:

• Explain why edge inference is preferred over cloud inference for always-on keyword spotting applications.
• Describe the tiny_conv model architecture: Conv2D layer → fully-connected layer → softmax classifier.
• Explain int8 quantization, the role of scale factors and zero points, and why it matters for embedded hardware.
• Identify the MFCC feature extraction pipeline that converts raw audio into the model's input tensor.

Why run AI at the edge?

The alternative: send sensor data to a cloud server, run inference there, return the result.

For keyword spotting ("Hey Siri", "OK Google"), cloud is not an option: the microphone must be always-on and listening locally, then wake the radio only when a keyword is detected. The edge processor is the gatekeeper.

The TinyML landscape

"TinyML" = machine learning inference on devices with < 1 MB RAM, typically running on microcontroller-class hardware (Cortex-M, RISC-V) at < 1 mW.

Key frameworks:

• TensorFlow Lite Micro (TFLM): Google's C++ runtime, targets < 256 KB flash + RAM.
• Edge Impulse: cloud-based training + deployment pipeline for TinyML.
• ONNX Runtime (mobile): Microsoft's cross-platform inference engine.

All share the same model lifecycle:

Our model: tiny_conv for Keyword Spotting

• Task: classify 1-second audio clips into 4 classes — yes, no, _unknown_, _silence_.
• Dataset: Google Speech Commands v2 (35 keywords, 105,000 clips).
• Architecture: tiny_conv — a small conv-net: one Conv2D layer + fully-connected classifier + softmax. This is the actual model from the TFLite Micro micro_speech example.
• Input: 49×40 int8 MFCC (Mel-Frequency Cepstral Coefficients) — a 2D spectrogram of the audio.
• Model size: ~19 KB (quantized .tflite) — fits comfortably in on-chip BRAM.
• Accuracy: ~91% on the 4-class task.

Why tiny_conv and not a transformer or RNN?

• Standard Conv2D is simple to understand: a single nested loop with MAC operations at the innermost level, one accelerator target.
• No recurrence (no hidden state to maintain between frames) — simpler to implement and accelerate.
• Small enough to profile and understand completely.
• The TFLite Micro micro_speech notebook trains and quantizes it end-to-end without modifications.

From audio to MFCC features

The model never sees raw audio samples. The pre-processing pipeline:

For this course: MFCC extraction is pre-computed and supplied as test vectors (test_vectors.h). You never implement the FFT — your processor receives the 1,960 int8 values and classifies them.

Int8 quantization: why and how

Neural networks are trained in float32. Running float32 on a microcontroller is slow (no FPU) and memory-hungry. Post-training quantization maps each weight and activation to int8:

And the inverse:

• scale: a per-layer float32 constant (e.g., 0.00392).
• zero_point: an int8 offset representing the float value 0.0.

For MAC operations: instead of float32 multiplications, we do int8 × int8 → int32 accumulations. The int32 accumulator is wide enough to avoid overflow. Only at layer output do we requantize back to int8 using the output scale.

Result: 4× memory reduction (float32 → int8), and on hardware with multiply instructions (Zmmul), 8–10× compute speedup over software float.

The tiny_conv layer structure

The exact kernel size, number of filters, and stride are determined by the reference implementation. The dominant cost is the single Conv2D layer — it accounts for the vast majority of all MACs in the model. The fully-connected layer has negligible MAC count by comparison.

For reference: ResNet-50 has ~4 billion MACs — tiny_conv is over 1,000× cheaper.

Fused ReLU: quantization-aware activation

In the tiny_conv model, the Conv2D layer is followed immediately by a ReLU activation. TFLite fuses these two operations at export time — the quantization parameters absorb the ReLU constraint:

Without ReLU: output range is [-128, 127] (full int8)
With fused ReLU (activation ≥ 0): output range is [0, 127]

In the quantized kernel, this means the requantize() function clamps to [0, 127] instead of [-128, 127]:

int8_t requantize_relu(int32_t acc, int32_t bias,
                       int32_t mult, int shift, int8_t zero_point) {
    acc += bias;
    int64_t scaled = ((int64_t)acc * mult) >> shift;
    int32_t q = (int32_t)scaled + zero_point;
    // fused ReLU: clamp to [0, 127] instead of [-128, 127]
    return (int8_t)(q < 0 ? 0 : (q > 127 ? 127 : q));
}

The TFLite flatbuffer encodes the activation type per operator — the conv_int8 kernel reads it and adjusts the clamp range accordingly. In our simplified kernel, we hardcode the range per layer.

TFLite model introspection: use names, not indices

When extracting weight tensors from a .tflite model, do not rely on hardcoded tensor indices — the TFLite converter may renumber them between versions:

# FRAGILE: tensor index may change with model version
weights = interpreter.get_tensor(2)   # was index 2, may be 3 after reexport

# ROBUST: find tensor by name
tensor_details = interpreter.get_tensor_details()
conv_weights = next(
    t for t in tensor_details if t['name'] == 'sequential/conv2d/Conv2D'
)
weights = interpreter.get_tensor(conv_weights['index'])

The export_weights.py script in the course reference implementation uses name-based lookup for this reason. Model re-training (even with identical architecture) can produce different tensor numbering.

Why not use the full TFLM runtime?

TensorFlow Lite Micro is a C++ library with:

• An interpreter that reads a .tflite flatbuffer at runtime.
• Operator kernels for 50+ layer types.
• A memory arena allocator.
• ~200 KB of code size.

Our RV32I core has:

• 4 KB instruction memory.
• No C++ standard library.
• No dynamic memory allocation.

TFLM will not fit. Instead, the course provides a dependency-free C kernel (kws_kernel.c): ~300 lines, no includes beyond stdint.h, implements exactly the tiny_conv operations we need. The weights and test vectors are in weights.h and test_vectors.h as const int8_t arrays.

This is the code you will profile (M04A03) and then accelerate (Project 3).

End-to-end flow on your platform

Running on the Tang Nano 9K via the Project 2 UART bootloader:

1. Compile: riscv64-unknown-elf-gcc -march=rv32im -O2 kws_kernel.c -o kws.elf
2. Send: objcopy -O ihex kws.elf kws.hex && cat kws.hex > /dev/ttyUSB0
3. Read UART output: class name + cycle counts per layer.

The numbers you collect here are your baseline for Project 3. Write them down.

Int8 quantization — a worked example

Given a float weight w_float = -0.83 and the layer's quantization parameters (scale = 0.0065, zero_point = 0):

And for w_float = 0.41, scale = 0.0065:

Dequantization reverses the operation: .

The quantization error is . This error propagates through all layers — which is why the model must be quantization-aware trained (QAT) or calibrated, not simply rounded post-hoc.

Key insight: the mult and shift arrays in weights.h encode one scale factor per output channel — not one per layer — because different filters need different scales to minimize accuracy loss.

In-class exercise — BRAM budget for tiny_conv

Before implementation, verify that the model fits in the Tang Nano 9K's 26 BRAM blocks (each 18 Kbit = 2.25 KB):

1. How many BRAM blocks are needed if all weights and activations are in BRAM?
2. Which component dominates the BRAM usage?
3. If the weights do not fit, which component could be kept in DMEM instead (at the cost of extra bus transactions)?

Expected: Conv2D weights dominate. Activations are small (< 4 KB). Total weights + biases for tiny_conv fit comfortably within the 26 available BRAMs — this is by design: the model was chosen to fit on-chip.

Next class

The KWS Inference Kernel: a detailed walkthrough of kws_kernel.c — the int8 MAC, the Conv2D inner loop, scale/zero_point requantization, and how to count MACs per layer. Lab 4 begins.