A Wakeword Detector on Hardware

An MFCC front end in C and a DTW classifier on the ESP32-S3 and NUCLEO-F446RE

The main page turns a frame of audio into thirteen numbers. This page turns a stream of those numbers into a working wakeword detector: say a chosen word, and a microcontroller lights an LED. It does it the old-fashioned way, with no neural network and no training run. The trick is template matching: store the MFCC trajectory of a few example utterances of the word, and at run time compare the live MFCC stream against them with dynamic time warping (DTW) (Sakoe and Chiba 1978), which aligns two sequences that say the same thing at different speeds. If the closest template is near enough, the word was spoken.

This is squarely the kind of classical, training-free pattern recognition that ADR-006 §3 allows as supporting material: it earns its place by showing the MFCC front end doing a real job on real silicon, not as a standalone machine-learning topic. It is honest about its limits, which are real, and it points at where you would go next when those limits start to bite.

The two reference platforms follow ADR-005: the ESP32-S3 (the natural host, with an I2S microphone and plenty of RAM) carries the full running example, and the NUCLEO-F446RE (Cortex-M4F, 180 MHz, CMSIS-DSP) runs the identical pipeline with its FFT supplied by a different library. Everything downstream of the FFT is portable float C that compiles unchanged on both.

---

The shape of the system

  INMP441 mic ──I2S──▶ frame ──▶ MFCC front end ──▶ MFCC stream ──┐
                                                                  ▼
   reference templates (a few stored MFCC sequences) ──▶ DTW ──▶ argmin distance
                                                                  ▼
                                                       below threshold? ──▶ wakeword!

Two phases. Enrolment (done once, off the critical path): record three or four clean utterances of the word, compute each one’s MFCC sequence, store them in flash as the templates. Detection (the real-time loop): every hop, append the new MFCC frame to a sliding buffer, and periodically run DTW between that buffer and each template. The smallest distance names the best match; if it falls below a threshold tuned at enrolment, declare a detection.

The audio front end: INMP441 over I2S

The INMP441 is a MEMS microphone with an I2S digital output, which means no analog front end and no ADC calibration: it streams 24-bit samples straight into the ESP32-S3’s I2S peripheral, which DMAs them into a buffer. We read mono, downshift to a float, and feed the MFCC pipeline.

#include "driver/i2s_std.h"

#define SAMPLE_RATE 16000
#define FRAME_LEN   400          // 25 ms at 16 kHz
#define HOP         160          // 10 ms hop

static i2s_chan_handle_t rx;

void mic_init(void)
{
    i2s_chan_config_t chan_cfg = I2S_CHANNEL_DEFAULT_CONFIG(I2S_NUM_0, I2S_ROLE_MASTER);
    i2s_new_channel(&chan_cfg, NULL, &rx);

    i2s_std_config_t std_cfg = {
        .clk_cfg  = I2S_STD_CLK_DEFAULT_CONFIG(SAMPLE_RATE),
        .slot_cfg = I2S_STD_PHILIPS_SLOT_DEFAULT_CONFIG(
                        I2S_DATA_BIT_WIDTH_32BIT, I2S_SLOT_MODE_MONO),
        .gpio_cfg = { .bclk = GPIO_NUM_4, .ws = GPIO_NUM_5, .din = GPIO_NUM_6 },
    };
    i2s_channel_init_std_mode(rx, &std_cfg);
    i2s_channel_enable(rx);
}

// Read HOP new samples, converting the INMP441's left-justified 32-bit slot to
// a normalised float. The mic's 24 valid bits sit in the top of the 32-bit word.
void mic_read_hop(float *dst)
{
    int32_t raw[HOP];
    size_t got = 0;
    i2s_channel_read(rx, raw, sizeof(raw), &got, portMAX_DELAY);
    for (int i = 0; i < HOP; i++)
        dst[i] = (float)(raw[i] >> 8) / 8388608.0f;   // >>8 to 24-bit, /2^23 to [-1,1)
}

On the NUCLEO-F446RE the same samples come from its own I2S/SAI peripheral (or, for an analog electret mic, the timer-triggered ADC with DMA from the biquad embedded page). From here the two platforms are identical.

The MFCC front end in C

Each stage of the Python pipeline maps to a few lines of C. The tables that the desktop computes on the fly (the window, the mel weights, the DCT matrix) are computed once at enrolment time and baked into flash as const arrays, so the run-time cost is just multiply-accumulates.

#include <math.h>

#define NFFT      512
#define NBINS     (NFFT/2 + 1)
#define NFILT     26
#define NCEP      13
#define PREEMPH   0.97f

// These tables are generated once (at enrolment, or by a small host program) from
// the formulas in mfcc.py and baked into flash: the Hamming taper, the triangular
// mel weights, and the first NCEP rows of the DCT-II matrix. They are shown const
// because they never change at run time. They MUST be populated; a const array
// left with no initializer is zero-filled, and zero tables make every MFCC come
// out zero with no error. A real build emits the initializer values here.
static const float hamming[FRAME_LEN];        // precomputed taper
static const float mel_fb[NFILT][NBINS];      // sparse triangular weights
static const float dct_mat[NCEP][NFILT];      // DCT-II basis, first NCEP rows

// One frame in, one 13-element MFCC vector out. power[] is the |FFT|^2 from the
// platform FFT below; prev holds the last sample of the previous frame for the
// one-tap pre-emphasis carry. NOT REENTRANT: the working buffers are static, so
// call this from one task only (on the dual-core ESP32-S3, pin enrolment and
// detection to the same core, or guard with a mutex).
void mfcc_frame(const float *frame_in, float prev, float *cep)
{
    // emph is sized to NFFT, not FRAME_LEN: the FFT transforms NFFT points, so the
    // FRAME_LEN windowed samples are zero-padded up to NFFT. This mirrors the
    // Python np.fft.rfft(frames, n=n_fft); without it the FFT reads past the frame.
    static float emph[NFFT], power[NBINS], mel[NFILT];

    // 1. pre-emphasis (one tap, carrying across the frame boundary)
    emph[0] = frame_in[0] - PREEMPH * prev;
    for (int n = 1; n < FRAME_LEN; n++)
        emph[n] = frame_in[n] - PREEMPH * frame_in[n - 1];

    // 2. window the FRAME_LEN samples in place, then zero-pad the tail to NFFT
    for (int n = 0; n < FRAME_LEN; n++) emph[n] *= hamming[n];
    for (int n = FRAME_LEN; n < NFFT; n++) emph[n] = 0.0f;

    // 3. FFT -> power spectrum (platform-specific, fills power[]; see below)
    fft_power(emph, power);

    // 4. mel filterbank + 5. log: sparse dot products, then log
    for (int m = 0; m < NFILT; m++) {
        float acc = 0.0f;
        for (int k = 0; k < NBINS; k++) acc += mel_fb[m][k] * power[k];
        mel[m] = logf(acc > 1e-12f ? acc : 1e-12f);   // log floor, as in Python
    }

    // 6. DCT-II, keeping the first NCEP coefficients
    for (int c = 0; c < NCEP; c++) {
        float acc = 0.0f;
        for (int m = 0; m < NFILT; m++) acc += dct_mat[c][m] * mel[m];
        cep[c] = acc;
    }
}

The mel filterbank loop above is written full-width for clarity; in practice each mel_fb[m] row is zero outside a short bin range, so a production version stores only the nonzero span and its start index, cutting the inner loop from NBINS to a handful of taps. The DCT is a fixed 13x26 matrix-vector product, 338 multiplies, negligible next to the FFT.

The FFT on each platform

Only this step differs between the two boards, and it is the same split as on the STFT hardware page, so we keep it short. On the NUCLEO-F446RE, CMSIS-DSP’s real FFT, minding its packed DC/Nyquist layout:

#include "arm_math.h"

static arm_rfft_fast_instance_f32 fft;        // arm_rfft_fast_init_f32(&fft, NFFT) once

void fft_power(float *frame, float *power)
{
    static float spec[NFFT];                   // packed complex output
    arm_rfft_fast_f32(&fft, frame, spec, 0);
    power[0]       = spec[0] * spec[0];         // DC (real)
    power[NFFT/2]  = spec[1] * spec[1];         // Nyquist (real)
    for (int k = 1; k < NFFT/2; k++)           // interleaved Re,Im for the rest
        power[k] = spec[2*k] * spec[2*k] + spec[2*k+1] * spec[2*k+1];
}

On the ESP32-S3, ESP-DSP’s radix-2 FFT (SIMD-accelerated on the LX7), following its own example for the exact packed-real layout:

#include "esp_dsp.h"

// dsps_fft2r_init_fc32(NULL, NFFT) once at startup.
void fft_power(float *frame, float *power)
{
    static float z[2 * NFFT];                   // interleaved (re, im)
    for (int n = 0; n < NFFT; n++) { z[2*n] = frame[n]; z[2*n+1] = 0.0f; }
    dsps_fft2r_fc32(z, NFFT);
    dsps_bit_rev_fc32(z, NFFT);
    for (int k = 0; k < NBINS; k++)
        power[k] = z[2*k] * z[2*k] + z[2*k+1] * z[2*k+1];
}

Both fill the same power[], so mfcc_frame above is genuinely identical on the two boards. As on the STFT page, verify each library’s bin order against its own example before trusting the numbers; CMSIS and ESP-DSP do not pack a real spectrum the same way.

Dynamic time warping: matching at any speed

Two people say “Marvin” and one drawls it. The MFCC sequences have the same shape but different lengths and different local stretching, so a sample-by-sample comparison fails. DTW (Sakoe and Chiba 1978) finds the lowest-cost alignment between two sequences by dynamic programming: build a matrix of local frame-to-frame distances and find the cheapest monotone path through it. The recurrence is three lines.

#include <math.h>                              // sqrtf, INFINITY

// Euclidean distance between two MFCC frames (the local cost the path pays).
static inline float dist(const float *a, const float *b)
{
    float s = 0.0f;
    for (int c = 0; c < NCEP; c++) { float d = a[c] - b[c]; s += d * d; }
    return sqrtf(s);
}

// DTW distance between a query of na frames and a template of nb frames, each
// frame an NCEP-vector. Uses two rolling rows, so O(nb) memory, not O(na*nb).
float dtw(const float (*a)[NCEP], int na, const float (*b)[NCEP], int nb,
          float *prev, float *curr)            // two scratch rows of length nb+1
{
    for (int j = 0; j <= nb; j++) prev[j] = INFINITY;
    prev[0] = 0.0f;
    for (int i = 1; i <= na; i++) {
        curr[0] = INFINITY;
        for (int j = 1; j <= nb; j++) {
            float c = dist(a[i-1], b[j-1]);
            float best = prev[j];                       // insertion
            if (prev[j-1] < best) best = prev[j-1];     // match (diagonal)
            if (curr[j-1] < best) best = curr[j-1];     // deletion
            curr[j] = c + best;
        }
        float *tmp = prev; prev = curr; curr = tmp;     // roll the rows
    }
    return prev[nb];
}

The classifier is then: run dtw against each stored template, take the smallest, and compare it to the enrolment threshold. The Python below is the same algorithm, and it both checks the recurrence (a sequence matched against itself costs zero, and the distance is symmetric) and shows the whole detector working: a sped-up, noisy copy of one word is correctly matched to its own template, not to the others.

from scipy.signal import butter, lfilter, resample
from mfcc import mfcc
fs = 16000

def _vowel(dur, f0):
    t = np.arange(int(dur * fs)) / fs; x = np.zeros_like(t)
    for k in range(1, 40):
        fk = k * f0
        if fk >= fs / 2: break
        g = np.exp(-((fk-700)/400)**2) + 0.7*np.exp(-((fk-1200)/500)**2) + 0.05
        x += g * np.sin(2*np.pi*fk*t)
    return x

def _fric(dur, seed):
    rng = np.random.default_rng(seed); b, a = butter(4, 4000/(fs/2), "high")
    return lfilter(b, a, rng.standard_normal(int(dur*fs)))

# three distinct "words" as patterns of voiced and unvoiced segments
words = {
    "word0": np.concatenate([_vowel(0.25,120), _fric(0.2,1), _vowel(0.25,180)]),
    "word1": np.concatenate([_fric(0.2,2), _vowel(0.3,150), _fric(0.2,3)]),
    "word2": np.concatenate([_vowel(0.2,100), _vowel(0.25,220), _fric(0.15,4)]),
}
templates = {k: mfcc(v, fs) for k, v in words.items()}

def dtw(a, b):
    na, nb = len(a), len(b)
    prev = np.full(nb + 1, np.inf); prev[0] = 0.0
    for i in range(1, na + 1):
        curr = np.full(nb + 1, np.inf)
        for j in range(1, nb + 1):
            c = np.linalg.norm(a[i-1] - b[j-1])
            curr[j] = c + min(prev[j], prev[j-1], curr[j-1])
        prev = curr
    return prev[nb]

# recurrence sanity: self-distance is zero, and DTW is symmetric
assert dtw(templates["word0"], templates["word0"]) == 0.0
assert np.isclose(dtw(templates["word0"], templates["word1"]),
                  dtw(templates["word1"], templates["word0"]))

# the detector: a 1.2x-stretched, noisy utterance of word0
rng = np.random.default_rng(99)
stretched = resample(words["word0"], int(len(words["word0"]) * 1.2))
query = mfcc(stretched + 0.05 * rng.standard_normal(len(stretched)), fs)
scores = {k: dtw(query, t) for k, t in templates.items()}
winner = min(scores, key=scores.get)
runner_up = min(scores["word1"], scores["word2"])
assert winner == "word0"                                  # matched the right word
assert runner_up > 1.3 * scores["word0"]                  # by a clear margin
print("DTW distances:", {k: round(v, 1) for k, v in scores.items()})
print(f"detected: {winner}")

DTW distances: {'word0': np.float64(826.3), 'word1': np.float64(2203.3), 'word2': np.float64(1432.8)}
detected: word0

Memory and time budget

The whole detector is small. The tables and templates dominate the RAM, and the per-frame work is dominated by the FFT.

flt = 4
FRAME_LEN, NFFT, NBINS, NFILT, NCEP = 400, 512, 257, 26, 13

# the baked-in tables
tables = (FRAME_LEN + NFILT * NBINS + NCEP * NFILT) * flt
# four ~0.7 s reference templates at a 10 ms hop, 13 coeffs each
frames_per_word = (int(0.7 * 16000) - FRAME_LEN) // 160 + 1
templates_bytes = 4 * frames_per_word * NCEP * flt
# detection scratch: a sliding MFCC buffer plus the two DTW rows
scratch = (frames_per_word * NCEP + 2 * (frames_per_word + 1)) * flt
# FFT working buffers held across the frame: emph + power + mel + the CMSIS spec
working = (NFFT + NBINS + NFILT + NFFT) * flt
total = tables + templates_bytes + scratch + working
print(f"tables {tables/1024:.1f} + templates {templates_bytes/1024:.1f} + scratch "
      f"{scratch/1024:.1f} + work {working/1024:.1f} KB = {total/1024:.1f} KB")
assert total < 64 * 1024   # fits the F446RE's 128 KB; the I2S DMA ring buffer adds a few KB more

# per-hop time: one FFT dominates; mel is sparse, DCT is 338 MACs
hop_period = 160 / 16000.0                      # 10 ms between frames
cycles = hop_period * 180e6                     # F446RE cycles per hop
# real-input FFT ~ (N/4) log2 N butterflies (conjugate symmetry halves the complex count)
fft_butterflies = (NFFT // 4) * int(np.log2(NFFT))
assert cycles > 100 * fft_butterflies          # the front end fits with margin
print(f"hop every {1000*hop_period:.0f} ms -> {cycles:,.0f} cycles; "
      f"FFT ~{fft_butterflies:,} butterflies")

tables 29.0 + templates 13.8 + scratch 4.0 + work 5.1 KB = 51.9 KB
hop every 10 ms -> 1,800,000 cycles; FFT ~1,152 butterflies

DTW itself runs once every few hops, not every hop, and costs na * nb frame distances (each NCEP multiply-accumulates) per template. For sub-second words that is tens of thousands of MACs per template, comfortably inside the gap between detection passes. The honest measurement on real hardware is a cycle counter (DWT->CYCCNT on the M4) around both the front end and the DTW, not these estimates.

What this does and does not do

It works, and it is worth being plain about how far.

• It is speaker-dependent. The templates are your utterances. A different voice, especially a different accent or pitch range, matches poorly. Enrolling several speakers’ templates helps a little and costs RAM linearly.
• It is fragile to noise and distance. A clean template recorded at 20 cm degrades fast in a noisy room or across the kitchen. Cepstral mean subtraction and an energy-based voice-activity gate (only run DTW when something is being said) help, but this is not a far-field smart-speaker front end.
• It scales poorly to many words. DTW against every template every detection pass is linear in the vocabulary. For a handful of commands it is fine; for hundreds of words it is the wrong tool.
• The threshold is a blunt knob. One distance threshold trades false accepts against false rejects with no middle ground. There is no calibrated confidence.

None of this makes it useless. For a single wakeword or a few fixed commands, on a battery part, with no cloud and no training pipeline, it is a genuinely good answer, and every piece of it is inspectable.

When these limits start to bite, the path forward is to replace the back end while keeping the front end: feed the log-mel features (the page’s stage 5, before the DCT) into a small neural network instead of a DTW template bank. That is exactly what TinyML frameworks and tools like Edge Impulse do, and they run on the same class of microcontroller. The MFCC (or its log-mel parent) survives as the front end; what changes is that the classifier is learned from many examples rather than matched against a few. The DSP you built here does not go away. It becomes the first layer.

References

Sakoe, Hiroaki, and Seibi Chiba. 1978. “Dynamic Programming Algorithm Optimization for Spoken Word Recognition.” IEEE Transactions on Acoustics, Speech, and Signal Processing 26 (1): 43–49.