The Voice Pitch Estimator on Hardware

The ESP32 implementation the design shipped on, and what a NUCLEO-F446RE port would take

This is the one embedded page in the estimation & detection arc where the hardware came first: the pipeline of the main page is the design of a 2022 wearable prototype, an ESP32 (TinyPico) with an SPH0645 I2S MEMS microphone, built as a building block for speech-therapy feedback. This page walks that implementation’s architecture: how the samples arrive, how the real-time structure is organised, and where the cycles go. The generic per-algorithm recipes (biquad code, the two-FFT cepstrum kernel, VAD snippets) live on the pitch-detection embedded page and are not repeated; this page is about making them into a system.

The Python reference implementation in vpe.py mirrors this structure stage for stage, and its ground-truth tests (test_vpe.py, pitch recovery within 2% across 100-300 Hz) are the specification any C implementation of this pipeline is checked against: run both on the same synthetic voiced signals and compare the tracks.

Build provenance: paper design verified against the tested Python model; the C here has not itself been executed. The I2S capture and FreeRTOS structure are transcribed from a 2022 ESP32 prototype that ran this front end (ADR-005 section 7).

System architecture: two tasks and a mutex

The implementation is FreeRTOS-shaped. A dedicated sampler task, pinned to core 1, owns the I2S peripheral and the whole DSP chain; the application (display, BLE, or a serial log) reads the latest result from whichever core it likes. The only shared state is the result triple (voiced flag, pitch, magnitude), guarded by a mutex; the signal path itself never crosses a task boundary, so there is nothing else to lock.

// Sampler task: owns I2S and the entire DSP chain (pinned to core 1)
void i2sSamplerTask(void *param) {
    Pitch *obj = (Pitch *)param;
    i2sSampler.start();                    // I2S DMA running from here on
    while (true) {
        int n = i2sSampler.read(obj->samples, SAMPLE_BUFFER_SIZE);
        if (n == SAMPLE_BUFFER_SIZE) {
            obj->preprocess();             // front-end biquads, per sample
            obj->compute_periodogram();    // hop through overlapped frames
            obj->detect_voice_activity();  // trackers + decision
        }
    }
}

// Any other task, any core: read the latest result
bool Pitch::getResult(float *freq, float *mag) {
    take_mutex(mutex);
    *freq = voice_pitch_frequency;
    *mag  = voice_pitch_magnitude;
    bool active = voice_active;
    give_mutex(mutex);
    return active;
}

A second, optional path forks the raw sample buffers into a FreeRTOS queue feeding a WiFi/websocket streaming task. That debug channel earned its keep during development: stream the raw I2S samples to a laptop, dump them to a file, and import them into Audacity as signed 16-bit mono at 16 kHz. Half of all audio-capture bugs (wrong channel, wrong bit alignment, wrong byte order) are audible or visible in ten seconds that way, before any DSP runs. The queue is sized at one buffer and the send does not block: if WiFi stalls, buffers are dropped and the estimator keeps its real-time deadline; the debug path is never allowed to back-pressure the signal path.

Getting samples in: the SPH0645 over I2S

The SPH0645 transmits 18-bit samples left-justified in 32-bit I2S frames, mono on the left channel by default. The peripheral is configured for 32-bit receive at 16 kHz with a modest DMA arrangement (four 1024-sample buffers), and each raw word is shifted down to a signed 16-bit sample before entering the filter chain:

i2s_config_t cfg = {
    .mode = I2S_MODE_MASTER | I2S_MODE_RX,
    .sample_rate = 16000,
    .bits_per_sample = I2S_BITS_PER_SAMPLE_32BIT,
    .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
    .dma_buf_count = 4,
    .dma_buf_len = 1024,
    // ...
};

Two hardware facts worth recording. First, the shift count is a per-microphone datasheet detail: the SPH0645 places 18 data bits in bits 31:14 of the frame, so a 16-bit sample is the word’s top 16 bits, a right shift by 16; the INMP441 places 24 bits in 31:8, which is why the pitch-detection embedded page’s capture code shifts by 8 and scales by $2^{23}$ into float instead. Mixing up the alignment either clips loud passages (shift too small: the sign-carrying top bits are truncated) or silently discards the quiet end of the dynamic range (shift too large). Second, this microphone family has a known DC offset, which is one of the two reasons the pipeline removes the frame mean before windowing (the other being that the periodogram’s DC bin would otherwise leak through the Hamming window’s skirts into the low bins the front end just cleaned).

Third, and the one that costs newcomers an afternoon: on the ESP32 the datasheet alignment is not what arrives by default. The SPH0645 updates its data line on the I2S clock edge the ESP32 peripheral samples on, so without a timing adjustment every word lands one bit off, corrupting the sign bit; the audible symptom in the Audacity debug stream is a quiet signal riding a violent DC rail. The implementation carries the community-standard two-register workaround (via the atomic14 capture pattern credited below), applied before the pins are configured:

if (m_fixSPH0645)
{
    // FIXES for SPH0645
    REG_SET_BIT(I2S_TIMING_REG(m_i2sPort), BIT(9));
    REG_SET_BIT(I2S_CONF_REG(m_i2sPort), I2S_RX_MSB_SHIFT);
}

This implementation used the classic driver/i2s.h API current in 2022; new ESP-IDF v5.x code should use the i2s_std API shown on the pitch-detection embedded page, whose standard-mode driver handles the Philips timing cleanly.

The DSP chain on the metal

Front end. Two second-order sections in transposed direct form II, coefficients from the offline SciPy design (the ADR-005 convention: design in Python, paste the SOS). Two implementation details from this system are worth stealing. The overall gain is factored out of the coefficients and applied as a single multiply per section output, which keeps the feedback path’s coefficients at full float precision. And the filter state is initialised to the steady-state response for a constant input rather than to zero:

\[w_0 = x_{\mathrm{in}}\,\frac{b_1 + b_2 - b_0 (a_1 + a_2)}{1 + a_1 + a_2}, \qquad w_1 = b_2\, x_{\mathrm{in}} - a_2\, y, \quad y = x_{\mathrm{in}} \frac{b_0 + b_1 + b_2}{1 + a_1 + a_2},\]

the closed forms obtained by setting output and states constant in the transposed-DF-II recurrence (a biquad has two states, and both must be set: initialising only $w_0$ still leaves a measurable transient). Both expressions match scipy.signal.sosfilt_zi(sos) * x_in exactly, which is the one-line way to generate them for a whole cascade; note that section 2’s $x_{\mathrm{in}}$ is section 1’s steady-state output, not the raw input. Starting from zero state instead means the microphone’s DC offset arrives as a step, and a 50 Hz high-pass edge rings for tens of milliseconds before the estimator sees clean data; the steady-state init removes that start-up transient entirely.

Windowing and spectra. The Hamming window is computed once at start-up, and the constructor then performs a small self-check worth imitating: it computes the window’s power in the time domain and again from its FFT through the one-sided periodogram convention, and logs both. That is Parseval’s theorem used as a boot-time invariant: if the FFT scaling, the factor-of-two convention, or the window generation is ever wrong, the two numbers disagree at start-up, on the device, before any measurement is trusted. (The Python module’s Parseval test is the same check promoted to CI.)

Cepstrum and readout. The pitch readout is the two-forward-FFT cepstrum kernel given in full on the pitch-detection embedded page: FFT, log-magnitude repack, FFT again, then a peak search over the quefrency window $q \in [f_s/400,\ f_s/50] = [40, 320]$ and the time-like readout $f_0 = f_s/q$, with the rahmonic (octave) guard and parabolic refinement of the main page. The only differences at 2048 points are size and the log-periodogram floor, applied here exactly as in vpe.py.

Detection and gating. The min/max power trackers, the dynamic threshold, the cepstral prominence test, and the Hampel gate are all scalar or tiny-window operations: two floats for the trackers, eleven for the gate. On the metal their cost is noise against the FFTs; their value is the entire difference between a demo and an instrument.

Where the cycles go (ESP32, 240 MHz)

Frames are 2048 samples (128 ms) at 50% overlap, so the real-time deadline is one full chain pass per 64 ms hop, 15.4M cycles at 240 MHz. Order-of-magnitude budget, dominated by the two transforms:

Stage	Operation	Est. cycles	Time
I2S read + shift	1024 new samples	~5K	21 us
Front end (2 SOS)	1024 x ~10 MACs	~20K	83 us
DC removal + window	2048 x 2 ops	~8K	33 us
FFT #1 (2048-pt real)	~11 N butterflies	~160K	0.67 ms
Periodogram + EMA	1024 bins	~15K	63 us
log() of 1024 bins	~40 cycles each	~40K	167 us
FFT #2 (cepstrum)	same kernel	~160K	0.67 ms
Peak search + trackers + gate	~300 bins + scalars	~5K	21 us
Total per 64 ms hop		~410K	~1.7 ms
Available per hop		15,360K	64 ms
Utilisation			~2.7%

Memory is equally comfortable: the 4096-sample capture buffer (int16, 8 kB), FFT input/output and window (three 2048-float arrays, 24 kB), and the averaged periodogram and cepstrum (two 1024-float arrays, 8 kB) total about 40 kB of the ESP32’s 520 kB SRAM. The lesson of the budget is the same one the pitch-detection embedded page reached at 512 points: classical pitch estimation is nowhere near the limit of this class of hardware, which is exactly the headroom argument the main page’s honest-limits section makes against reaching for a neural estimator on-device.

Porting to the NUCLEO-F446RE: a feasibility note

Per ADR-005 the workshop’s second platform is the NUCLEO-F446RE (Cortex-M4F, 180 MHz, 128 kB SRAM). This system has not been ported to it; the note below is the feasibility assessment, not a report.

Compute. CMSIS-DSP’s arm_rfft_fast_f32 at 2048 points costs a few hundred thousand cycles; two of them plus the scalar stages lands the hop budget near 600K of the 11.5M cycles available per 64 ms, roughly 5% utilisation. No obstacle.
Memory. The ~40 kB working set fits the 128 kB SRAM with room for the application, but it is no longer negligible; the int16 capture buffer and float frame should not be duplicated casually.
Plumbing, not maths. The real porting cost is infrastructure: I2S capture moves to the SAI/I2S peripheral with DMA double-buffering (the ping-pong pattern on the pitch-detection embedded page), the FreeRTOS task-and-mutex structure becomes either bare-metal ISR flags or FreeRTOS-on-STM32, and the ESP-DSP FFT-plus-bit-reverse call pairs become single arm_rfft_fast_f32 calls. The DSP code between the transforms is portable float C and moves unchanged.
What is lost. The ESP32’s built-in WiFi/BLE carried both the debug stream and the intended wearable output; on the Nucleo those become a UART link or an external module, which for a therapy wearable is a real system cost, not a detail. This is the honest reason the 2022 prototype was an ESP32 project.

Credits

The implementation stands on three credited third-party components: the HampelFilter Arduino library by Florian Roscheck (the streaming median/MAD gate), a split-radix real FFT derived from Robin Scheibler’s ESP32 FFT work, and I2S capture patterns from atomic14’s esp32_audio examples. Everything else (the filter design, the cepstral readout, the VAD, and the system structure) is the original design this page documents.

--- title: "The Voice Pitch Estimator on Hardware" subtitle: "The ESP32 implementation the design shipped on, and what a NUCLEO-F446RE port would take" --- This is the one embedded page in the [estimation & detection arc](../estimation-and-detection.qmd) where the hardware came first: the pipeline of the [main page](index.qmd) is the design of a 2022 wearable prototype, an ESP32 (TinyPico) with an SPH0645 I2S MEMS microphone, built as a building block for speech-therapy feedback. This page walks that implementation's architecture: how the samples arrive, how the real-time structure is organised, and where the cycles go. The generic per-algorithm recipes (biquad code, the two-FFT cepstrum kernel, VAD snippets) live on the [pitch-detection embedded page](../pitch-detection/embedded.qmd) and are not repeated; this page is about making them into a system. The Python reference implementation in [`vpe.py`](vpe.py) mirrors this structure stage for stage, and its ground-truth tests (`test_vpe.py`, pitch recovery within 2% across 100-300 Hz) are the specification any C implementation of this pipeline is checked against: run both on the same synthetic voiced signals and compare the tracks. *Build provenance: paper design verified against the tested Python model; the C here has not itself been executed. The I2S capture and FreeRTOS structure are transcribed from a 2022 ESP32 prototype that ran this front end ([ADR-005](../../docs/adr/005-embedded-platforms.md) section 7).* <hr> ## System architecture: two tasks and a mutex The implementation is FreeRTOS-shaped. A dedicated sampler task, pinned to core 1, owns the I2S peripheral and the whole DSP chain; the application (display, BLE, or a serial log) reads the latest result from whichever core it likes. The only shared state is the result triple (voiced flag, pitch, magnitude), guarded by a mutex; the signal path itself never crosses a task boundary, so there is nothing else to lock. ```c // Sampler task: owns I2S and the entire DSP chain (pinned to core 1) void i2sSamplerTask(void *param) { Pitch *obj = (Pitch *)param; i2sSampler.start(); // I2S DMA running from here on while (true) { int n = i2sSampler.read(obj->samples, SAMPLE_BUFFER_SIZE); if (n == SAMPLE_BUFFER_SIZE) { obj->preprocess(); // front-end biquads, per sample obj->compute_periodogram(); // hop through overlapped frames obj->detect_voice_activity(); // trackers + decision } } } // Any other task, any core: read the latest result bool Pitch::getResult(float *freq, float *mag) { take_mutex(mutex); *freq = voice_pitch_frequency; *mag = voice_pitch_magnitude; bool active = voice_active; give_mutex(mutex); return active; } ``` A second, optional path forks the raw sample buffers into a FreeRTOS queue feeding a WiFi/websocket streaming task. That debug channel earned its keep during development: stream the raw I2S samples to a laptop, dump them to a file, and import them into Audacity as signed 16-bit mono at 16 kHz. Half of all audio-capture bugs (wrong channel, wrong bit alignment, wrong byte order) are *audible* or *visible* in ten seconds that way, before any DSP runs. The queue is sized at one buffer and the send does not block: if WiFi stalls, buffers are dropped and the estimator keeps its real-time deadline; the debug path is never allowed to back-pressure the signal path. ## Getting samples in: the SPH0645 over I2S The SPH0645 transmits 18-bit samples left-justified in 32-bit I2S frames, mono on the left channel by default. The peripheral is configured for 32-bit receive at 16 kHz with a modest DMA arrangement (four 1024-sample buffers), and each raw word is shifted down to a signed 16-bit sample before entering the filter chain: ```c i2s_config_t cfg = { .mode = I2S_MODE_MASTER | I2S_MODE_RX, .sample_rate = 16000, .bits_per_sample = I2S_BITS_PER_SAMPLE_32BIT, .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT, .dma_buf_count = 4, .dma_buf_len = 1024, // ... }; ``` Two hardware facts worth recording. First, the shift count is a per-microphone datasheet detail: the SPH0645 places 18 data bits in bits 31:14 of the frame, so a 16-bit sample is the word's top 16 bits, a right shift by 16; the INMP441 places 24 bits in 31:8, which is why the [pitch-detection embedded page's](../pitch-detection/embedded.qmd) capture code shifts by 8 and scales by $2^{23}$ into float instead. Mixing up the alignment either clips loud passages (shift too small: the sign-carrying top bits are truncated) or silently discards the quiet end of the dynamic range (shift too large). Second, this microphone family has a known DC offset, which is one of the two reasons the pipeline removes the frame mean before windowing (the other being that the periodogram's DC bin would otherwise leak through the Hamming window's skirts into the low bins the front end just cleaned). Third, and the one that costs newcomers an afternoon: on the ESP32 the datasheet alignment is not what arrives by default. The SPH0645 updates its data line on the I2S clock edge the ESP32 peripheral samples on, so without a timing adjustment every word lands one bit off, corrupting the sign bit; the audible symptom in the Audacity debug stream is a quiet signal riding a violent DC rail. The implementation carries the community-standard two-register workaround (via the atomic14 capture pattern credited below), applied before the pins are configured: ```c if (m_fixSPH0645) { // FIXES for SPH0645 REG_SET_BIT(I2S_TIMING_REG(m_i2sPort), BIT(9)); REG_SET_BIT(I2S_CONF_REG(m_i2sPort), I2S_RX_MSB_SHIFT); } ``` This implementation used the classic `driver/i2s.h` API current in 2022; new ESP-IDF v5.x code should use the `i2s_std` API shown on the [pitch-detection embedded page](../pitch-detection/embedded.qmd#i2s-microphone-setup), whose standard-mode driver handles the Philips timing cleanly. ## The DSP chain on the metal **Front end.** Two second-order sections in transposed direct form II, coefficients from the offline SciPy design (the [ADR-005](../../docs/adr/005-embedded-platforms.md) convention: design in Python, paste the SOS). Two implementation details from this system are worth stealing. The overall gain is factored out of the coefficients and applied as a single multiply per section output, which keeps the feedback path's coefficients at full float precision. And the filter state is initialised to the steady-state response for a constant input rather than to zero: $$w_0 = x_{\mathrm{in}}\,\frac{b_1 + b_2 - b_0 (a_1 + a_2)}{1 + a_1 + a_2}, \qquad w_1 = b_2\, x_{\mathrm{in}} - a_2\, y, \quad y = x_{\mathrm{in}} \frac{b_0 + b_1 + b_2}{1 + a_1 + a_2},$$ the closed forms obtained by setting output and states constant in the transposed-DF-II recurrence (a biquad has two states, and both must be set: initialising only $w_0$ still leaves a measurable transient). Both expressions match `scipy.signal.sosfilt_zi(sos) * x_in` exactly, which is the one-line way to generate them for a whole cascade; note that section 2's $x_{\mathrm{in}}$ is section 1's steady-state *output*, not the raw input. Starting from zero state instead means the microphone's DC offset arrives as a step, and a 50 Hz high-pass edge rings for tens of milliseconds before the estimator sees clean data; the steady-state init removes that start-up transient entirely. **Windowing and spectra.** The Hamming window is computed once at start-up, and the constructor then performs a small self-check worth imitating: it computes the window's power in the time domain and again from its FFT through the one-sided periodogram convention, and logs both. That is Parseval's theorem used as a boot-time invariant: if the FFT scaling, the factor-of-two convention, or the window generation is ever wrong, the two numbers disagree at start-up, on the device, before any measurement is trusted. (The Python module's Parseval test is the same check promoted to CI.) **Cepstrum and readout.** The pitch readout is the two-forward-FFT cepstrum kernel given in full on the [pitch-detection embedded page](../pitch-detection/embedded.qmd#cepstral-pitch-estimation-two-ffts): FFT, log-magnitude repack, FFT again, then a peak search over the quefrency window $q \in [f_s/400,\ f_s/50] = [40, 320]$ and the time-like readout $f_0 = f_s/q$, with the rahmonic (octave) guard and parabolic refinement of the [main page](index.qmd#cepstral-pitch-read-the-quefrency-axis-the-right-way-around). The only differences at 2048 points are size and the log-periodogram floor, applied here exactly as in `vpe.py`. **Detection and gating.** The min/max power trackers, the dynamic threshold, the cepstral prominence test, and the Hampel gate are all scalar or tiny-window operations: two floats for the trackers, eleven for the gate. On the metal their cost is noise against the FFTs; their value is the entire difference between a demo and an instrument. ## Where the cycles go (ESP32, 240 MHz) Frames are 2048 samples (128 ms) at 50% overlap, so the real-time deadline is one full chain pass per 64 ms hop, 15.4M cycles at 240 MHz. Order-of-magnitude budget, dominated by the two transforms: | Stage | Operation | Est. cycles | Time | |---|---|---|---| | I2S read + shift | 1024 new samples | ~5K | 21 us | | Front end (2 SOS) | 1024 x ~10 MACs | ~20K | 83 us | | DC removal + window | 2048 x 2 ops | ~8K | 33 us | | FFT #1 (2048-pt real) | ~11 N butterflies | ~160K | 0.67 ms | | Periodogram + EMA | 1024 bins | ~15K | 63 us | | log() of 1024 bins | ~40 cycles each | ~40K | 167 us | | FFT #2 (cepstrum) | same kernel | ~160K | 0.67 ms | | Peak search + trackers + gate | ~300 bins + scalars | ~5K | 21 us | | **Total per 64 ms hop** | | **~410K** | **~1.7 ms** | | Available per hop | | 15,360K | 64 ms | | **Utilisation** | | | **~2.7%** | Memory is equally comfortable: the 4096-sample capture buffer (int16, 8 kB), FFT input/output and window (three 2048-float arrays, 24 kB), and the averaged periodogram and cepstrum (two 1024-float arrays, 8 kB) total about 40 kB of the ESP32's 520 kB SRAM. The lesson of the budget is the same one the [pitch-detection embedded page](../pitch-detection/embedded.qmd) reached at 512 points: classical pitch estimation is nowhere near the limit of this class of hardware, which is exactly the headroom argument the [main page's honest-limits section](index.qmd#honest-limits-what-classical-buys-and-where-it-stops) makes against reaching for a neural estimator on-device. ## Porting to the NUCLEO-F446RE: a feasibility note Per [ADR-005](../../docs/adr/005-embedded-platforms.md) the workshop's second platform is the NUCLEO-F446RE (Cortex-M4F, 180 MHz, 128 kB SRAM). This system has not been ported to it; the note below is the feasibility assessment, not a report. * **Compute.** CMSIS-DSP's `arm_rfft_fast_f32` at 2048 points costs a few hundred thousand cycles; two of them plus the scalar stages lands the hop budget near 600K of the 11.5M cycles available per 64 ms, roughly 5% utilisation. No obstacle. * **Memory.** The ~40 kB working set fits the 128 kB SRAM with room for the application, but it is no longer negligible; the int16 capture buffer and float frame should not be duplicated casually. * **Plumbing, not maths.** The real porting cost is infrastructure: I2S capture moves to the SAI/I2S peripheral with DMA double-buffering (the ping-pong pattern on the [pitch-detection embedded page](../pitch-detection/embedded.qmd#i2s-audio-input)), the FreeRTOS task-and-mutex structure becomes either bare-metal ISR flags or FreeRTOS-on-STM32, and the ESP-DSP FFT-plus-bit-reverse call pairs become single `arm_rfft_fast_f32` calls. The DSP code between the transforms is portable float C and moves unchanged. * **What is lost.** The ESP32's built-in WiFi/BLE carried both the debug stream and the intended wearable output; on the Nucleo those become a UART link or an external module, which for a therapy wearable is a real system cost, not a detail. This is the honest reason the 2022 prototype was an ESP32 project. ## Credits The implementation stands on three credited third-party components: the [HampelFilter Arduino library](https://github.com/flrs/HampelFilter) by Florian Roscheck (the streaming median/MAD gate), a split-radix real FFT derived from [Robin Scheibler's ESP32 FFT work](http://www.robinscheibler.org/2017/12/12/esp32-fft.html), and I2S capture patterns from [atomic14's esp32_audio](https://github.com/atomic14/esp32_audio) examples. Everything else (the filter design, the cepstral readout, the VAD, and the system structure) is the original design this page documents.