A real-time spectrogram as a block FFT in a loop, on STM32F4 and ESP32-S3
A spectrogram on a microcontroller is, at heart, a block FFT in a loop: gather a frame of samples, window it, transform it, take the magnitude, advance by one hop, repeat. The FFT itself is a library call on both default platforms. What the desktop version hides, and what actually decides whether this runs on a given chip, is three things: where the overlapping samples are kept between frames, how much RAM the frame buffers eat, and whether the FFT keeps up with the audio.
For the transform itself, the resolution trade-off, and the Python prototypes, see the main STFT page. This page is the metal.
The two platforms follow ADR-005: NUCLEO-F446RE (Cortex-M4F, 180 MHz) and ESP32-S3, single-precision float first. Both have an FPU and a vendor DSP library with a fast real FFT (CMSIS-DSP and ESP-DSP), so the float path is the natural one.
The real-time loop: a ring buffer for overlap
The catch with overlap is that consecutive frames share samples. At 50% overlap, half of each frame is the second half of the previous one. You do not want to re-read or re-copy those samples from the source each time; you want them already in place. A ring buffer of one frame length does this: samples arrive from the ADC or I2S into the ring, and when a full hop of new samples has landed, you copy out the most recent \(L\) samples (wrapping around the ring) into a contiguous work buffer, window them, and FFT.
The shape of the loop:
#define NFFT 1024// frame length L (power of two for the FFT)#define HOP 512// 50% overlap#define NBINS (NFFT/2+1)// one-sided spectrumstaticfloat ring[NFFT];// holds the last NFFT samplesstaticint ring_pos =0;// write cursorstaticint since_hop =0;// samples since the last frame was emittedstaticfloat window[NFFT];// precomputed once (Hann)staticfloat frame[NFFT];// contiguous, windowed work bufferstaticfloat mag[NBINS];// magnitude spectrum out// called once per incoming sample (from the ADC ISR or I2S callback)void stft_push(float x){ ring[ring_pos]= x; ring_pos =(ring_pos +1)% NFFT;if(++since_hop < HOP)return;// not a frame boundary yet since_hop =0;// copy the most recent NFFT samples, oldest first, applying the windowfor(int n =0; n < NFFT; n++){int idx =(ring_pos + n)% NFFT;// ring_pos points at the oldest sample frame[n]= ring[idx]* window[n];} fft_magnitude(frame, mag);// platform FFT, see below// mag[] is now one spectrogram column; consume it (display, feature, log)}
Two details matter. The window table is computed once at startup, not per frame: a Hann window is 0.5f * (1.0f - cosf(2.0f * M_PI * n / NFFT)). And after writing the newest sample, ring_pos points at the oldest sample still in the ring, which is exactly where a frame should start, so the copy loop reads oldest-to-newest with no separate bookkeeping.
Two caveats come with this skeleton. The ring starts all zeros, so the very first emitted frame blends HOP real samples with NFFT - HOP zeros left over from initialisation; that first spectrogram column reads low and should be discarded, since it is normal cold-start padding that is gone once a full NFFT of real audio has arrived. And stft_push runs in interrupt context and keeps all its state in file-scope statics (the ring, the cursors, and fft_magnitude’s own work buffer), so it is not reentrant: only one interrupt source may call it, and nothing may preempt it mid-frame. On the M4 that means giving the audio interrupt a fixed NVIC preemption priority and not sharing these buffers with another context. On the ESP32-S3 there is a second requirement: a function called from an ISR (and everything it calls) must live in IRAM, so mark stft_push and its callees IRAM_ATTR, or a flash-cache miss during an OTA or NVS write will fault mid-interrupt.
Memory budget
The frame length sets the RAM. Everything scales with \(L\): the ring, the window table, the windowed frame, the FFT output buffer, and the output bins.
NFFT =1024flt =4# bytes per float32ring = NFFT * flt # overlap ring bufferwindow = NFFT * flt # precomputed Hann tableframe = NFFT * flt # contiguous windowed inputspec = NFFT * flt # CMSIS packed-complex FFT output (not in-place)mag = (NFFT //2+1) * flt # one-sided magnitudetotal = ring + window + frame + spec + magprint(f"NFFT={NFFT}: ring {ring//1024} + window {window//1024} + frame {frame//1024} "f"+ spec {spec//1024} + mag {mag/1024:.1f} KB = {total/1024:.1f} KB")# the F446RE has 128 KB SRAM, the ESP32-S3 has 512 KB; an NFFT=1024 float# spectrogram front end is a small fraction of eitherassert total <20*1024# under 20 KBassert total <0.20*128*1024# under 20% of the F446RE's SRAMprint(f"-> {100*total/(128*1024):.1f}% of the F446RE's 128 KB SRAM")
NFFT=1024: ring 4 + window 4 + frame 4 + spec 4 + mag 2.0 KB = 18.0 KB
-> 14.1% of the F446RE's 128 KB SRAM
The arm_rfft_fast_f32 shown below is not in-place: it reads the windowed frame and writes a separate spec buffer, so both count. The frame length is then bounded by your time-frequency needs, not by memory, until you reach into the multi-thousand-point frames that fine low-frequency analysis wants. Doubling \(L\) doubles the RAM and (roughly) the FFT cost while halving \(\Delta f\).
Keeping up: the per-hop time budget
The FFT does not run every sample; it runs once per hop. That is the deadline. At a 16 kHz sample rate with a 512-sample hop, a new frame is due every 32 ms, and at 180 MHz that is millions of cycles of headroom for a single 1024-point FFT that costs only thousands.
fs =16000.0HOP =512f_cpu =180e6# F446RE core clockhop_period = HOP / fs # seconds between framescycles_per_hop = hop_period * f_cpu # cycles available before the next frameprint(f"hop every {1000*hop_period:.1f} ms -> {cycles_per_hop:,.0f} cycles at 180 MHz")# a radix-2 N-point complex FFT needs (N/2) log2 N butterflies; a real-input# transform (arm_rfft_fast_f32) exploits conjugate symmetry and does roughly# half that, about (N/4) log2 N. This is the *shape* of the cost, not a# benchmark: the true count depends on the CMSIS-DSP/ESP-DSP implementation.NFFT =1024butterflies = (NFFT //4) *int(np.log2(NFFT))print(f"NFFT={NFFT}: ~{butterflies:,} butterflies (order N/4 log2 N for a real FFT)")assert cycles_per_hop >100* butterflies # generous margin even at tens of cycles/butterflyprint("the FFT fits the hop budget with a wide margin")
hop every 32.0 ms -> 5,760,000 cycles at 180 MHz
NFFT=1024: ~2,560 butterflies (order N/4 log2 N for a real FFT)
the FFT fits the hop budget with a wide margin
The budget tightens with a smaller hop (more overlap), a higher sample rate, or a longer frame, and it competes with whatever else the core is doing per frame (the feature extraction or classifier the spectrogram feeds). The honest check on real hardware is a cycle counter (DWT->CYCCNT on the M4) around the FFT call, not an estimate.
NUCLEO-F446RE with CMSIS-DSP
CMSIS-DSP provides arm_rfft_fast_f32, a real-input FFT that returns the one-sided spectrum. Initialise the instance once, then call it per frame.
#include "arm_math.h"#include <math.h>// fabsf (and cosf/M_PI for the window)// The Cortex-M4F FPU is disabled at reset. CubeMX's SystemInit() enables it;// a bare-metal project must do so before the first float instruction:// SCB->CPACR |= (0xFUL << 20); __DSB(); __ISB();#define NFFT 1024// as in the ring-buffer skeleton above#define NBINS (NFFT /2+1)static arm_rfft_fast_instance_f32 fft;void fft_init(void){// call once at startup (from main()/init); arm_rfft_fast_init_f32(&fft, NFFT);// NFFT must be a power of two in 32..4096}void fft_magnitude(float*frame,float*mag){staticfloat spec[NFFT];// packed complex output arm_rfft_fast_f32(&fft, frame, spec,0);// 0 = forward// CMSIS packs the real-FFT output as:// spec[0] = DC (real), spec[1] = Nyquist (real),// then interleaved Re,Im for bins 1 .. NFFT/2 - 1 mag[0]= fabsf(spec[0]);// DC mag[NFFT/2]= fabsf(spec[1]);// Nyquist arm_cmplx_mag_f32(&spec[2],&mag[1], NFFT/2-1);// the rest}
The one thing that trips everyone the first time is that packing. arm_rfft_fast_f32 does not give you a tidy array of NFFT/2+1 complex numbers: it folds the purely-real DC and Nyquist bins into spec[0] and spec[1], then interleaves real and imaginary parts for the bins in between. Feed that array straight into arm_cmplx_mag_f32 without separating out DC and Nyquist and the first two magnitudes come out wrong. Handle those two bins by hand, as above, and let arm_cmplx_mag_f32 do the interleaved remainder.
ESP32-S3 with ESP-DSP
ESP-DSP (Espressif’s DSP library) offers a comparable real FFT and a windowing helper. The structure mirrors the CMSIS path: initialise tables once, then window, transform, and combine per frame.
#include "esp_dsp.h"#include <math.h>// sqrtfdsps_fft2r_init_fc32(NULL, NFFT);// once, at startup: FFT twiddle tables// ESP-DSP's radix-2 FFT works on interleaved complex (re, im, re, im ...). Pack// the windowed real frame into the real slots, zero the imaginary slots, run the// transform, bit-reverse to unscramble, then take the magnitude of the one-sided// spectrum (bins 0 .. NFFT/2). The frame is windowed by the caller, exactly as on// the CMSIS path above. This is the same packing the MFCC page uses for power.void fft_magnitude(float*frame,float*mag){staticfloat z[2* NFFT];// interleaved (re, im)for(int n =0; n < NFFT; n++){ z[2*n]= frame[n]; z[2*n+1]=0.0f;} dsps_fft2r_fc32(z, NFFT);// in-place radix-2 FFT dsps_bit_rev_fc32(z, NFFT);// unscramble the bit-reversed orderfor(int k =0; k <= NFFT/2; k++)// one-sided magnitude, NFFT/2 + 1 bins mag[k]= sqrtf(z[2*k]* z[2*k]+ z[2*k+1]* z[2*k+1]);}
ESP-DSP is SIMD-accelerated on the ESP32-S3’s extended instruction set, so the FFT and windowing run noticeably faster than scalar C, but the same hop-budget arithmetic applies. As with CMSIS, verify the exact output bin layout against the library’s own example before trusting the magnitudes; the two libraries do not pack the spectrum the same way.
When there is no FPU, or no float at all
On a smaller core (a Cortex-M0+ or an AVR), a 1024-point float FFT is either slow or impossible, and you move to a fixed-point (Q15) FFT: arm_rfft_q15 on CMSIS, or an integer FFT of your own. The frame buffers halve in size (int16_t rather than float), but every stage now needs scaling to stay inside the 16-bit range, and the windowing and magnitude steps inherit the finite word-length effects that the float path lets you ignore. The architecture above (ring buffer, hop, per-frame FFT) is unchanged; only the arithmetic type and its scaling discipline differ.
References
Source Code
---title: "The STFT on Hardware"subtitle: "A real-time spectrogram as a block FFT in a loop, on STM32F4 and ESP32-S3"bibliography: ../../references.bib---A spectrogram on a microcontroller is, at heart, a block FFT in a loop: gather a frame of samples, window it, transform it, take the magnitude, advance by one hop, repeat. The FFT itself is a library call on both default platforms. What the desktop version hides, and what actually decides whether this runs on a given chip, is three things: where the **overlapping** samples are kept between frames, how much **RAM** the frame buffers eat, and whether the FFT **keeps up** with the audio.For the transform itself, the resolution trade-off, and the Python prototypes, see the [main STFT page](index.qmd). This page is the metal.The two platforms follow [ADR-005](../../docs/adr/005-embedded-platforms.md): NUCLEO-F446RE (Cortex-M4F, 180 MHz) and ESP32-S3, single-precision `float` first. Both have an FPU and a vendor DSP library with a fast real FFT (CMSIS-DSP and ESP-DSP), so the float path is the natural one.```{python}#| echo: falseimport numpy as np```<hr>## The real-time loop: a ring buffer for overlapThe catch with overlap is that consecutive frames share samples. At 50% overlap, half of each frame is the second half of the previous one. You do not want to re-read or re-copy those samples from the source each time; you want them already in place. A **ring buffer** of one frame length does this: samples arrive from the ADC or I2S into the ring, and when a full hop of new samples has landed, you copy out the most recent $L$ samples (wrapping around the ring) into a contiguous work buffer, window them, and FFT.The shape of the loop:```c#define NFFT 1024// frame length L (power of two for the FFT)#define HOP 512// 50% overlap#define NBINS (NFFT/2+1)// one-sided spectrumstaticfloat ring[NFFT];// holds the last NFFT samplesstaticint ring_pos =0;// write cursorstaticint since_hop =0;// samples since the last frame was emittedstaticfloat window[NFFT];// precomputed once (Hann)staticfloat frame[NFFT];// contiguous, windowed work bufferstaticfloat mag[NBINS];// magnitude spectrum out// called once per incoming sample (from the ADC ISR or I2S callback)void stft_push(float x){ ring[ring_pos]= x; ring_pos =(ring_pos +1)% NFFT;if(++since_hop < HOP)return;// not a frame boundary yet since_hop =0;// copy the most recent NFFT samples, oldest first, applying the windowfor(int n =0; n < NFFT; n++){int idx =(ring_pos + n)% NFFT;// ring_pos points at the oldest sample frame[n]= ring[idx]* window[n];} fft_magnitude(frame, mag);// platform FFT, see below// mag[] is now one spectrogram column; consume it (display, feature, log)}```Two details matter. The window table is computed **once** at startup, not per frame: a Hann window is `0.5f * (1.0f - cosf(2.0f * M_PI * n / NFFT))`. And after writing the newest sample, `ring_pos` points at the *oldest* sample still in the ring, which is exactly where a frame should start, so the copy loop reads oldest-to-newest with no separate bookkeeping.Two caveats come with this skeleton. The ring starts all zeros, so the very first emitted frame blends `HOP` real samples with `NFFT - HOP` zeros left over from initialisation; that first spectrogram column reads low and should be discarded, since it is normal cold-start padding that is gone once a full `NFFT` of real audio has arrived. And `stft_push` runs in interrupt context and keeps all its state in file-scope statics (the ring, the cursors, and `fft_magnitude`'s own work buffer), so it is not reentrant: only one interrupt source may call it, and nothing may preempt it mid-frame. On the M4 that means giving the audio interrupt a fixed NVIC preemption priority and not sharing these buffers with another context. On the ESP32-S3 there is a second requirement: a function called from an ISR (and everything it calls) must live in IRAM, so mark `stft_push` and its callees `IRAM_ATTR`, or a flash-cache miss during an OTA or NVS write will fault mid-interrupt.## Memory budgetThe frame length sets the RAM. Everything scales with $L$: the ring, the window table, the windowed frame, the FFT output buffer, and the output bins.```{python}#| label: stft-memory-budgetNFFT =1024flt =4# bytes per float32ring = NFFT * flt # overlap ring bufferwindow = NFFT * flt # precomputed Hann tableframe = NFFT * flt # contiguous windowed inputspec = NFFT * flt # CMSIS packed-complex FFT output (not in-place)mag = (NFFT //2+1) * flt # one-sided magnitudetotal = ring + window + frame + spec + magprint(f"NFFT={NFFT}: ring {ring//1024} + window {window//1024} + frame {frame//1024} "f"+ spec {spec//1024} + mag {mag/1024:.1f} KB = {total/1024:.1f} KB")# the F446RE has 128 KB SRAM, the ESP32-S3 has 512 KB; an NFFT=1024 float# spectrogram front end is a small fraction of eitherassert total <20*1024# under 20 KBassert total <0.20*128*1024# under 20% of the F446RE's SRAMprint(f"-> {100*total/(128*1024):.1f}% of the F446RE's 128 KB SRAM")```The `arm_rfft_fast_f32` shown below is not in-place: it reads the windowed `frame` and writes a separate `spec` buffer, so both count. The frame length is then bounded by your time-frequency needs, not by memory, until you reach into the multi-thousand-point frames that fine low-frequency analysis wants. Doubling $L$ doubles the RAM and (roughly) the FFT cost while halving $\Delta f$.## Keeping up: the per-hop time budgetThe FFT does not run every sample; it runs once per hop. That is the deadline. At a 16 kHz sample rate with a 512-sample hop, a new frame is due every 32 ms, and at 180 MHz that is millions of cycles of headroom for a single 1024-point FFT that costs only thousands.```{python}#| label: stft-time-budgetfs =16000.0HOP =512f_cpu =180e6# F446RE core clockhop_period = HOP / fs # seconds between framescycles_per_hop = hop_period * f_cpu # cycles available before the next frameprint(f"hop every {1000*hop_period:.1f} ms -> {cycles_per_hop:,.0f} cycles at 180 MHz")# a radix-2 N-point complex FFT needs (N/2) log2 N butterflies; a real-input# transform (arm_rfft_fast_f32) exploits conjugate symmetry and does roughly# half that, about (N/4) log2 N. This is the *shape* of the cost, not a# benchmark: the true count depends on the CMSIS-DSP/ESP-DSP implementation.NFFT =1024butterflies = (NFFT //4) *int(np.log2(NFFT))print(f"NFFT={NFFT}: ~{butterflies:,} butterflies (order N/4 log2 N for a real FFT)")assert cycles_per_hop >100* butterflies # generous margin even at tens of cycles/butterflyprint("the FFT fits the hop budget with a wide margin")```The budget tightens with a smaller hop (more overlap), a higher sample rate, or a longer frame, and it competes with whatever else the core is doing per frame (the feature extraction or classifier the spectrogram feeds). The honest check on real hardware is a cycle counter (`DWT->CYCCNT` on the M4) around the FFT call, not an estimate.## NUCLEO-F446RE with CMSIS-DSPCMSIS-DSP provides `arm_rfft_fast_f32`, a real-input FFT that returns the one-sided spectrum. Initialise the instance once, then call it per frame.```c#include "arm_math.h"#include <math.h>// fabsf (and cosf/M_PI for the window)// The Cortex-M4F FPU is disabled at reset. CubeMX's SystemInit() enables it;// a bare-metal project must do so before the first float instruction:// SCB->CPACR |= (0xFUL << 20); __DSB(); __ISB();#define NFFT 1024// as in the ring-buffer skeleton above#define NBINS (NFFT /2+1)static arm_rfft_fast_instance_f32 fft;void fft_init(void){// call once at startup (from main()/init); arm_rfft_fast_init_f32(&fft, NFFT);// NFFT must be a power of two in 32..4096}void fft_magnitude(float*frame,float*mag){staticfloat spec[NFFT];// packed complex output arm_rfft_fast_f32(&fft, frame, spec,0);// 0 = forward// CMSIS packs the real-FFT output as:// spec[0] = DC (real), spec[1] = Nyquist (real),// then interleaved Re,Im for bins 1 .. NFFT/2 - 1 mag[0]= fabsf(spec[0]);// DC mag[NFFT/2]= fabsf(spec[1]);// Nyquist arm_cmplx_mag_f32(&spec[2],&mag[1], NFFT/2-1);// the rest}```The one thing that trips everyone the first time is that **packing**. `arm_rfft_fast_f32` does not give you a tidy array of `NFFT/2+1` complex numbers: it folds the purely-real DC and Nyquist bins into `spec[0]` and `spec[1]`, then interleaves real and imaginary parts for the bins in between. Feed that array straight into `arm_cmplx_mag_f32` without separating out DC and Nyquist and the first two magnitudes come out wrong. Handle those two bins by hand, as above, and let `arm_cmplx_mag_f32` do the interleaved remainder.## ESP32-S3 with ESP-DSPESP-DSP (Espressif's DSP library) offers a comparable real FFT and a windowing helper. The structure mirrors the CMSIS path: initialise tables once, then window, transform, and combine per frame.```c#include "esp_dsp.h"#include <math.h>// sqrtfdsps_fft2r_init_fc32(NULL, NFFT);// once, at startup: FFT twiddle tables// ESP-DSP's radix-2 FFT works on interleaved complex (re, im, re, im ...). Pack// the windowed real frame into the real slots, zero the imaginary slots, run the// transform, bit-reverse to unscramble, then take the magnitude of the one-sided// spectrum (bins 0 .. NFFT/2). The frame is windowed by the caller, exactly as on// the CMSIS path above. This is the same packing the MFCC page uses for power.void fft_magnitude(float*frame,float*mag){staticfloat z[2* NFFT];// interleaved (re, im)for(int n =0; n < NFFT; n++){ z[2*n]= frame[n]; z[2*n+1]=0.0f;} dsps_fft2r_fc32(z, NFFT);// in-place radix-2 FFT dsps_bit_rev_fc32(z, NFFT);// unscramble the bit-reversed orderfor(int k =0; k <= NFFT/2; k++)// one-sided magnitude, NFFT/2 + 1 bins mag[k]= sqrtf(z[2*k]* z[2*k]+ z[2*k+1]* z[2*k+1]);}```ESP-DSP is SIMD-accelerated on the ESP32-S3's extended instruction set, so the FFT and windowing run noticeably faster than scalar C, but the same hop-budget arithmetic applies. As with CMSIS, verify the exact output bin layout against the library's own example before trusting the magnitudes; the two libraries do not pack the spectrum the same way.## When there is no FPU, or no float at allOn a smaller core (a Cortex-M0+ or an AVR), a 1024-point float FFT is either slow or impossible, and you move to a fixed-point (Q15) FFT: `arm_rfft_q15` on CMSIS, or an integer FFT of your own. The frame buffers halve in size (`int16_t` rather than `float`), but every stage now needs scaling to stay inside the 16-bit range, and the windowing and magnitude steps inherit the [finite word-length effects](../finite-wordlength/index.qmd) that the float path lets you ignore. The architecture above (ring buffer, hop, per-frame FFT) is unchanged; only the arithmetic type and its scaling discipline differ.## References::: {#refs}:::