Verification

How to check a DSP page’s numbers, and what checking this site actually found

Every page here carries numbers that were not produced by the code you can see: a cutoff frequency in a sentence, a cycle count in a budget table, a dB figure in a caption, the answer at the bottom of an exercise. Tests do not reach any of them. pytest runs the modules, and a module that passes its tests says nothing about a paragraph three lines above it claiming the correction is “a few percent”.

That gap is worse in teaching material than anywhere else. A reader who is learning cannot tell a typo from a result. They have no prior to violate: if the page says the correction factor is $(2^{1/2N}-1)^{-1/2N}$, that is simply what the correction factor is, and it goes into their notes and their code and, eventually, someone’s filter.

This page is about closing that gap. It describes the method, works one real example all the way through, and reports what the method has cost and caught on this site.

The distinction the whole method rests on

There are two ways to look at a number on a page.

Assessing it means reading the surrounding argument and judging whether the number is plausible. This is what review does, and it is nearly useless against arithmetic. A wrong number that arrived through a plausible derivation looks exactly like a right one. Reviewers, human and machine alike, agree with confident prose at a rate that should worry anyone relying on them.

Computing it means reproducing the number from first principles, without looking at the answer, and then comparing. This catches what assessment cannot, because it never consults the page’s own reasoning. The claim either falls out of the mathematics or it does not.

Everything below follows from preferring the second. It is slower, it is duller, and it is the only one that works.

A worked example

The zero-phase filtering page needs a small piece of design arithmetic. Filtering forward and then backward applies the magnitude response twice, so the cutoff moves: a filter designed for $-3$ dB at $f_c$ lands well inside $f_c$ after the second pass. To get $-3$ dB at $f_c$ from the round trip you design the single pass for a slightly higher cutoff, and the page gave the correction factor as

\[f_\text{design} = f_c \left(2^{1/2N}-1\right)^{-1/2N}\]

It is a reasonable-looking formula. It has the right shape, the right variables, and it reduces to something sensible. It is also wrong for every order above the first.

Here is the derivation done independently. A Butterworth magnitude response of order $N$ is

\[|H(\omega)|^2 = \frac{1}{1+(\omega/\omega_c)^{2N}}\]

Forward-backward filtering applies $|H|^2$, so the round-trip power response is $|H|^4$. The $-3$ dB condition on the round trip is therefore

\[\left(\frac{1}{1+(\omega/\omega_c)^{2N}}\right)^{2} = \frac{1}{2} \quad\Longrightarrow\quad 1+(\omega/\omega_c)^{2N} = \sqrt{2} \quad\Longrightarrow\quad (\omega/\omega_c)^{2N} = \sqrt{2}-1\]

which gives the correction factor

\[f_\text{design} = f_c\left(\sqrt{2}-1\right)^{-1/2N}\]

The two formulas differ in one place: the base is $\sqrt{2}-1$, a constant, not $2^{1/2N}-1$, which depends on the order. They agree at $N=1$ and diverge from there. The error is that the one-half was applied at the wrong power: to $|H|^2$ rather than to the round-trip $|H|^4$.

Note what the check did not require. No judgement about whether the formula looked right, no appeal to a source, no discussion. The derivation runs, and the two expressions either match or they do not.

Now measure the consequence, which is the part that decides whether a defect matters:

import numpy as np
from scipy import signal

fs, N, fc = 1000.0, 4, 10.0          # 4th-order, 10 Hz target, 1 kHz sampling

def round_trip_minus3db(design_fc):
    """Where the forward-backward -3 dB point actually lands."""
    b, a = signal.butter(N, design_fc / (fs / 2))
    w, h = signal.freqz(b, a, worN=200_000, fs=fs)
    db = 20 * np.log10(np.abs(h) ** 2)        # forward-backward applies |H| twice
    return w[np.argmin(np.abs(db + 3.0))]

as_published = fc * (2 ** (1 / (2 * N)) - 1) ** (-1 / (2 * N))
as_derived   = fc * (2 ** 0.5 - 1) ** (-1 / (2 * N))

print(f"design cutoff   as published {as_published:6.3f} Hz    as derived {as_derived:6.3f} Hz")
print(f"-3 dB lands at  as published {round_trip_minus3db(as_published):6.3f} Hz"
      f"    as derived {round_trip_minus3db(as_derived):6.3f} Hz")
print(f"target was      {fc:.3f} Hz")

assert abs(round_trip_minus3db(as_derived) - fc) < 0.02, "derived formula must hit the target"
assert round_trip_minus3db(as_published) > 1.15 * fc, "published formula must miss it badly"

design cutoff   as published 13.503 Hz    as derived 11.165 Hz
-3 dB lands at  as published 12.090 Hz    as derived  9.995 Hz
target was      10.000 Hz

The corrected formula lands on 10 Hz. The published one lands at 12.09 Hz, a 21% error in the quantity the reader asked for, at an order (fourth) that anyone filtering physiological or audio data reaches within a week.

The same check condemned a second sentence on that page as collateral. The text described the correction as “small, a few percent for $N \geq 4$”. Having a correct formula makes that measurable:

for order in (2, 4, 8, 16):
    factor = (2 ** 0.5 - 1) ** (-1 / (2 * order))
    print(f"N = {order:2d}   correction = {100 * (factor - 1):5.2f}%")

assert abs(100 * ((2 ** 0.5 - 1) ** (-1 / 8) - 1) - 11.65) < 0.01

N =  2   correction = 24.65%
N =  4   correction = 11.65%
N =  8   correction =  5.66%
N = 16   correction =  2.79%

At $N=4$ the correction is 11.65%, not a few percent. The prose was calibrated against an intuition, and the intuition was for a different order.

This is what the record looks like

The audit trail for this example lives in _vv/zero-phase.md: the claim as published, the value recomputed, the measured consequence, and the fix. Every number-bearing page has such a file. They are kept out of the site itself, because they are process records rather than lessons, but they are in the repository and they are what makes the claim on this page checkable rather than merely stated.

Four channels, covering different things

No single mechanism reaches all of it, so this site runs four, each aimed at what the others structurally miss.

pytest over the modules. The ordinary one. Every topic’s .py is tested, and CI blocks the deploy on a failure. It reaches implementations and nothing else.

Assertions inside the pages. Code blocks in a .qmd execute when the site is built, so an assert in a page runs on every build, in CI, before anything ships. This is how a figure’s caption gets tied to the figure: if a caption claims a 30-fold sharpening, an assertion on the same page measures the sharpening and the build fails when the two part company. The two cells above are doing exactly this; the numbers in the prose around them cannot drift without turning the site red.

Recomputation of prose numbers. The channel this page is about, and the only one that reaches captions, budget tables and worked solutions. It is run by an agent instructed to compute rather than assess, and its output is the per-topic record in _vv/.

Adversarial multi-model review. Several models read the same page through different lenses (mathematics, embedded C, pedagogy, fabricated citations), and every finding any of them raises is then handed to a different model told to refute it. Roughly half of all findings do not survive that step, which is the point: a review pass that trusts its own output is a review pass that generates work.

What it has cost, and what it caught

Two full sweeps have been run across every page carrying equations or hand-authored numbers.

	Checks	Defects confirmed
First sweep, topics and basics	390	19
Second sweep, adding the exercise sets	1256	33

Roughly 1600 checks, 52 numbers that did not reproduce. That is a rate of about one defect for every thirty numbers on a site written carefully by someone who knows the material.

The distribution is more useful than the total. Sixteen of the second sweep’s thirty-three defects were in the exercise sets, which is where a reader is least able to defend themselves: a worked solution is consulted precisely when the reader does not yet know the answer. It is also the surface with the least test coverage, since pytest reaches modules and a published solution is prose. The most exposed material was the material nothing was watching.

Three failure modes that keep coming back

Across those sweeps the same shapes recur, and none of them is the kind of mistake review catches, because in every case the reasoning is sound and the artifact is still wrong.

A bound quoted for the wrong parameterisation. A Cramér-Rao bound is a statement about a specific parameter in specific units. Quoted from memory into a page that parameterises differently, it is off by a constant, and the constant is invisible because the formula is otherwise correct. On this site the same factor of two was introduced twice in one session, in the frequency bound and then in the phase bound, and was caught only by building the Fisher information matrix numerically and dividing.

A statistic calibrated on an ingredient rather than as computed. A threshold is designed against the distribution of something adjacent to what the code actually evaluates: the marginal of one candidate rather than the maximum over a window, the raw sample count rather than the effective degrees of freedom left after a filter. Three occurrences here. In all three the ingredient’s closed form was exactly right, and the deployed threshold was still wrong, by factors of 3.6, of 2, and once of thirteen orders of magnitude. The deeper the threshold, the more of the answer is decided by a tail nobody checked.

A cross-check less accurate than the thing it grades. An independent calculation brought in to referee a closed form needs its own accuracy established first. A numerical method here, introduced to validate a run-length formula, was itself unconverged and graded a formula accurate to 0.02% as being 0.2% wrong. The referee was the error.

What is still not verified

The honest inventory, because a page about verification that only lists successes is advertising.

Almost none of the C has been run. Every embedded page now states how far its code has actually been taken, in one line, under a four-state scheme (ADR-005 section 7). As of this writing, no page claims the top state: the C here is transcribed and checked against tested Python models, not measured on a bench. Three pages sit at the bottom state, with performance figures that are hand calculations and no executable model behind them. Those three carry more hand-authored performance numbers than any others on the site, which is the sort of thing an audit tells you and intuition does not.

Rendering is not verified by the tools that verify the mathematics. Diagrams were checked for years by rasterising them locally, and the local renderer honoured an SVG attribute that browsers ignore, so every subscript and exponent in six figures was flat in Chrome and Safari while every check passed. The verification loop agreed with itself and disagreed with reality. That class of error is still open: there is no automated check that the deployed page looks like the intended page.

Worked solutions remain the thinnest surface. They were swept twice and corrected, but nothing runs them on every build the way an assertion in a page is run. A defect reintroduced there today would ship.

None of this makes the material unreliable. It makes the reliability bounded, and the point of stating the bounds is that a reader can then decide what to trust and what to check themselves, which is in any case the only durable position to hold toward someone else’s numbers.

--- title: "Verification" subtitle: "How to check a DSP page's numbers, and what checking this site actually found" --- Every page here carries numbers that were not produced by the code you can see: a cutoff frequency in a sentence, a cycle count in a budget table, a dB figure in a caption, the answer at the bottom of an exercise. Tests do not reach any of them. `pytest` runs the modules, and a module that passes its tests says nothing about a paragraph three lines above it claiming the correction is "a few percent". That gap is worse in teaching material than anywhere else. A reader who is learning cannot tell a typo from a result. They have no prior to violate: if the page says the correction factor is $(2^{1/2N}-1)^{-1/2N}$, that is simply what the correction factor is, and it goes into their notes and their code and, eventually, someone's filter. This page is about closing that gap. It describes the method, works one real example all the way through, and reports what the method has cost and caught on this site. <hr> ## The distinction the whole method rests on There are two ways to look at a number on a page. **Assessing** it means reading the surrounding argument and judging whether the number is plausible. This is what review does, and it is nearly useless against arithmetic. A wrong number that arrived through a plausible derivation looks exactly like a right one. Reviewers, human and machine alike, agree with confident prose at a rate that should worry anyone relying on them. **Computing** it means reproducing the number from first principles, without looking at the answer, and then comparing. This catches what assessment cannot, because it never consults the page's own reasoning. The claim either falls out of the mathematics or it does not. Everything below follows from preferring the second. It is slower, it is duller, and it is the only one that works. <hr> ## A worked example The [zero-phase filtering](topics/zero-phase/index.qmd) page needs a small piece of design arithmetic. Filtering forward and then backward applies the magnitude response twice, so the cutoff moves: a filter designed for $-3$ dB at $f_c$ lands well inside $f_c$ after the second pass. To get $-3$ dB at $f_c$ from the round trip you design the single pass for a slightly higher cutoff, and the page gave the correction factor as $$f_\text{design} = f_c \left(2^{1/2N}-1\right)^{-1/2N}$$ It is a reasonable-looking formula. It has the right shape, the right variables, and it reduces to something sensible. It is also wrong for every order above the first. Here is the derivation done independently. A Butterworth magnitude response of order $N$ is $$|H(\omega)|^2 = \frac{1}{1+(\omega/\omega_c)^{2N}}$$ Forward-backward filtering applies $|H|^2$, so the round-trip power response is $|H|^4$. The $-3$ dB condition on the round trip is therefore $$\left(\frac{1}{1+(\omega/\omega_c)^{2N}}\right)^{2} = \frac{1}{2} \quad\Longrightarrow\quad 1+(\omega/\omega_c)^{2N} = \sqrt{2} \quad\Longrightarrow\quad (\omega/\omega_c)^{2N} = \sqrt{2}-1$$ which gives the correction factor $$f_\text{design} = f_c\left(\sqrt{2}-1\right)^{-1/2N}$$ The two formulas differ in one place: the base is $\sqrt{2}-1$, a constant, not $2^{1/2N}-1$, which depends on the order. They agree at $N=1$ and diverge from there. The error is that the one-half was applied at the wrong power: to $|H|^2$ rather than to the round-trip $|H|^4$. Note what the check did *not* require. No judgement about whether the formula looked right, no appeal to a source, no discussion. The derivation runs, and the two expressions either match or they do not. Now measure the consequence, which is the part that decides whether a defect matters: ```{python} #| label: zero-phase-correction #| code-fold: false import numpy as np from scipy import signal fs, N, fc = 1000.0, 4, 10.0 # 4th-order, 10 Hz target, 1 kHz sampling def round_trip_minus3db(design_fc): """Where the forward-backward -3 dB point actually lands.""" b, a = signal.butter(N, design_fc / (fs / 2)) w, h = signal.freqz(b, a, worN=200_000, fs=fs) db = 20 * np.log10(np.abs(h) ** 2) # forward-backward applies |H| twice return w[np.argmin(np.abs(db + 3.0))] as_published = fc * (2 ** (1 / (2 * N)) - 1) ** (-1 / (2 * N)) as_derived = fc * (2 ** 0.5 - 1) ** (-1 / (2 * N)) print(f"design cutoff as published {as_published:6.3f} Hz as derived {as_derived:6.3f} Hz") print(f"-3 dB lands at as published {round_trip_minus3db(as_published):6.3f} Hz" f" as derived {round_trip_minus3db(as_derived):6.3f} Hz") print(f"target was {fc:.3f} Hz") assert abs(round_trip_minus3db(as_derived) - fc) < 0.02, "derived formula must hit the target" assert round_trip_minus3db(as_published) > 1.15 * fc, "published formula must miss it badly" ``` The corrected formula lands on 10 Hz. The published one lands at 12.09 Hz, a 21% error in the quantity the reader asked for, at an order (fourth) that anyone filtering physiological or audio data reaches within a week. The same check condemned a second sentence on that page as collateral. The text described the correction as "small, a few percent for $N \geq 4$". Having a correct formula makes that measurable: ```{python} #| label: correction-magnitude #| code-fold: false for order in (2, 4, 8, 16): factor = (2 ** 0.5 - 1) ** (-1 / (2 * order)) print(f"N = {order:2d} correction = {100 * (factor - 1):5.2f}%") assert abs(100 * ((2 ** 0.5 - 1) ** (-1 / 8) - 1) - 11.65) < 0.01 ``` At $N=4$ the correction is 11.65%, not a few percent. The prose was calibrated against an intuition, and the intuition was for a different order. ::: {.callout-note title="This is what the record looks like"} The audit trail for this example lives in [`_vv/zero-phase.md`](https://github.com/ducroq/dsp-workshop/blob/main/_vv/zero-phase.md): the claim as published, the value recomputed, the measured consequence, and the fix. Every number-bearing page has such a file. They are kept out of the site itself, because they are process records rather than lessons, but they are in the repository and they are what makes the claim on this page checkable rather than merely stated. ::: <hr> ## Four channels, covering different things No single mechanism reaches all of it, so this site runs four, each aimed at what the others structurally miss. **`pytest` over the modules.** The ordinary one. Every topic's `.py` is tested, and CI blocks the deploy on a failure. It reaches implementations and nothing else. **Assertions inside the pages.** Code blocks in a `.qmd` execute when the site is built, so an `assert` in a page runs on every build, in CI, before anything ships. This is how a figure's caption gets tied to the figure: if a caption claims a 30-fold sharpening, an assertion on the same page measures the sharpening and the build fails when the two part company. The two cells above are doing exactly this; the numbers in the prose around them cannot drift without turning the site red. **Recomputation of prose numbers.** The channel this page is about, and the only one that reaches captions, budget tables and worked solutions. It is run by an agent instructed to compute rather than assess, and its output is the per-topic record in `_vv/`. **Adversarial multi-model review.** Several models read the same page through different lenses (mathematics, embedded C, pedagogy, fabricated citations), and every finding any of them raises is then handed to a different model told to *refute* it. Roughly half of all findings do not survive that step, which is the point: a review pass that trusts its own output is a review pass that generates work. <hr> ## What it has cost, and what it caught Two full sweeps have been run across every page carrying equations or hand-authored numbers. | | Checks | Defects confirmed | |---|---:|---:| | First sweep, topics and basics | 390 | 19 | | Second sweep, adding the exercise sets | 1256 | 33 | Roughly 1600 checks, 52 numbers that did not reproduce. That is a rate of about one defect for every thirty numbers on a site written carefully by someone who knows the material. The distribution is more useful than the total. **Sixteen of the second sweep's thirty-three defects were in the exercise sets**, which is where a reader is least able to defend themselves: a worked solution is consulted precisely when the reader does not yet know the answer. It is also the surface with the least test coverage, since `pytest` reaches modules and a published solution is prose. The most exposed material was the material nothing was watching. <hr> ## Three failure modes that keep coming back Across those sweeps the same shapes recur, and none of them is the kind of mistake review catches, because in every case the reasoning is sound and the artifact is still wrong. **A bound quoted for the wrong parameterisation.** A Cramér-Rao bound is a statement about a specific parameter in specific units. Quoted from memory into a page that parameterises differently, it is off by a constant, and the constant is invisible because the formula is otherwise correct. On this site the same factor of two was introduced twice in one session, in the frequency bound and then in the phase bound, and was caught only by building the Fisher information matrix numerically and dividing. **A statistic calibrated on an ingredient rather than as computed.** A threshold is designed against the distribution of something adjacent to what the code actually evaluates: the marginal of one candidate rather than the maximum over a window, the raw sample count rather than the effective degrees of freedom left after a filter. Three occurrences here. In all three the ingredient's closed form was exactly right, and the deployed threshold was still wrong, by factors of 3.6, of 2, and once of thirteen orders of magnitude. The deeper the threshold, the more of the answer is decided by a tail nobody checked. **A cross-check less accurate than the thing it grades.** An independent calculation brought in to referee a closed form needs its own accuracy established first. A numerical method here, introduced to validate a run-length formula, was itself unconverged and graded a formula accurate to 0.02% as being 0.2% wrong. The referee was the error. <hr> ## What is still not verified The honest inventory, because a page about verification that only lists successes is advertising. **Almost none of the C has been run.** Every embedded page now states how far its code has actually been taken, in one line, under a four-state scheme ([ADR-005](docs/adr/005-embedded-platforms.md) section 7). As of this writing, **no page claims the top state**: the C here is transcribed and checked against tested Python models, not measured on a bench. Three pages sit at the bottom state, with performance figures that are hand calculations and no executable model behind them. Those three carry more hand-authored performance numbers than any others on the site, which is the sort of thing an audit tells you and intuition does not. **Rendering is not verified by the tools that verify the mathematics.** Diagrams were checked for years by rasterising them locally, and the local renderer honoured an SVG attribute that browsers ignore, so every subscript and exponent in six figures was flat in Chrome and Safari while every check passed. The verification loop agreed with itself and disagreed with reality. That class of error is still open: there is no automated check that the deployed page looks like the intended page. **Worked solutions remain the thinnest surface.** They were swept twice and corrected, but nothing runs them on every build the way an assertion in a page is run. A defect reintroduced there today would ship. None of this makes the material unreliable. It makes the reliability *bounded*, and the point of stating the bounds is that a reader can then decide what to trust and what to check themselves, which is in any case the only durable position to hold toward someone else's numbers.