Verification

How to check a DSP page’s numbers, and what checking this site actually found

Every page here carries numbers that were not produced by the code you can see: a cutoff frequency in a sentence, a cycle count in a budget table, a dB figure in a caption, the answer at the bottom of an exercise. Ordinary unit tests do not reach them. pytest runs the modules, and a module that passes its tests says nothing about a paragraph three lines above it claiming the correction is “a few percent”. Two of the channels described below do reach prose numbers, and both were built for that reason; neither is what “running the tests” ordinarily means.

That gap is worse in teaching material than anywhere else. A reader who is learning cannot tell a typo from a result. They have no prior to violate: if the page says the correction factor is $(2^{1/2N}-1)^{-1/2N}$, that is simply what the correction factor is, and it goes into their notes and their code and, eventually, someone’s filter.

This page is about closing that gap. It describes the method, says plainly who did what, works one real example all the way through, and reports what the method has cost and caught on this site.

The distinction the whole method rests on

There are two ways to look at a number on a page.

Assessing it means reading the surrounding argument and judging whether the number is plausible. This is what review does, and it is nearly useless against arithmetic. A wrong number that arrived through a plausible derivation looks exactly like a right one. Reviewers, human and machine alike, agree with confident prose at a rate that should worry anyone relying on them.

Computing it means reproducing the number from first principles, without looking at the answer, and then comparing. This catches what assessment cannot, because it never consults the page’s own reasoning. The claim either falls out of the mathematics or it does not.

Everything below follows from preferring the second. It is slower, it is duller, and it is the only one that works.

Augmented engineering

Computing rather than assessing is dull, mechanical work, and there is a great deal of it: every formula, every budget table, every worked answer, on every page. That is the half worth handing to a machine, and this site hands it over. Cards face up: the sweeps described below are run by models, the prose was drafted with them, and much of the Python was written alongside them.

What is not handed over is the part that decides whether any of it is worth anything – which claim matters, what counts as proof, and which of a model’s objections is real rather than confident noise. The apparatus is built to distrust its own output: every finding goes to a second model told to refute it, and 236 of 556 did not survive that step. What survives is adjudicated by a person. A process that accepts whatever a model says has an author who is not the person whose name is on it.

Call it augmented engineering: the engineer keeps the questions, the judgement and the responsibility; the machine takes the elaboration. The elaboration is not the small half – it is the sweeping, the recomputing, the drafting, the third opinion at two in the morning, and there is far more of it than there is of the other. It is simply not the half that decides anything.

Every number here is the author’s, including the ones a model produced and the ones a model caught. The four failure modes further down are what the arrangement still misses, and they are on this page because it missed them.

A worked example

The zero-phase filtering page needs a small piece of design arithmetic. Filtering forward and then backward applies the magnitude response twice, so the cutoff moves: a filter designed for $-3$ dB at $f_c$ lands well inside $f_c$ after the second pass. To get $-3$ dB at $f_c$ from the round trip you design the single pass for a slightly higher cutoff, and the page gave the correction factor as

\[f_\text{design} = f_c \left(2^{1/2N}-1\right)^{-1/2N}\]

It is a reasonable-looking formula. It has the right shape, the right variables, and it reduces to something sensible. It is also wrong for every order above the first.

Here is the derivation done independently. A Butterworth magnitude response of order $N$ is

\[|H(\omega)|^2 = \frac{1}{1+(\omega/\omega_c)^{2N}}\]

Forward-backward filtering applies $|H|^2$, so the round-trip power response is $|H|^4$. The $-3$ dB condition on the round trip is therefore

\[\left(\frac{1}{1+(\omega/\omega_c)^{2N}}\right)^{2} = \frac{1}{2} \quad\Longrightarrow\quad 1+(\omega/\omega_c)^{2N} = \sqrt{2} \quad\Longrightarrow\quad (\omega/\omega_c)^{2N} = \sqrt{2}-1\]

which gives the correction factor

\[f_\text{design} = f_c\left(\sqrt{2}-1\right)^{-1/2N}\]

The two formulas differ in one place: the base is $\sqrt{2}-1$, a constant, not $2^{1/2N}-1$, which depends on the order. They agree at $N=1$ and diverge from there. The error is that the one-half was applied at the wrong power: to $|H|^2$ rather than to the round-trip $|H|^4$.

Note what the check did not require. No judgement about whether the formula looked right, no appeal to a source, no discussion. The derivation runs, and the two expressions either match or they do not.

Now measure the consequence, which is the part that decides whether a defect matters:

import numpy as np
from scipy import signal

fs, N, fc = 1000.0, 4, 10.0          # 4th-order, 10 Hz target, 1 kHz sampling

def round_trip_minus3db(design_fc):
    """Where the forward-backward -3 dB point actually lands."""
    b, a = signal.butter(N, design_fc / (fs / 2))
    w, h = signal.freqz(b, a, worN=200_000, fs=fs)
    db = 20 * np.log10(np.abs(h) ** 2)        # forward-backward applies |H| twice
    return w[np.argmin(np.abs(db + 3.0))]

as_published = fc * (2 ** (1 / (2 * N)) - 1) ** (-1 / (2 * N))
as_derived   = fc * (2 ** 0.5 - 1) ** (-1 / (2 * N))

print(f"design cutoff   as published {as_published:6.3f} Hz    as derived {as_derived:6.3f} Hz")
print(f"-3 dB lands at  as published {round_trip_minus3db(as_published):6.3f} Hz"
      f"    as derived {round_trip_minus3db(as_derived):6.3f} Hz")
print(f"target was      {fc:.3f} Hz")

assert abs(round_trip_minus3db(as_derived) - fc) < 0.02, "derived formula must hit the target"
assert round_trip_minus3db(as_published) > 1.15 * fc, "published formula must miss it badly"

# Pin the two figures the prose below quotes, so neither can drift while the build stays green.
# Without these the paragraph's "12.09 Hz" and "21%" were held only by the > 1.15 * fc above,
# which tolerates anything down to 11.5 Hz.
assert abs(round_trip_minus3db(as_published) - 12.09) < 0.01, "prose quotes 12.09 Hz"
assert abs(100 * (round_trip_minus3db(as_published) - fc) / fc - 20.9) < 0.1, "prose quotes 21%"

design cutoff   as published 13.503 Hz    as derived 11.165 Hz
-3 dB lands at  as published 12.090 Hz    as derived  9.995 Hz
target was      10.000 Hz

The corrected formula lands on 10 Hz. The published one lands at 12.09 Hz, a 21% error in the quantity the reader asked for, at an order (fourth) that anyone filtering physiological or audio data reaches within a week.

The same check condemned a second sentence on that page as collateral. The text described the correction as “small, a few percent for $N \geq 4$”. Having a correct formula makes that measurable:

for order in (2, 4, 8, 16):
    factor = (2 ** 0.5 - 1) ** (-1 / (2 * order))
    print(f"N = {order:2d}   correction = {100 * (factor - 1):5.2f}%")

assert abs(100 * ((2 ** 0.5 - 1) ** (-1 / 8) - 1) - 11.65) < 0.01

N =  2   correction = 24.65%
N =  4   correction = 11.65%
N =  8   correction =  5.66%
N = 16   correction =  2.79%

At $N=4$ the correction is 11.65%, not a few percent. The prose was calibrated against an intuition, and the intuition was for a different order.

This is what the record looks like

Every check like the one above is written down: the claim as published, the value recomputed, the measured consequence, and the fix. Every number-bearing topic and Basics page has such a record; the exercise sets are covered by one collective sweep rather than one record each. Those records are kept off the site because they are working notes rather than lessons, and the repository is private at the time of writing, so this is at present a description of the record rather than a link to it. That is a real limit on what this page can ask you to take on trust, and it is why the example above is worked in full here instead of being pointed at.

Five channels, covering different things

No single mechanism reaches all of it, so this site runs five over the published content, each aimed at what the others structurally miss. Two more sit outside CI and are described further down: the browser checks, and a runner that re-measures the project’s own claims about itself.

pytest over the modules. The ordinary one. Every topic that ships a .py is tested – four topics ship none – and CI blocks the deploy on a failure. It reaches implementations, and between it and the structural channel below (which runs under the same pytest) nothing else.

Assertions inside the pages. Code blocks in a .qmd execute when the site is built, so an assert in a page runs on every build, in CI, before anything ships. This is how a figure’s caption gets tied to the figure: if a caption claims an n-fold sharpening, an assertion on the same page measures the sharpening and the build fails when the two part company. The two cells above are doing exactly this; every number the prose around them quotes is pinned by an assert, so none of them can drift without turning the site red. That is a property of those two cells, not of the site: an assertion reaches only what an author wrote one for.

Recomputation of prose numbers. The channel this page is about, and the only one that reaches captions, budget tables and worked solutions systematically. An assertion can reach any of them, as the cells above do, but only where someone thought to add one; this channel sweeps them whether or not anyone thought of it. It is run under a standing instruction to compute rather than assess, and its output is the per-page record described above.

Adversarial multi-model review. Several models read the same page through different lenses (mathematics, embedded C, pedagogy, fabricated citations), and every finding any of them raises is then handed to a different model told to refute it. A large fraction of findings do not survive that step: 236 of 556 were rejected, across the 27 pages whose record states both counts. That is the point – a review pass that trusts its own output is a review pass that generates work.

Structural checks over the site as an object. A separate suite, most of which tests no mathematics at all. It asks whether the Basics index links every chapter that exists on disk and whether the landing page stays a signpost rather than half an inventory, whether every answer the checker recognises is really behind a collapsed panel rather than merely near one, and whether the checks the site relies on are installed where they will actually run. These are mostly not numerical claims and no amount of recomputation would reach them; they are the site’s promises about itself, and they broke often enough to be worth a test each. Most of these checks were written after the corresponding promise had already been found broken, which is the fourth failure mode below; two were written alongside the thing they check.

What it has cost, and what it caught

Two full sweeps have been run across every page that carried equations or hand-authored numbers at the time. The second, on 2026-06-12, covered 44 pages; the site has 98 .qmd files today, so more than half of what is now published postdates both sweeps. Those pages were verified individually on arrival instead, one record each – a different mechanism, and one whose coverage nothing re-checks in aggregate.

	Checks	Defects confirmed
First sweep, topics and basics (excluding its pilot)	390	19
Second sweep, adding the exercise sets	1256	33

Roughly 1600 checks, 52 numbers that did not reproduce. The first sweep was preceded by a 28-check pilot on one page, which found three more and is excluded from both the row and the total, so those are the two sweeps proper rather than everything ever checked. That is a rate of about one defect for every thirty numbers on a site written carefully by someone who knows the material. The two sweeps overlap – the second re-checked the 29 pages the first had covered – so 1600 counts checks performed, not distinct numbers examined.

The distribution is more useful than the total. Sixteen of the second sweep’s thirty-three defects were in the exercise sets, which is where a reader is least able to defend themselves: a worked solution is consulted precisely when the reader does not yet know the answer. It is also the surface with the least test coverage, since pytest reaches modules and a published solution is prose. The most exposed material was the material nothing was watching.

Four failure modes that keep coming back

Across those sweeps the same shapes recur, and none of them is the kind of mistake review catches, because in every case the reasoning is sound and the artifact is still wrong.

A bound quoted for the wrong parameterisation. A Cramér-Rao bound is a statement about a specific parameter in specific units. Quoted from memory into a page that parameterises differently, it is off by a constant, and the constant is invisible because the formula is otherwise correct. On this site the same factor of two was introduced twice in one session, in the frequency bound and then in the phase bound, and was caught only by building the Fisher information matrix numerically and dividing.

A statistic calibrated on an ingredient rather than as computed. A threshold is designed against the distribution of something adjacent to what the code actually evaluates: the marginal of one candidate rather than the maximum over a window, the raw sample count rather than the effective degrees of freedom left after a filter. Three occurrences here. In all three the ingredient’s closed form was exactly right, and the deployed threshold was still wrong, by factors of 3.6, of 2, and once of thirteen orders of magnitude. The deeper the threshold, the more of the answer is decided by a tail nobody checked.

A cross-check less accurate than the thing it grades. An independent calculation brought in to referee a closed form needs its own accuracy established first. A numerical method here, introduced to validate a run-length formula, was itself unconverged and graded a formula accurate to 0.02% as being 0.2% wrong. The referee was the error.

A check that was never running. The fourth is the one that took longest to see, because it does not produce a wrong number. It produces no signal at all, and an absence looks exactly like a pass. Six instances here, none of them caught by a failing run, because there was no failing run to have:

The CI workflows invoked pytest topics/. A path argument silently overrides the configured test paths, so the entire basics/ directory sat outside the deploy gate for six weeks while every run reported green.
A raised timeout on the deploy step was clamped back to the maximum with only a warning. The configuration read as applied, was inspected more than once, and changed nothing.
One of the automated review passes was installed in a layout that registers nothing. No error said so; it simply never loaded, for three days.
A second was mis-placed the same way and had never loaded once.
Every claim this project makes about its own current state carries a command that is supposed to check it. Twenty-two of those commands were written in a syntax the runner could not execute, so in the twelve weeks since the practice was adopted not one of them had ever checked anything – twenty errored on the first run that reached them, and two were never reached at all. Every count they guarded happened to be correct, which is why twelve weeks of review never noticed: the claims were true and only the checking was imaginary.
One of those commands asked a dependency for its own latest release and reported the answer as this project’s pinned version. It could not disagree with itself, so it read as a healthy check while being incapable of detecting the drift it existed to detect.

The common shape is that the healthy and the broken states emit identical output. A test suite that runs 40 files and one that runs 60 both print dots. This is a known class rather than local bad luck, and the sharpest statement of the test for it is: can I tell this check working apart from this check being absent, by looking at its output? If not, make it print a count. Its corollary is to enumerate the mechanisms by which something can leave the count – never matched, never collected, overridden, silently clamped – rather than the intents, because intents are unbounded and mechanisms can be read off the configuration. Each of the six above was found by hand, one at a time, long after it started; that is what the corollary is for.

It is worth being plain that this failure mode attacks verification itself. The other three corrupt a number on a page. This one corrupts the apparatus that would have caught it, and it does so quietly, which makes every other claim on this page conditional on it.

What is still not verified

The honest inventory, because a page about verification that only lists successes is advertising.

Performance numbers are budgets. Where an embedded page quotes a per-sample cost, that figure is nearly always derived from instruction counts and the core’s timing rather than read off a timer on a running board, and a budget that assumes single-cycle multiply-accumulate and no memory stalls is optimistic by a margin that depends on the part. The pages doing this mark it at the figure; change detection also carries the bench protocol that would turn its budgets into measurements. Nothing in CI can close that gap, because the build machine has no board attached.

Three embedded pages go further and carry performance figures with no tested Python model behind them at all. By the audit’s own count they rank first, fourth and seventh for hand-authored numbers, which makes them the densest with nothing executable underneath – the sort of thing an audit tells you and intuition does not.

Rendering is verified by hand, and only where someone thought to look. Diagrams were checked for years by rasterising them locally, and the local renderer honoured an SVG attribute that the browser the site was read in did not, so all 22 index glyphs across six figures were flat on the live site while every check passed. Which browsers honour that attribute was never systematically tested, and an earlier screenshot run had shown the same labels rendering correctly, so the attribution is to one reading of one deployed page and no further. The verification loop agreed with itself and disagreed with reality. There are now browser checks that drive a real headless browser and screenshot the result, which is how a later diagram was caught rendering $1 - z^{M}$ where it should have read $1 - z^{-M}$ – the wrong transfer function – with its source file, its deployed image and the same index elsewhere in the same figure all correct. The same harness caught an interactive widget that had gone blank while passing every arithmetic test, because a pole on the unit circle sent one sample to $+3000$ dB and the auto-scaling followed it. But these run on a developer’s machine, on demand, and deliberately not in CI, which installs no browser. Nothing automated compares the deployed page to the intended one; a person has to look at the screenshot.

Worked solutions remain the thinnest surface. They were swept twice and corrected, but almost nothing recomputes them on every build the way an assertion in a page is run: two asserts inside solution panels on one of the nine exercise pages, and nothing at all on the other eight. A wrong number reintroduced there today would ship. What is checked on every build is their packaging – that each answer sits behind its own collapsed panel – which was added after two pages were found publishing answers in the open. That check reaches the fold and not the arithmetic inside it, and the distinction is exactly the sort that makes a suite feel more protective than it is.

None of this makes the material unreliable. It makes the reliability bounded, and the point of stating the bounds is that a reader can then decide what to trust and what to check themselves, which is in any case the only durable position to hold toward someone else’s numbers.

The same method elsewhere

None of this is DSP-specific, and none of it needs an agent to be useful. The underlying idea is that a claim should carry the command that checks it, which is as true of a paper, a codebase or a forecast as it is of a filter design formula. Three public repositories carry the method in full.

agent-ready-papers is the verification approach this page describes, written up properly: computing rather than assessing, the per-claim record, the citation checklist that keeps a plausible-looking reference from entering a bibliography unread, and the test for a check that is not running. It is aimed at academic and technical writing, where the cost of a number nobody reproduced is highest. This site runs a deliberately light profile of it, skipping the coverage gates and the peer-review apparatus, which are paper-specific.

agent-ready-projects is what stops the checking being reinvented every session: how to structure a project so the work survives, what belongs in durable memory rather than a project file, and the diff-driven review that decides how much scrutiny a change has earned from what it actually touches. The discipline of writing a verify command next to every claim of current state comes from there – including the one whose absence is the fifth entry in the failure-mode list above.

Augur is the same discipline pointed at something that answers back: Dutch electricity prices forecast 48 hours ahead from eighteen-odd data sources, a model that learns continuously, and a live dashboard. A teaching site can be quietly wrong for years. A forecast is graded by reality every morning, which makes it the more honest test of whether any of this survives contact with a system that has to work.

All three are public. This site, at the time of writing, is not, which is why the defect above is worked in full here rather than pointed at.

--- title: "Verification" subtitle: "How to check a DSP page's numbers, and what checking this site actually found" description-meta: "How the numbers on this site are checked, and what checking actually found: the figures that ordinary unit tests never reach." --- Every page here carries numbers that were not produced by the code you can see: a cutoff frequency in a sentence, a cycle count in a budget table, a dB figure in a caption, the answer at the bottom of an exercise. Ordinary unit tests do not reach them. `pytest` runs the modules, and a module that passes its tests says nothing about a paragraph three lines above it claiming the correction is "a few percent". Two of the channels described below do reach prose numbers, and both were built for that reason; neither is what "running the tests" ordinarily means. That gap is worse in teaching material than anywhere else. A reader who is learning cannot tell a typo from a result. They have no prior to violate: if the page says the correction factor is $(2^{1/2N}-1)^{-1/2N}$, that is simply what the correction factor is, and it goes into their notes and their code and, eventually, someone's filter. This page is about closing that gap. It describes the method, says plainly who did what, works one real example all the way through, and reports what the method has cost and caught on this site. <hr> ## The distinction the whole method rests on There are two ways to look at a number on a page. **Assessing** it means reading the surrounding argument and judging whether the number is plausible. This is what review does, and it is nearly useless against arithmetic. A wrong number that arrived through a plausible derivation looks exactly like a right one. Reviewers, human and machine alike, agree with confident prose at a rate that should worry anyone relying on them. **Computing** it means reproducing the number from first principles, without looking at the answer, and then comparing. This catches what assessment cannot, because it never consults the page's own reasoning. The claim either falls out of the mathematics or it does not. Everything below follows from preferring the second. It is slower, it is duller, and it is the only one that works. <hr> ## Augmented engineering Computing rather than assessing is dull, mechanical work, and there is a great deal of it: every formula, every budget table, every worked answer, on every page. That is the half worth handing to a machine, and this site hands it over. Cards face up: the sweeps described below are run by models, the prose was drafted with them, and much of the Python was written alongside them. What is not handed over is the part that decides whether any of it is worth anything -- which claim matters, what counts as proof, and which of a model's objections is real rather than confident noise. The apparatus is built to distrust its own output: every finding goes to a second model told to refute it, and **236 of 556 did not survive that step**. What survives is adjudicated by a person. A process that accepts whatever a model says has an author who is not the person whose name is on it. Call it **augmented engineering**: the engineer keeps the questions, the judgement and the responsibility; the machine takes the elaboration. The elaboration is not the small half -- it is the sweeping, the recomputing, the drafting, the third opinion at two in the morning, and there is far more of it than there is of the other. It is simply not the half that decides anything. Every number here is the author's, including the ones a model produced and the ones a model caught. The four failure modes further down are what the arrangement still misses, and they are on this page because it missed them. <hr> ## A worked example The [zero-phase filtering](basics/zero-phase.qmd) page needs a small piece of design arithmetic. Filtering forward and then backward applies the magnitude response twice, so the cutoff moves: a filter designed for $-3$ dB at $f_c$ lands well inside $f_c$ after the second pass. To get $-3$ dB at $f_c$ from the round trip you design the single pass for a slightly higher cutoff, and the page gave the correction factor as $$f_\text{design} = f_c \left(2^{1/2N}-1\right)^{-1/2N}$$ It is a reasonable-looking formula. It has the right shape, the right variables, and it reduces to something sensible. It is also wrong for every order above the first. Here is the derivation done independently. A Butterworth magnitude response of order $N$ is $$|H(\omega)|^2 = \frac{1}{1+(\omega/\omega_c)^{2N}}$$ Forward-backward filtering applies $|H|^2$, so the round-trip power response is $|H|^4$. The $-3$ dB condition on the round trip is therefore $$\left(\frac{1}{1+(\omega/\omega_c)^{2N}}\right)^{2} = \frac{1}{2} \quad\Longrightarrow\quad 1+(\omega/\omega_c)^{2N} = \sqrt{2} \quad\Longrightarrow\quad (\omega/\omega_c)^{2N} = \sqrt{2}-1$$ which gives the correction factor $$f_\text{design} = f_c\left(\sqrt{2}-1\right)^{-1/2N}$$ The two formulas differ in one place: the base is $\sqrt{2}-1$, a constant, not $2^{1/2N}-1$, which depends on the order. They agree at $N=1$ and diverge from there. The error is that the one-half was applied at the wrong power: to $|H|^2$ rather than to the round-trip $|H|^4$. Note what the check did *not* require. No judgement about whether the formula looked right, no appeal to a source, no discussion. The derivation runs, and the two expressions either match or they do not. Now measure the consequence, which is the part that decides whether a defect matters: ```{python} #| label: zero-phase-correction #| code-fold: false import numpy as np from scipy import signal fs, N, fc = 1000.0, 4, 10.0 # 4th-order, 10 Hz target, 1 kHz sampling def round_trip_minus3db(design_fc): """Where the forward-backward -3 dB point actually lands.""" b, a = signal.butter(N, design_fc / (fs / 2)) w, h = signal.freqz(b, a, worN=200_000, fs=fs) db = 20 * np.log10(np.abs(h) ** 2) # forward-backward applies |H| twice return w[np.argmin(np.abs(db + 3.0))] as_published = fc * (2 ** (1 / (2 * N)) - 1) ** (-1 / (2 * N)) as_derived = fc * (2 ** 0.5 - 1) ** (-1 / (2 * N)) print(f"design cutoff as published {as_published:6.3f} Hz as derived {as_derived:6.3f} Hz") print(f"-3 dB lands at as published {round_trip_minus3db(as_published):6.3f} Hz" f" as derived {round_trip_minus3db(as_derived):6.3f} Hz") print(f"target was {fc:.3f} Hz") assert abs(round_trip_minus3db(as_derived) - fc) < 0.02, "derived formula must hit the target" assert round_trip_minus3db(as_published) > 1.15 * fc, "published formula must miss it badly" # Pin the two figures the prose below quotes, so neither can drift while the build stays green. # Without these the paragraph's "12.09 Hz" and "21%" were held only by the > 1.15 * fc above, # which tolerates anything down to 11.5 Hz. assert abs(round_trip_minus3db(as_published) - 12.09) < 0.01, "prose quotes 12.09 Hz" assert abs(100 * (round_trip_minus3db(as_published) - fc) / fc - 20.9) < 0.1, "prose quotes 21%" ``` The corrected formula lands on 10 Hz. The published one lands at 12.09 Hz, a 21% error in the quantity the reader asked for, at an order (fourth) that anyone filtering physiological or audio data reaches within a week. The same check condemned a second sentence on that page as collateral. The text described the correction as "small, a few percent for $N \geq 4$". Having a correct formula makes that measurable: ```{python} #| label: correction-magnitude #| code-fold: false for order in (2, 4, 8, 16): factor = (2 ** 0.5 - 1) ** (-1 / (2 * order)) print(f"N = {order:2d} correction = {100 * (factor - 1):5.2f}%") assert abs(100 * ((2 ** 0.5 - 1) ** (-1 / 8) - 1) - 11.65) < 0.01 ``` At $N=4$ the correction is 11.65%, not a few percent. The prose was calibrated against an intuition, and the intuition was for a different order. ::: {.callout-note title="This is what the record looks like"} Every check like the one above is written down: the claim as published, the value recomputed, the measured consequence, and the fix. Every number-bearing topic and Basics page has such a record; the exercise sets are covered by one collective sweep rather than one record each. Those records are kept off the site because they are working notes rather than lessons, and the repository is private at the time of writing, so this is at present a description of the record rather than a link to it. That is a real limit on what this page can ask you to take on trust, and it is why the example above is worked in full here instead of being pointed at. ::: <hr> ## Five channels, covering different things No single mechanism reaches all of it, so this site runs five over the published content, each aimed at what the others structurally miss. Two more sit outside CI and are described further down: the browser checks, and a runner that re-measures the project's own claims about itself. **`pytest` over the modules.** The ordinary one. Every topic that ships a `.py` is tested -- four topics ship none -- and CI blocks the deploy on a failure. It reaches implementations, and between it and the structural channel below (which runs under the same `pytest`) nothing else. **Assertions inside the pages.** Code blocks in a `.qmd` execute when the site is built, so an `assert` in a page runs on every build, in CI, before anything ships. This is how a figure's caption gets tied to the figure: if a caption claims an *n*-fold sharpening, an assertion on the same page measures the sharpening and the build fails when the two part company. The two cells above are doing exactly this; every number the prose around them quotes is pinned by an assert, so none of them can drift without turning the site red. That is a property of those two cells, not of the site: an assertion reaches only what an author wrote one for. **Recomputation of prose numbers.** The channel this page is about, and the only one that reaches captions, budget tables and worked solutions *systematically*. An assertion can reach any of them, as the cells above do, but only where someone thought to add one; this channel sweeps them whether or not anyone thought of it. It is run under a standing instruction to compute rather than assess, and its output is the per-page record described above. **Adversarial multi-model review.** Several models read the same page through different lenses (mathematics, embedded C, pedagogy, fabricated citations), and every finding any of them raises is then handed to a different model told to *refute* it. A large fraction of findings do not survive that step: 236 of 556 were rejected, across the 27 pages whose record states both counts. That is the point -- a review pass that trusts its own output is a review pass that generates work. **Structural checks over the site as an object.** A separate suite, most of which tests no mathematics at all. It asks whether the Basics index links every chapter that exists on disk and whether the landing page stays a signpost rather than half an inventory, whether every answer the checker recognises is really behind a collapsed panel rather than merely near one, and whether the checks the site relies on are installed where they will actually run. These are mostly not numerical claims and no amount of recomputation would reach them; they are the site's promises about itself, and they broke often enough to be worth a test each. Most of these checks were written after the corresponding promise had already been found broken, which is the fourth failure mode below; two were written alongside the thing they check. <hr> ## What it has cost, and what it caught Two full sweeps have been run across every page that carried equations or hand-authored numbers *at the time*. The second, on 2026-06-12, covered 44 pages; the site has 98 `.qmd` files today, so more than half of what is now published postdates both sweeps. Those pages were verified individually on arrival instead, one record each -- a different mechanism, and one whose coverage nothing re-checks in aggregate. | | Checks | Defects confirmed | |---|---:|---:| | First sweep, topics and basics (excluding its pilot) | 390 | 19 | | Second sweep, adding the exercise sets | 1256 | 33 | Roughly 1600 checks, 52 numbers that did not reproduce. The first sweep was preceded by a 28-check pilot on one page, which found three more and is excluded from both the row and the total, so those are the two sweeps proper rather than everything ever checked. That is a rate of about one defect for every thirty numbers on a site written carefully by someone who knows the material. The two sweeps overlap -- the second re-checked the 29 pages the first had covered -- so 1600 counts checks performed, not distinct numbers examined. The distribution is more useful than the total. **Sixteen of the second sweep's thirty-three defects were in the exercise sets**, which is where a reader is least able to defend themselves: a worked solution is consulted precisely when the reader does not yet know the answer. It is also the surface with the least test coverage, since `pytest` reaches modules and a published solution is prose. The most exposed material was the material nothing was watching. <hr> ## Four failure modes that keep coming back Across those sweeps the same shapes recur, and none of them is the kind of mistake review catches, because in every case the reasoning is sound and the artifact is still wrong. **A bound quoted for the wrong parameterisation.** A Cramér-Rao bound is a statement about a specific parameter in specific units. Quoted from memory into a page that parameterises differently, it is off by a constant, and the constant is invisible because the formula is otherwise correct. On this site the same factor of two was introduced twice in one session, in the frequency bound and then in the phase bound, and was caught only by building the Fisher information matrix numerically and dividing. **A statistic calibrated on an ingredient rather than as computed.** A threshold is designed against the distribution of something adjacent to what the code actually evaluates: the marginal of one candidate rather than the maximum over a window, the raw sample count rather than the effective degrees of freedom left after a filter. Three occurrences here. In all three the ingredient's closed form was exactly right, and the deployed threshold was still wrong, by factors of 3.6, of 2, and once of thirteen orders of magnitude. The deeper the threshold, the more of the answer is decided by a tail nobody checked. **A cross-check less accurate than the thing it grades.** An independent calculation brought in to referee a closed form needs its own accuracy established first. A numerical method here, introduced to validate a run-length formula, was itself unconverged and graded a formula accurate to 0.02% as being 0.2% wrong. The referee was the error. **A check that was never running.** The fourth is the one that took longest to see, because it does not produce a wrong number. It produces no signal at all, and an absence looks exactly like a pass. Six instances here, none of them caught by a failing run, because there was no failing run to have: - The CI workflows invoked `pytest topics/`. A path argument silently overrides the configured test paths, so the entire `basics/` directory sat outside the deploy gate for six weeks while every run reported green. - A raised timeout on the deploy step was clamped back to the maximum with only a warning. The configuration read as applied, was inspected more than once, and changed nothing. - One of the automated review passes was installed in a layout that registers nothing. No error said so; it simply never loaded, for three days. - A second was mis-placed the same way and had never loaded once. - Every claim this project makes about its own current state carries a command that is supposed to check it. Twenty-two of those commands were written in a syntax the runner could not execute, so in the twelve weeks since the practice was adopted not one of them had ever checked anything -- twenty errored on the first run that reached them, and two were never reached at all. Every count they guarded happened to be correct, which is why twelve weeks of review never noticed: the claims were true and only the checking was imaginary. - One of those commands asked a dependency for its own latest release and reported the answer as this project's pinned version. It could not disagree with itself, so it read as a healthy check while being incapable of detecting the drift it existed to detect. The common shape is that the healthy and the broken states emit identical output. A test suite that runs 40 files and one that runs 60 both print dots. This is a known class rather than local bad luck, and the sharpest statement of the test for it is: *can I tell this check working apart from this check being absent, by looking at its output? If not, make it print a count.* Its corollary is to enumerate the **mechanisms** by which something can leave the count -- never matched, never collected, overridden, silently clamped -- rather than the intents, because intents are unbounded and mechanisms can be read off the configuration. Each of the six above was found by hand, one at a time, long after it started; that is what the corollary is for. It is worth being plain that this failure mode attacks verification itself. The other three corrupt a number on a page. This one corrupts the apparatus that would have caught it, and it does so quietly, which makes every other claim on this page conditional on it. <hr> ## What is still not verified The honest inventory, because a page about verification that only lists successes is advertising. **Performance numbers are budgets.** Where an embedded page quotes a per-sample cost, that figure is nearly always derived from instruction counts and the core's timing rather than read off a timer on a running board, and a budget that assumes single-cycle multiply-accumulate and no memory stalls is optimistic by a margin that depends on the part. The pages doing this mark it at the figure; [change detection](topics/change-detection/embedded.qmd) also carries the bench protocol that would turn its budgets into measurements. Nothing in CI can close that gap, because the build machine has no board attached. Three embedded pages go further and carry performance figures with no tested Python model behind them at all. By the audit's own count they rank first, fourth and seventh for hand-authored numbers, which makes them the densest with nothing executable underneath -- the sort of thing an audit tells you and intuition does not. **Rendering is verified by hand, and only where someone thought to look.** Diagrams were checked for years by rasterising them locally, and the local renderer honoured an SVG attribute that the browser the site was read in did not, so all 22 index glyphs across six figures were flat on the live site while every check passed. Which browsers honour that attribute was never systematically tested, and an earlier screenshot run had shown the same labels rendering correctly, so the attribution is to one reading of one deployed page and no further. The verification loop agreed with itself and disagreed with reality. There are now browser checks that drive a real headless browser and screenshot the result, which is how a later diagram was caught rendering $1 - z^{M}$ where it should have read $1 - z^{-M}$ -- the wrong transfer function -- with its source file, its deployed image and the same index elsewhere in the same figure all correct. The same harness caught an interactive widget that had gone blank while passing every arithmetic test, because a pole on the unit circle sent one sample to $+3000$ dB and the auto-scaling followed it. But these run on a developer's machine, on demand, and deliberately not in CI, which installs no browser. Nothing automated compares the deployed page to the intended one; a person has to look at the screenshot. **Worked solutions remain the thinnest surface.** They were swept twice and corrected, but almost nothing recomputes them on every build the way an assertion in a page is run: two asserts inside solution panels on one of the nine exercise pages, and nothing at all on the other eight. A wrong number reintroduced there today would ship. What *is* checked on every build is their packaging -- that each answer sits behind its own collapsed panel -- which was added after two pages were found publishing answers in the open. That check reaches the fold and not the arithmetic inside it, and the distinction is exactly the sort that makes a suite feel more protective than it is. None of this makes the material unreliable. It makes the reliability *bounded*, and the point of stating the bounds is that a reader can then decide what to trust and what to check themselves, which is in any case the only durable position to hold toward someone else's numbers. <hr> ## The same method elsewhere None of this is DSP-specific, and none of it needs an agent to be useful. The underlying idea is that a claim should carry the command that checks it, which is as true of a paper, a codebase or a forecast as it is of a filter design formula. Three public repositories carry the method in full. **[agent-ready-papers](https://github.com/ducroq/agent-ready-papers)** is the verification approach this page describes, written up properly: computing rather than assessing, the per-claim record, the citation checklist that keeps a plausible-looking reference from entering a bibliography unread, and the test for a check that is not running. It is aimed at academic and technical writing, where the cost of a number nobody reproduced is highest. This site runs a deliberately light profile of it, skipping the coverage gates and the peer-review apparatus, which are paper-specific. **[agent-ready-projects](https://github.com/ducroq/agent-ready-projects)** is what stops the checking being reinvented every session: how to structure a project so the work survives, what belongs in durable memory rather than a project file, and the diff-driven review that decides how much scrutiny a change has earned from what it actually touches. The discipline of writing a verify command next to every claim of current state comes from there -- including the one whose absence is the fifth entry in the failure-mode list above. **[Augur](https://github.com/ducroq/augur)** is the same discipline pointed at something that answers back: Dutch electricity prices forecast 48 hours ahead from eighteen-odd data sources, a model that learns continuously, and a [live dashboard](https://energy.jeroenveen.nl/). A teaching site can be quietly wrong for years. A forecast is graded by reality every morning, which makes it the more honest test of whether any of this survives contact with a system that has to work. All three are public. This site, at the time of writing, is not, which is why the defect above is worked in full here rather than pointed at.