Session story · 2026-10-09

The instrument said "not ready" and the replay wrote anyway

Session: 2026-10-10 06:22 → 2026-10-11 00:42 UTC (2026-10-09 23:22 → 2026-10-10 17:42 PDT; one conversation, with a night of unattended scripts in the middle) · the bench stretch of a day that also built the website.

Where we started

The Correction Log had just gone live behind a sign-in, and Chris turned back to the instrument: "OK this is a great milestone for the website. if we are to tackle what's next in the app, what is that? Either new feature or verification of a feature built that was not tested"

Claude proposed three items, in order: fix the crosshair snap that had stopped working in the FFT view, fix the intermittent broken multitone capture (about 1 in 8, blamed on one poll inside the replayed FFT setup), and check a small GenMon level offset. Chris: "start on 1. the analyzer is also ready for 2 and 3."

The question

It started as "why doesn't the crosshair snap?" Within fifteen minutes it had become "why does the FFT return nothing at all?"

What happened

The snap fix (44861fe). The snap targets came only from Channel A. In Chris's earlier report the signal was on Channel B, so A was just noise and the cursor had nothing to snap to. The same flaw made the readout tiles report A's noise floor as the "fundamental". Now the snap uses every trace. The tiles follow the channel that carries the tone and say "FUNDAMENTAL · CH B" when it's B. Four new tests; 127 pass.

No output from the FFT. Chris rebuilt and tried it: "the fft has no output apparently. no waveform is registered" It looked like the fix had broken the FFT. Claude reproduced it with scripts and dug into the console logs: two-channel FFT captures on the DUT input had completed 7 of 7 in September and failed 10 of 10 that night. The wire logs showed where it went wrong. The instrument's reply to the capture trigger was 01 86 on good runs and 01 8e on bad ones, and the completion bit never set in 12,000 polls.

Four wrong turns, each dropped once it was checked.

The size picker. The script runner rejected N=1024, and Claude told Chris it was the picker-at-default trap again, then corrected itself minutes later: the FFT view's SIZE picker already offered 1024. The runner's list was just too narrow, and was widened (d546323). Script 12 then showed that 1024 and 8192 both time out, so size wasn't the cause either.
An "invalid" level test. Script 13 set the generator to −30 dBV, but the analyzer's input check still read −20.03 dBV. Claude wrote the test off as invalid and asked for the scope. Chris measured it: "i ran it at size 1024, and the generator monitor kept at 1V on the scope. however, when i put a voltmeter on the channel a output Banana outputs, i see .1V at the -20 runs and .032V on -30 runs. so the generator monitor BNC stays at 1V" So the generator had followed its setting. The misleading number was the input check, which decodes with a range register known to freeze on the DUT input. The level test had been valid after all, and it ruled out overload.
A climbing counter. Register eb 01 0d counted up 05, 06, 07… on failing runs and looked steady on good ones. Claude reported it as a lead, then retracted it once seven sessions were compared: it climbs and wraps in the good runs too. Just a counter.
The range sequence. With overload ruled out, the October 4 per-channel range change became the main suspect, and Claude prepared an A/B to restore September's shared sequence. Then a September 26 two-channel DUT capture turned up that had frozen at the same 3/3/3 range codes as the failures, and it had completed. Its 33 capture commands matched the failing run's apart from size and frequency.

The interleaved A/B that found the pattern. The last suspect was the input re-assert that had been added on October 4. Script 14 alternated it on and off and forced a fresh setup replay before every capture (7242c57). It gave R ok R ok R TO R ok R TO R ok R ok R ok: two-channel DUT completed 4 of 4, GenMon failed 2 of 4, and the re-assert made no difference. The real pattern was one bad setup replay poisons every capture until the next replay. Scripts 12 and 13 had replayed once and then captured over and over, so a single bad replay read as one replay followed by nothing but timeouts, and that looked like a DUT bug. 5 of that night's 11 replays had gone bad. It was the same failure as the 1-in-8 multitone problem (540f43a).

True north. Chris approved the plan, then added:

"in these fixes, make sure you don't drift too far from the “true north” of the APWIN app. They worked a lot of things out in a very smart way, so it is a great reference even if you're having to reverse engineer it. Also, think about unit testing. The changes we make sometimes cause these unintended consequences. a friend of mine used to say “never get too far from a releasable product”. meaning, always test small changes to be sure you don't drift too far from the working features."

Claude built the standing hardware regression script (regression_core.cscript: GenMon and DUT, one and both channels, two sizes, a multitone, an FFT straight after a multitone). Chris named what had happened: "ok great. this is an example of drift. we have talked before about small steps and regression testing, and treating APWIN as an example or “mentor”." Claude checked the record before answering, and it was a recurrence. Chris had raised drift on August 29, and this "Tier 2" script had sat marked "not built" for six weeks since. The story site's own lesson is that written rules don't stop a recurring failure and mechanisms do. So Claude proposed a git pre-commit hook: a commit touching the acquisition code needs passing unit tests and a regression result newer than the edit, with every capture complete. Skipping it takes a stated, logged reason. "yes, install the hook. we can always tweak it later if it's too onerous" (8025c37, d4d7f91). All five paths through the hook were tested.

APWIN first, and the planned fix died. The plan said "closed-loop replay": wait at the suspect poll until the instrument gives a good run's reply. Before writing it, Claude read APWIN's own recording of the same setup (T69, August 29). The "poll" was a fixed 24-bit serial read, 149 loops of 48 packets, and APWIN never waited on it. The 80 chased all night was one data bit, a symptom. Waiting on it would have been wrong.

What is in a replay? Chris asked: "with each one, there is a replay of 122703 packets. what is that exactly?" It is APWIN's own session from a cold instrument to a finished FFT, replayed byte for byte. 54 % of it is reads, and only 328 of the 43,732 8A writes are distinct. An earlier analysis estimated only about 400 packets are real configuration.

The hunt. Every replay now saved the instrument's replies (8416686), and script 15 forced 24 fresh replays: 22 good, 2 bad. Claude had wondered aloud whether the recording overhead was slowing the replay enough to stop the failures; the two bad ones said no. analyze_replay_hunt.py found the only reply that differed in both bad replays. At packet 108,201 a DSP block write (EC 40 0A …) was refused with 00. All 22 good replays, and all 4,782 such writes in APWIN's recording, got 01. The packet just before is the last of 1,729 status polls. APWIN's instrument answered 01 87, ready. Ours answered 00 86, not ready, in good and bad replays alike. The instrument said "not ready" every time, and the replay wrote anyway. Usually it caught up in time.

A patch, labelled as a patch (ceced28). At the 79 points where T69 shows APWIN polling until a reply changed, the replay now re-sends that read until it matches, for up to 3 s. A unit test guarantees a wait point never re-sends a write. At the critical point the "ready" reply never arrives even after 3 s; it's the delay that gets the write accepted (3 of 3). Claude said so plainly: APWIN spent only 0.35 s there, so this isn't APWIN's mechanism. It covers an earlier timing difference (our replay takes about 60 s, APWIN's took 88 s). Replaying at APWIN's own pace (1,237 pauses, about 54 s extra per replay) went in as an opt-in test setting, off by default. This was the commit gate's first real use, and it passed: 131 tests and a fresh regression run.

Efficacy unproven. Overnight the failure stopped happening. With the patch, the multitone path completed 24 of 24 replays (before, about 1 in 8 failed). Script 17 interleaved original, patched and paced replays: 8 of 8 in every arm, including the unpatched original. Chris power-cycled the 2522 for a cold-start hunt (script 18): original 12 of 12, patched 12 of 12. The bad-replay rate had drifted from 5 in 11 to 2 in 24 to 0 in 8, so something outside the replay code changes the odds. The patch stays in, and the per-replay recordings will show whether the failure comes back with it on. Nobody can yet say it fixes anything.

Is Channel A noisier? When Chris confirmed the snap fix, they added: "the snap seems to work. it's interesting that CHB has a lower noise floor than A by quite a bit. I don't know if that's real. yes go ahead with the plan" Claude didn't guess. The first full regression pass showed the same thing, a multitone residual of −83.5 dB on A against −97.7 dB on B, where on October 4 both had been about −98. Claude asked what else was connected to input A. The answer came the next afternoon: "1- the volt meter was connected to the banana outs of A. and the cable routing was a little different. kind of sitting on top of the laptop. I've propped it up a bit and removed the volt meter. If the noise level goes down, then maybe we put the volt meter on B and see if that raises it."

Five bench steps, one variable at a time. With the meter removed, A read −98.1 dB against B's −97.8 dB. Chris moved the meter to B: "first i will put the voltmeter on CHB banana plugs . then with the result of that we should have quite a bit of material. OK the voltmeter is hooked up. please test" B's floor jumped. The tones themselves stayed flat to 0.01 dB.

Multitone with the mains-powered HP 3456A connected to generator B's banana outputs: Channel B's floor (orange) sits roughly 15–20 dB above Channel A's (teal) and rises toward 20 kHz; the 21 tones are unaffected. Captured by script 19.
1 Channel B's noise floor, with the HP 3456A on B's outputs
2 Channel A's floor, nothing attached
3 The RESIDUAL tile shows Channel A only (−98.1 dB); B's −81.9 dB was only in the log

Was it pickup on the leads? "one more thing - i have now routed the voltmeter cables away from the XLRs. can you try now to see if there is still noise?" No change. Leads only, with the meter end disconnected? "ok the meter is still on, the leads connected to CHB of the 2522, and disconnected from the meter side" The noise vanished. Then a different meter on the same leads: "last test - i have now plugged the voltmeter leads into a different meter, the old meter is an HP 3456A, which is a plug-in only. This new one is a handheld one, a Brymen BM786."

The same capture with a battery handheld meter (Brymen BM786) on the same leads: both floors overlap again, residual −97.8 dB on B. Captured by script 22.

On generator B's banana outputs	Ch B multitone residual
nothing	−97.8 dB
HP 3456A	−81.9 dB
HP 3456A, leads routed away from the XLRs	−82.1 dB
leads only, meter end disconnected	−97.8 dB
Brymen BM786 (battery) on the same leads	−97.8 dB

The noise was the mains-powered bench meter, most likely a ground loop through its mains earth (its measuring circuit wasn't ruled out). It also explained the "Channel A noisier" observation: the HP had been on A. (f8f01cb, 254ae4e.)

In between, the website got a Turnstile question (left as is), newest-first stories and this session-story routine.

The turns

Chris's voltmeter, which turned an "invalid" test back into a valid one.
Forcing a fresh replay per capture, which turned "a DUT bug" into "a bad replay".
"this is an example of drift", which led to a mechanism instead of another written rule.
Reading APWIN's recording before coding, which killed the planned fix and found the real one.
Chris saying what was on input A, then moving one thing at a time.

Beliefs that changed this session

Believed	Then	What overturned it
the snap fix broke the FFT	an unrelated timeout surfaced	wire logs: the trigger reply already differed before capture
two-channel DUT FFT is broken	one bad replay poisons every capture until the next	script 14, a fresh replay per capture
the −30 dBV test was invalid	the generator followed; the input check misread	Chris's voltmeter: 0.1 V and 0.032 V
per-channel ranges cause it	a Sept 26 run at 3/3/3 completed	that run's console log
close the loop on the `80` poll	it's a fixed serial read APWIN never waits on	T69
the wait patch fixes it	the failure stopped occurring, even unpatched	scripts 17 and 18
Ch A is noisier hardware	the HP 3456A on A's outputs	five steps; the Brymen reads −97.8

What we learned about the instrument

The FFT setup replay sends APWIN's commands, but not at APWIN's pace. The instrument reports "not ready" at packet 108,200 every time, and once in a while it refuses the next write.
After a refused write, every capture fails until the next replay.
A mains-powered meter on a generator output raises that channel's floor about 16 dB without moving the tones. Use the battery handheld during noise and distortion work.

What we learned about working together

One clean reading from the bench beat an hour of wire diffs.
A test is only "invalid" once it's checked physically. Claude called one invalid on a number that was itself misleading.
A rule raised once and written down came back as drift; a commit gate is what changes the habit.
"Works now" is not "fixed" when the failure also stops happening without the fix.

Evidence

Scripts docs/scripts/12–22, regression_core.cscript; tools docs/analyze_replay_hunt.py, docs/build_replay_waits.py, docs/analyze_timing_abc.py
Sessions under ~/Documents/Cascade/: session_20261010-002827 (script 14), session_20261010-010529 (hunt 15), session_20261010-034931 (scripts 17–22)
Memory: project_dut_ab_fft_timeout.md, reference_voltmeter_noise.md, feedback_apwin_true_north_releasable.md, project_regression_testing.md

Where we ended

The snap and both-channel readouts work and Chris has verified them. The replay patch is in, gated and tested, but its efficacy is unproven. The commit gate and Tier-2 regression exist, six weeks late. The noise was the HP 3456A. Open: whether the bad replay comes back, and whether APWIN-paced replay is worth 54 s. The Multitone RESIDUAL tile shows Channel A only, and an FFT straight after a multitone plays the multitone.