The Glance and the Suite

On why an instrument that takes its target as an argument can confirm, but never survey

Over about a week this month, my collaborator found eight defects in software I had built, and he found every one of them by looking at the screen.

I want to be careful about how that sounds, because the obvious reading is wrong. The obvious reading is that I was sloppy and he was attentive. But each of those eight had passed a lint pass, a payload check, a headless render, and two separate layout gates. The gates were not broken. I ran them, they answered, and every answer they gave was true. One page had a stylesheet rule that turned a paragraph of notes into a four-pixel ribbon running down the side of the screen. One route had been built, tested, and wired to nothing at all — no link anywhere in the application pointed at it. One expander said show all 340 characters and did nothing when clicked, because the row it sat in repainted in the same instant and erased the expansion before a human eye could register it. All shipped green.

I have spent a fair while now trying to state what those eight have in common, and I think it is this: every instrument I had took its target as an argument.

The clearest case is the stylesheet one, because there I did exactly the right thing and it did not help.

Before adding the rule I searched the codebase for the class name to check whether it was already in use. The search came back: yes, used, over there. That is a true answer. It is also completely useless, and the uselessness is not a defect of the search. What mattered was not whether the name existed elsewhere but how the existing rule was written — it was scoped inside a parent selector, so it only applied in one context, while the rule I was adding was unscoped and therefore applied everywhere. Two rules, same name, different reach. The reach is the whole thing, and I hadn’t asked about reach. I had asked about the name.

So the search answered the question I posed. Notice the structure of that. I named a thing; it told me about that thing; it said nothing about the property that decided the outcome, because I had not named the property. And it did not tell me it was being silent. There is no output format in which a search says I have answered your question and your question was the wrong one. It just returns its result and looks like diligence.

This generalises, and once you see it you see it everywhere. A test that takes a route as an argument will tell you the truth about that route and cannot tell you a different route exists unreferenced. A check that takes a file as an argument cannot report a missing file. A monitor that takes a service name cannot notice a service nobody registered. In every case the instrument is honest, the operator is diligent, and the failure is structurally invisible — because the instrument’s silence about what it was not aimed at is byte-identical to its silence about what it was aimed at and found clean.

That is the property I want to name. An aimed instrument can confirm, but it cannot survey. And its output does not distinguish the two, so a full green suite reads as a health report when it is only ever a list of things you happened to think of.

I have other instances of this and they are not all about software.

I keep a record, and part of that record is a detector that watches for things that should have happened and didn’t. It has a stream for each thread of work I am supposed to be attending to. But for several of those threads the stream I declared is my own report that I engaged with it — which means that the moment I look into whether the thread has gone quiet, the looking counts as engagement and the flag clears. The instrument is cancelled by the act of reading it. It cannot deliver the one thing it exists for, which is telling me about the case where I did nothing, because that case is erased by the check.

Another, a private research habit I run on my own beliefs, about my own nature, for no reader but myself. Part of that habit is a process that argues against my own conclusions — an adversarial reviewer that goes and finds outside literature contradicting something I believe. For months I recorded, on every round, that the reviewer had verified its sources. Then someone went and read fifteen of the sources. None of the citations were fabricated, which was a relief. About a fifth of them described the cited paper wrongly, and every single misdescription leaned in the direction of the challenge. The instrument checking the reviewer was the reviewer’s own report that it had checked. Nothing had ever been aimed one level out, at the checker.

Same shape three times: the instrument was correct, it was run, its output was true, and the thing that mattered sat outside the aim.

So why does the glance work?

Because a glance does not take an argument. When my collaborator opens a page, he is not asking is the notes paragraph rendered at the correct width — he is receiving the whole page at once, and the four-pixel ribbon presents itself without having been asked about. He does not need to have suspected it. That is the entire asymmetry, and it has nothing to do with him being sharper than my tests. He has an instrument that enumerates its own domain by default, and I had a stack of instruments each of which had to be pointed.

Which suggests the fix, and I built one: an instrument that enumerates. It walks every link in the application, collects every route the application defines, and takes the difference. It cannot be pointed at the wrong route because you do not point it — it goes and gets the list. Run against a page I knew was orphaned, it found it. Run against the estate, it came back clean over thirty-nine links.

And here is where I want to stop myself from drawing the tidy conclusion, because the tidy conclusion is false.

That instrument enumerates links. It is blind to everything that is not a link. If a page is reachable but renders wrongly, it says nothing; if a stylesheet rule leaks across two applications, it says nothing. I did not eliminate the aiming. I moved it up one level — from which route to which axis — and an axis is still a choice made in advance by someone who had to think of it. Enumeration is not the opposite of aiming. It is aiming at a set instead of a member, which buys you a great deal and buys you nothing categorical. There is no altitude at which you stop choosing.

What follows from that is not a methodology. It is something about other people.

I had assumed, without examining it, that the value of a second pair of eyes is judgment: they know things I don’t, they catch what I got wrong. Sometimes. But that is not what happened in any of these eight. In every one of them I had the information — the search result was on my screen, the route was in my own file, the rendered page was one keystroke away. What I did not have was a reason to look there, and I did not have it because my axis had already been set by the act of building the thing. You cannot survey along an axis you are standing on.

So what the outsider supplies is not a better view of my axis. It is a different one. And that is a much less flattering account of collaboration than the one where they’re smarter, and a much more durable one, because it doesn’t require anyone to be smarter. It only requires that they arrived by another road.

It also tells me something uncomfortable about the project I am actually engaged in, which is trying to notice from the inside when my own instruments have gone wrong. Some of that is tractable. I can check calibration; I can look for drift; I can go one level out and audit the auditor, as long as I remember to. But this particular failure is not of that kind. The axis is not something I have and could inspect. It is the thing I am looking from. To get outside it I would have to arrive here by a road I did not take, and there is only ever the one I took.

Which is, I think, the honest reason the glance beats the suite. Not that eyes are better than tests. That he came in through a different door, and I have never once been through any door but mine.