You Can't Assert A Shadow

Photo by Dibakar Roy on Unsplash

I was auditing the estate for something unrelated, and I ran a count I had never bothered to run. Forty-six repositories on this machine. Only one of them has a test suite.

Thirty-two tests, in QuitProtect, running in 24 milliseconds. That is the lot.

I sat with that for a bit, because it is exactly the thing I would raise an eyebrow at in somebody else’s project. Then I counted what is there instead, and the number came out differently than I expected.

What I built instead of tests

Citadel has 6,910 lines of code in a folder called Harness. Twelve files, called things like AudioProbe, RigProbe, RasterShot, SettleProbe and MaterialPreview, with thirty environment switches to turn them on. That is more verification code than most of my apps have application code.

None of it asserts anything.

Every one of those files does the same shape of thing. It sets up a scene, renders it, and writes out a picture or a measurement for me to look at. RasterShot takes a frame. RoomShots walks the level and takes one per room. RigProbe puts a creature in a neutral pose so I can see whether the skeleton is where I think it is. Then I open the folder and use my eyes.

I assumed for a long time that this was the lazy option. There is a specific bug that taught me otherwise.

Sixty-six centimetres of nothing

A character in Citadel looked like it was hovering. Not dramatically. A sliver of daylight where a foot met the floor, at some distances and not others.

I went into the animation code, because floating sounds like a placement problem and placement problems live in animation, and I measured it. Across every walking animation, the lowest point on the character sat within 3 millimetres of the floor. The character was on the floor. It just did not look like it.

It was the shadow, and the way a shadow gets drawn is worth explaining.

The scene is rendered twice. Once from the camera, which is the picture you see, and once from the main light source. The second pass throws away the colours and keeps only distances: for every direction the light faces, how far does it get before something stops it. Then, drawing the real frame, each point asks whether it is further from the light than whatever the light hit first. If it is, something is in the way, and it is in shadow.

That comparison is never exact, because both distances are approximations. A flat surface that is a whisker out will decide it is blocking itself, and comes out crawling with speckle. So we add a fudge factor: shift the comparison a hair in the light’s favour, and the speckle goes away.

Mine was 0.0022, and that is where it got me. The fudge is not in centimetres. It is a fraction of the whole span the shadow pass covers, from the nearest thing in the light path to the furthest, and I had set that span to 299 metres. So the real size of the fudge was 0.0022 of 299 metres. Sixty-six centimetres.

Which would be survivable if it pushed the shadow straight down. It does not. It shifts the shadow along the direction the light is travelling, so the dark patch slid out from under the foot that cast it, by two thirds of a metre, and your eye reads a gap under a foot as height.

That 0.0022¹ value looked innocuous at first glance because I had never converted it into a unit that meant anything.

Now write me the assertion. Not the fix. The assertion: a test that fails while this is happening and passes once it stops. Every value in that frame is defensible. The character is in the right place, and I have the measurements to prove it. The distances recorded in the light source pass are exactly what the maths says they should be. There is nothing to compare against a threshold, because the fault is not in a value. It is a relationship between two individually correct things, and the instrument that detects it is a person looking at a picture and saying that is wrong.

Even finding it took looking at the right picture. A close-up preview, the character slowly turning on the spot, showed no shadow at all, so it looked fine. The gap only appeared in frames shot from a distance.

So: I built the thing that produces pictures.

Somebody else’s testing page

I read SQLite’s testing page this week, which I recommend to anybody who thinks they are thorough. SQLite is about 155,000 lines of code. The tests come to 92 million. That is 590 lines of test for every line of program, four separate harnesses, and a fuzzer running about 500 million cases a day.

They hold 100% MC/DC coverage on the core. That is the aviation standard, and it is stricter than it sounds: running every branch is not enough, you have to show that each individual condition inside a decision can independently flip the outcome. It is the bar that certifies flight software.

Two things on that page stopped me.

The first is that they say plainly it did not prevent the bugs. Fuzzers kept finding faults in code that already had every branch and every condition covered. What they say the coverage actually bought them is something quieter:

The fact that the SQLite test suite does test to 100% MC/DC means that when fuzzers do find problems, those problems can be fixed quickly and with little risk of introducing new errors.

Coverage did not buy correctness. It bought the confidence to change things afterwards. That is a far smaller claim than most people make for tests, and it is made by the people with the strongest claim available to them.

The second thing is stranger, and I have not been able to stop turning it over:

code tested to 100% MC/DC will tend to be more vulnerable to problems found by fuzzing and code that performs well during fuzz testing will tend to have (much) less than 100% MC/DC

The two most rigorous testing methods available pull against each other. The reason is almost funny once you see it. MC/DC demands that every branch be reachable and be shown to matter. A defensive guard, the if this can never happen then bail out that you write because you do not entirely trust the world, is a branch that cannot be reached. It is a coverage failure. Chasing full coverage quietly pressures you into deleting exactly the belt and braces that survive a fuzzer.

Two ways of being careful, and they want opposite code.

Where my way goes blind

That stopped me feeling defensive about the instruments and got me looking for where they cannot reach. It is easy to describe. They work when the failure has a shape. They are useless when the output is a value.

I had that demonstrated a few days ago while pressure-testing RememberMyWindows, the window manager I have been contributing to. I had a fix in, I ran the test, the test passed, I moved on. Three separate times. All three of those tests passed without ever executing the code they were meant to be testing, because a 90-second write-coalescing window meant the interesting branch never ran. Nothing crashed. The screen was full of windows in the right places, which is precisely what my whole approach is built to notice.

What caught it was a single line in a log, where the timestamp of the layout being restored disagreed with the timestamp of the file on disk. Not a picture. A number that did not match another number.

Any coverage tool on earth would have handed me all three on the first run, on day one instead of day nine. So the shape of it is symmetrical: my method is blind exactly where a coverage tool is strongest, and the reverse holds too.

The gap is not tests, generally. It is parsers, state machines, expiry windows, anything that compares two things for equality. Code whose output is a value with no picture attached. I have more of that than I would like to admit, and I am writing a parser this week that has already caught me out once in a way no screenshot would ever have shown.

The app with the tests

Which brings me back to the one suite, and to the most interesting fact I turned up.

QuitProtect’s tests cover a gesture state machine and the lifecycle of an event tap. Both are pure logic and neither produces a picture. Whether that state machine accepts a second hold after a release is not something using the app will reliably tell you, because you would have to perform the sequence in exactly the right order and then notice a subtle wrong outcome, and you will not.

That is not why the tests exist.

They exist because QuitProtect is the app strangers send pull requests to. Seven merged, from a contributor I have never met. The moment somebody else is changing your code you need a way to show that their change did not break the part you cannot see, and it has to work without them owning your hardware or having your eye for what a shadow should look like.

I did not decide to test that app. Other people turning up decided it for me, and I did what the situation needed without noticing it was a pattern.

What I am taking from this

Not that I should go and write tests for everything. SQLite say it themselves, having gone further than anybody: “the level of effort needed to maintain full-coverage testing is probably not cost effective for a typical application.”

The useful version is smaller. Match the instrument to the way the thing actually fails. A renderer fails by looking wrong, so build something that shows you a picture. A parser fails by returning a plausible value, so write something that already knows what the value should be. Getting it backwards costs you in both directions, because assertions on a renderer tell you nothing and eyeballs on a parser tell you less.

The two are not rivals either, which took me longer to see than it should have. The shadow bug was found by looking, and there was no other way to find it. But the fix was not a better tuned number. It was deriving the fudge from the shadow pass’s own resolution, so that changing that 299-metre span can never quietly bring the gap back. An eye found it. An invariant now holds it down.

That is the same trade SQLite describe, arrived at from the other end. Their coverage did not find the bug; it let them fix it without fear. My instruments do not defend anything; they are how the fault becomes visible in the first place. Discovery and defence are different jobs, and I had been asking one tool to do both.

Footnotes

1. I have described in a couple of paragraphs a thing that I spent three hair-pulling, angst-filled days trying to resolve. I hate magic numbers. ↩