Who Finds the Bugs?
Photo by Chris Benson on Unsplash
In April I wrote that my apps don’t phone home. No analytics, no crash reports, no usage data. In the same post I admitted what that costs: when someone hits a bug, I rely on them telling me, and most people won’t.
Six months on, there is an obvious question. If nothing reports back, who finds the bugs?
Mostly, I do. I am the telemetry.
An ordinary Monday
Today was a normal working day, and two of my apps let me down during the course of it.
Jorvik Daily News is the first thing I read each morning. This afternoon I removed the lead story, and the next headline moved up into its place. The picture didn’t. For about half a minute, a headline about the internet sat under a drawing of New York’s dogs. The app had kept the old picture on screen until the new one finished downloading, and that site was slow.
A little later, the paper led with a portrait of a roboticist standing beside his robot. Both of them had lost their heads. The cropping code was wrong. Six weeks, and I never noticed.
Then I took a screenshot of a work conversation in Microsoft Teams, and Nomen named it after Ghostty, my terminal. macOS saves a screenshot a few seconds after you take it. In those few seconds I had switched to Ghostty to paste it, and Nomen asked which app was in front too late.
All three are fixed, and the fixes shipped this evening. None of them came from a dashboard or a crash log. Each one got in my way while I was trying to do something else.
Why it works
This only works because I am a real user, not a tester. First and foremost, I write these apps to satisfy my own needs. Nobody commissioned them. Each one exists because something on my Mac annoyed me enough to fix it.
So I read the paper every day because I want the news. Nomen renames my screenshots because I take dozens of them at work. I don’t open these apps to look for bugs. I open them to get something done, and a bug has to interrupt that before I see it.
That turns out to be a good filter. The bugs I find are the ones that cost a real person time. Nobody has to argue for them in a planning meeting. I have already felt them.
It helps that I am my own biggest critic. A picture cropped through someone’s forehead is not a bug report I file and come back to next quarter. It bothers me until it is fixed.
Why it isn’t enough
The robot shows the limit.
That crop bug was in every copy of the app for six weeks. It affected only the pictures that needed cropping, and on a typical front page that is about three in twenty-four. A centred crop often looks fine. So I read the paper every morning for six weeks and saw nothing wrong, until a picture arrived with the faces in exactly the wrong place.
One person sees what one person’s screen shows. My Mac, my displays, my feeds, my habits. The Nomen bug existed because I paste screenshots into a terminal. Somebody else’s habits would uncover bugs mine never will, and mine will keep hiding them.
The telemetry I didn’t collect
This is where other people come in.
When strangers started opening issues on GitHub, they did what telemetry is supposed to do. fuzzy76 found that MenuTidy ignored the first click on Tahoe. thechurchofthechad-dotcom told me that Jorvik Daily News had stopped opening articles at all. emjayess found that RainbowApple left its logo floating in mid-air when the menu bar hides itself. I don’t hide my menu bar. I would never have found that one.
Each of them told me something I could not have learned from my own usage. That is exactly what telemetry promises. The difference is that they chose to tell me. They decided what to send, they wrote it in their own words, and some sent screenshots and steps to reproduce.
Others sent something telemetry can’t send at all. RSS1102 translated QuitProtect into Simplified Chinese, and opened a pull request that exposed a significant bug I had shipped in several apps. Christophe has added six features to Save Cannes, from playback profiles for each display to an art mode for a single screen. None of that was a bug report. It was a person telling me what they needed, with the code to go with it. No dashboard has ever done that.
I wrote about that in August as a thank-you. It is also the other half of the decision I made in April. I gave up knowing things about my users in secret. What I get instead is a smaller number of people telling me things in the open, and I think that is the better deal.
Dog food
There is an old phrase for using your own product: eating your own dog food. The point is to suffer what your customers suffer, before they have to.
I do eat mine, every day, and today it was gritty in three places. But I am one dog. If something in one of these apps gets in your way, it may simply not have got in mine yet. The issue tracker for every app is open, and I read every report.