On the largest system we have built, twenty defects reached a running system. We kept a record of how each one was found, not to be diligent, but because the answer turned out to be useful. Here is the distribution.
- Eight from running an operation end to end
- Four from scanning for a pattern after finding it once
- Three from adversarial security testing
- Two from load testing at twelve thousand orders
- Two from pre-deploy inspection
- One from the platform linter
- Zero from reading the code
Code review found none of them. Not a small number. None.
This is not an argument against review, which catches design problems and stops bad decisions early. It is an argument about where to put the hours in a plan, because a schedule that allocates a week to review and a day to exercising the system is allocating backwards.
The four that were genuinely instructive
The application hid the screens. The database did not.
Unprovisioned users could read the entire business. The interface correctly showed them nothing, and every navigation item was hidden, which is exactly what made it invisible. A direct query returned everything. The lesson is not subtle and it is still the most common finding in enterprise applications: if access control lives in the interface, you do not have access control. It has to be enforced at the data layer, where the query lands.
A margin view reported minus three hundred and forty six per cent.
Two figures in the same calculation used different bases. Nobody reading the SQL would have blinked, because each half was correct. It only became visible when a real job flowed through and produced a number so absurd that it could not be ignored. Had the error been six per cent rather than three hundred, it would still be there.
An upsert erased every field the caller did not send.
Master data was being silently corrupted by a routine that looked entirely reasonable. Update the two fields you have, write the record. Every other field went to null. This surfaced on a screen showing a customer with no payment terms, three days after the corruption started.
A test passed because its fixture was empty.
The assertion was correct, the logic was correct, and the test had been green for weeks over a table with no rows in it. A green suite is evidence of nothing until you have checked that the test would fail if the code were wrong.
What we changed as a result
Three things, all of them cheap.
Exercise before you inspect. Every operation is run end to end on real shaped data before anyone reviews the diff. The order matters, because running it tells you where to look.
When you find one, hunt the pattern. Four of the twenty came from asking where else the same mistake could exist. That is the highest yield activity on the list and it costs an hour.
Load test before the demo, not after. One view worked perfectly at demonstration scale and timed out at twelve thousand orders. Demonstration scale is a dangerous place to build confidence, because everything works there.
We publish this record for a reason that is partly commercial. A supplier who tells you their system had no defects is telling you they did not look. The useful question to ask any partner is not whether faults occurred, but how they were found, and what changed afterwards.