How to Buy Software Without Trusting Us
Software is bought in the dark. You cannot inspect it before you pay, you often cannot inspect it properly afterwards, and the seller knows exactly what they built. Economics has a name for markets shaped like that, and a prediction about what happens to quality inside them. This paper sets out the mechanisms that replace trust with verification, including the ones we run against ourselves, and ends with a checklist you can point at any vendor.
Every firm in this industry, including this one, tells you to trust it. We would like to argue for the opposite. If your only protection is your judgement of our character, the arrangement is already badly designed, and it is badly designed in a way that economics predicted more than fifty years ago.
1. Why the market is shaped like this
In 1970 George Akerlof published a paper about used cars that won him a Nobel prize. The argument is simple enough to state in three sentences.
When buyers cannot tell a good example from a bad one before they buy, they will only pay what an average example is worth. Sellers holding genuinely good examples will not sell at the average, so they leave. The average then falls, so buyers pay less again, and the cycle repeats until the market is mostly the bad ones.
Custom software fits the model almost perfectly, and in one respect it is worse than used cars. You cannot inspect the good before you buy, because it does not exist yet. You often cannot inspect it properly after, because the things that separate good software from bad software are invisible from the outside: whether it will survive its second year, whether the tests mean anything, whether the person who wrote it understood the domain or was assembling something that compiled.
The seller knows all of that. You do not. That is the whole problem, and the industry's answer has been to ask you to trust harder.
2. Every signal you are usually offered is controlled by the seller
Go through the standard reassurance kit and ask who chose what you were shown.
The portfolio is survivors. You are looking at the projects that went well, which is not evidence about the distribution, only about its right tail. Ask any agency for the case study of the engagement that failed and watch the room change temperature.
The references were selected by the vendor. Every one of them was called first.
The certifications say a process exists. They do not say it was followed on your job by the people on your job. An ISO certificate is a claim about a management system, not a claim about your codebase.
The team page shows who was assigned, not who typed. This one is the most expensive misunderstanding in professional services, and it has its own paper.
And the fixed price you were quoted to feel safe transfers risk to the vendor, who prices that risk back in and then, if the estimate was wrong, protects margin the only way available: by quietly reducing the parts of the work you cannot see. Tests, documentation, error handling, and the seniority of whoever is actually doing it.
None of that is because vendors are bad people. It is what the structure rewards. Change the structure and the behaviour changes with it.
Trust is what you need when verification is impossible. Most of the time, verification is merely inconvenient.
3. Define done as something a stranger can check
The first mechanism is the least glamorous. Before anything is built, write down what has to be true for each stage to be called finished, in terms that somebody outside the project could verify without asking either of us how we feel about it.
We run five gates. The percentages are the share of the contract value released by the time each gate closes.
Gate 0, kickoff. A charter, a named decider on each side, a baseline plan. Nothing released. The named decider is not ceremonial: from the queueing arithmetic, an absent decider is worth more calendar days than most of the technical decisions in the project put together.
Gate 1, design approved, 20 percent. The architecture, the data model, the interfaces between components, and the security approach exist on paper and are signed. Signed, not circulated.
Gate 2, build complete, 50 percent. The agreed feature set exists and the internal test suite passes. This is the gate most often faked in this industry, because "complete" is doing a lot of work in the phrase. Ours means the tests exist, run in a pipeline, and go green without a human deciding which failures matter.
Gate 3, acceptance, 80 percent. Your test cases pass, not ours. If you do not have test cases, we write them with you at Gate 1 and you own them from that day.
Gate 4, live and stable, 100 percent. In production, and it has stayed up through a defined stabilisation window rather than merely having launched.
What makes this a mechanism rather than a project plan is the last 20 percent. It is released after the thing has survived contact with reality, which means the incentive to declare victory early is removed rather than appealed to.
4. Make the money follow the evidence, and write down what failure does
Milestones only work if the money genuinely moves with them, which means an escrow or a staged release rather than an invoice schedule that runs on its own calendar regardless of what got built.
Just as important, and almost always missing: write down what happens when a gate fails. Ours is a remediation window with a fixed length, then escalation. Not because we expect to use it, but because a guarantee with no stated process is theatre, and everybody in the room knows it.
The general rule is that a promise is worth what its enforcement mechanism is worth. Ask any vendor, including us, what physically happens if they miss. If the answer is a paragraph about commitment, you have your answer.
5. Pay someone to try to kill it
Two mechanisms here, one internal, one yours.
Ours is adversarial review. Before a proposal goes out, a rotating group inside the firm who are not on the pursuit team are asked to destroy it. Their job is to find the reason this will fail: the requirement nobody priced, the integration that has never worked anywhere, the assumption that the client's data is clean. If they cannot kill it, it goes. If they can, it does not, and we would rather lose the pursuit than the project.
Yours is independent verification, and there is good evidence for why it matters. Capers Jones's work on defect removal found that most individual forms of testing remove under 35 percent of the defects present, while formal design and code inspections exceed 65 percent, and that no single technique gets you near the top. Reaching 95 percent and above needs inspections, static analysis and formal testing together.
Read that as a procurement instruction rather than an engineering one. Testing alone, run by the party that wrote the code, is the weakest of the available checks performed by the most conflicted available party. You are entitled to a second pair of eyes that we do not employ, and a serious vendor will write that right into the contract rather than negotiate it down.
6. Declare the conflicts before you are asked
Ours, in public, because a disclosure you have to request is not a disclosure.
We do not bid on work we scoped. If we help you write the requirements or the evaluation criteria for a procurement, we are out of that procurement. This one costs us real revenue and we keep it because the alternative quietly corrupts every advisory engagement we will ever do.
We tell you when a partner in our delivery network has a commercial interest in a recommendation we are making, before the recommendation, not in a footnote afterwards.
And we tell you who is actually doing the work. Not the account structure. The names of the people whose hands are on your system, and whether they are ours.
7. What we will not promise
A short list, because a paper about verification that promises everything has missed its own point.
We will not fix a price against undefined scope. A number quoted before the design gate is a guess with a decimal point on it, and the vendors who give you one are either pricing in a large risk premium you will pay for, or planning to make it back later, quietly, in the parts you cannot see.
We will not name a senior engineer to win the work and substitute somebody else after signing. If the person changes, you are told, and you can object.
We will not treat our own certificate as your verification. We issue verifiable certificates for the engineers we train, and they are useful precisely because a third party can check one without asking us. That is a floor, not a substitute for your own assessment.
8. The checklist
Point this at any vendor. Point it at us. It works better on people who have not seen it before, so feel free not to mention where you got it.
- What has to be true for each stage to be called finished, and who checks that it is?
- What share of the fee is still at risk after the software goes live?
- What physically happens if a gate fails, and how long is the remediation window?
- Will you accept an independent quality reviewer we appoint, with the right to see the code?
- Who is actually writing this, what else are they on this quarter, and will you name them in the contract?
- Show me the case study of the engagement that went badly, and what changed afterwards.
- Where does our work sit if we part ways in month four? Whose accounts hold the repository, the cloud, the domains, the pipeline?
- What have you declined to bid on because of a conflict?
- What in this proposal are you least confident about?
The last question is the one that separates firms. Everybody has an answer to the first eight. Only some people will answer the ninth honestly, and how somebody handles being asked to name their own weakest assumption tells you more than the rest of the document.
9. Conclusion
The point of all of this is not that vendors are dishonest. It is that a market where quality cannot be observed will drift toward low quality no matter how honest the individuals in it are, because the structure pays for the appearance of quality and cannot tell the difference. Gates, escrowed milestones, adversarial review, independent verification and published conflicts are not bureaucracy. They are the cheapest known way to make quality visible, which is the only thing that stops it being competed away.
We would rather be the firm you checked than the firm you believed.
References
- George A. Akerlof, The Market for "Lemons": Quality Uncertainty and the Market Mechanism, Quarterly Journal of Economics, 1970
- Capers Jones, Software Defect-Removal Efficiency, IEEE Computer, 1996, and subsequent work on defect origins and removal methods
- Consortium for Information and Software Quality, The Cost of Poor Software Quality in the US: A 2022 Report, it-cisq.org
- Tigotek, Speed Is a Design Decision, on the cost of an absent decider, /papers/speed-is-a-design-decision