Why do so many IT projects fail?

IT projects fail for reasons that are mostly not about technology. The recurring causes are unclear requirements, no real sponsor, scope that grows without anyone deciding it should, a habit of measuring activity instead of outcomes, and a reluctance to stop once money has been spent. Each is a human failure of governance, and each is avoidable with attention that costs far less than the overrun it prevents.

The numbers around IT failure are worth handling with care, because some of the most quoted ones are also the most disputed. What is not in dispute is that large technology projects run over with grim regularity, and that when they run over, they tend to do so spectacularly.

How often do IT projects really fail?

Often enough to be the expectation rather than the exception, and the failure is usually one of budget and value rather than a total collapse.

McKinsey's study of large IT projects, Delivering large-scale IT projects on time, on budget, and on value, found that on average these projects ran 45 per cent over budget and 7 per cent over time, while delivering 56 per cent less value than predicted. The report went further, noting that half of all large IT projects, meaning those with initial price tags above 15 million dollars, massively blow their budgets. A project can therefore be delivered, go live and still be a failure, because it cost nearly half as much again as promised and returned barely half of what was pledged.

The point of that statistic is not to frighten. It is to reset the baseline. If you begin a project believing failure is rare, every warning sign looks like bad luck. If you begin knowing that overruns are the norm, you watch for them, and watching is most of the cure.

It also changes how you plan. A team that expects to be exactly right sets one date and one figure and treats any deviation as a scandal to be hidden. A team that expects to be wrong builds in the room to be wrong, checks its assumptions early and often, and treats a revised estimate as new information rather than an admission of failure. The second team overruns less, not because it forecasts better, but because it is honest about forecasting sooner.

Should I trust the famous failure statistics?

Only with the caveats attached, and this matters more than it might seem. The single most cited source on software failure is the Standish Group's CHAOS report, which claimed a project success rate as low as 16 per cent in its 1994 edition. That figure has been repeated in thousands of presentations, usually with no mention that it has been seriously challenged.

The challenge is the important part. In The Rise and Fall of the Chaos Report Figures, published in IEEE Software in 2010, the researchers J. Laurenz Eveleens and Chris Verhoef tested the CHAOS methodology against their own dataset of 5,457 forecasts across 1,211 real projects. Their verdict was blunt: the Standish figures did not reflect the reality of the case studies at all. They set out four problems, arguing that the definitions are misleading because they rest solely on the accuracy of cost, time and functionality estimates, that they are one-sided and so produce unrealistic success rates, that using them as management targets perverts good estimation practice, and that they average numbers with an unknown bias.

So the honest position is this. Yes, IT projects fail at a rate that should worry anyone funding one. No, the precise figures you have heard are not gospel, and a project measured only against its original estimate can be branded a failure for being estimated badly rather than delivered badly. Both things are true. Anyone who quotes the scary number without the caveat is selling something, and it is usually their own methodology.

Why do requirements go wrong so often?

Because everyone assumes they agree, and no one checks until it is expensive to find out they did not. Requirements are where a project is won or lost, and they are the least glamorous part, so they get the least care.

The trouble is that a requirement written down is not the same as a requirement understood. A sentence like "the system should make reporting easier" feels like agreement in the room and means five different things to the five people who nodded at it. The gap does not show until the build is half done and someone sees the screen and says that is not what I meant. By then the cost of the misunderstanding has been baked into code.

McKinsey's breakdown of where overruns come from puts unclear objectives and shifting content near the centre of the problem, alongside weak governance and quality control. The remedy is unfashionable and effective: slow down at the start, write requirements as concrete outcomes a user could test, and make the people who will use the thing sign off on examples rather than adjectives. An hour of disagreement at the requirements stage is the cheapest hour in the whole project. Every hour it is postponed costs more.

What difference does a sponsor actually make?

The difference between a project that can make decisions and one that cannot. A sponsor is not a name on a slide. A sponsor is the senior person who wants the outcome badly enough to spend their own credibility protecting it, settling disputes, saying no to distractions and standing up when the project hits its bad month.

Projects without a real sponsor die a particular death. Nobody kills them. They simply stop being anyone's priority. Decisions that need a senior voice wait in a queue, the team fills the silence with guesses, the guesses turn out wrong, and rework piles up while everyone waits for a direction that never comes. The project does not fail loudly. It fades.

McKinsey found that attention to strategy and stakeholders prevented roughly half of all cost overruns, and a sponsor is how strategy and stakeholders get attended to. When the person who is meant to own the outcome is too busy, too junior or too indifferent to make the calls, no amount of project management underneath them can compensate. The sponsor is the load-bearing wall. Remove it and the decorative work is irrelevant.

What is scope creep, and why is it so hard to stop?

Scope creep is the slow addition of features and changes that each seem small and reasonable, until together they have doubled the work without anyone ever deciding to double the work. It is hard to stop because every single addition has a good argument behind it, and refusing feels petty.

The danger is that scope creep is invisible in the moment and obvious in hindsight. No one asks for a 40 per cent bigger project. They ask for one more field, one more report, one more edge case handled, and each request is granted by someone who does not see the others. The budget and the deadline were set for the original scope, so the additions are paid for in overruns and late nights that nobody signed off.

The cure is not to refuse all change, which is impossible, but to make change visible and priced. Every addition should cost something openly, in time or money or in something else being dropped, and that trade should be a decision someone with authority makes on the record. Scope creep thrives in the dark, where changes are absorbed quietly. It withers the moment each change has to be paid for in daylight.

There is a subtler form worth naming, which is scope creep by gold-plating. This is the team, rather than the customer, adding polish that no one asked for because it is satisfying to build. It is harder to catch than a customer request, because it comes dressed as diligence. The discipline is the same. If it was not in the agreed outcome, it is a change, and a change has to be chosen rather than smuggled.

Why do teams measure activity instead of outcomes?

Because activity is easy to count and outcomes are hard, and a status report full of green ticks is more comfortable to present than an honest one. So the project reports tasks completed, hours logged and milestones passed, all of which can look healthy while the thing being built is quietly failing to do the job it was funded for.

This is how a project can be on schedule and on budget right up to the moment it is cancelled. The activity metrics were all green. What was never measured was whether the work delivered the outcome, and by the time anyone checked, the money was gone. The green ticks measured motion, not progress, and the two are easy to confuse when you would rather not know the difference.

The fix is to define, before a line is written, the one or two outcomes the project exists to move, and to report against those even when the news is bad. A hard truth early is cheap. The same truth discovered at go-live is ruinous. Measuring outcomes means accepting uncomfortable status meetings, which is exactly why so few projects do it, and exactly why the ones that do tend to survive.

Why is it so hard to stop a failing project?

Because of the sunk-cost trap, one of the most reliable ways smart people make bad decisions. Having spent a great deal already, we feel bound to carry on, even when carrying on is plainly the worse choice.

The behavioural scientists at The Decision Lab describe the sunk-cost fallacy as our tendency to follow through with something we have invested heavily in, whether time, money or effort, even when giving it up is clearly the better idea. The more we have poured in, they note, the more committed we feel to pouring in more. This is escalation of commitment, and a struggling IT project is where it does its worst work, because the sums are large and the embarrassment of stopping is public.

Bent Flyvbjerg and Alexander Budzier gave the vivid warning in their Harvard Business Review article Why Your IT Project May Be Riskier Than You Think. Their opening case is a project at Levi Strauss budgeted at under 5 million dollars that ended in a charge of 192.5 million. Blow-ups on that scale do not happen because a project was 10 per cent wrong on day one. They happen because a project that was 10 per cent wrong was allowed to keep going, each stage justified by the last, until the loss was catastrophic. The money already spent is gone whatever you decide next. The only real question is whether the money still to be spent will be well spent, and a project that has stopped delivering value cannot answer yes.

What would you do differently, then?

Not much that is clever, and everything that is disciplined. Insist on requirements written as testable outcomes and signed off by the people who will use the system. Appoint a sponsor with the seniority and the appetite to make hard calls, and if you cannot find one, do not start. Make every change to scope a priced, recorded decision. Measure the outcome, not the activity, and report it honestly when it is bad. And build in the habit of asking, at each stage, whether the project would be worth starting today given what is now known, treating the money already spent as gone. If the answer is no, stop, and count the stopping as a success rather than a defeat.

David Luria's book The Failure of I.T. Project Management works through these failures at length and draws the same conclusion from a different angle: the technology is rarely the thing that breaks. What breaks is the ordinary discipline of deciding clearly, owning decisions and being willing to change course.

None of this is a secret. It is simply harder to do than to say, and it asks for honesty at the moments honesty is least welcome. That is the whole of it. The projects that succeed are not the ones with the best engineers. They are the ones that were governed like they mattered.

← All posts