Why do software projects fail?

On the radio I heard a couple of these horror stories of large (semi-)government software projects failing. The presenter asked himself the eternal question: why do these projects fail so often?

Can large projects be done in an Agile way? Without proof, I'm pretty sure they can. All you need is some people with talent and vision. And that's exactly what lacks on these large projects. I've seen a few in my days. The terror of SDM-like project management tools make it impossible, even for the good people, to make the project work.

For people who know nothing about making software this is all a mystery. They compare to physical building projects, and see that the Waterfall-conducted projects are just like that. Why would that approach not work in software? Here's one refutation to this analogy. Here's mine.

Let's say the physical project we're talking of is the installment of a sewage system in a part of a city. Planning encompasses re-directing traffic, breaking up roads, getting the old systems out, placing new ones, taking care of cables, and so forth. This is all planned ahead and executed in a tight schedule.

Now consider the computer world. There, things change at an enormous speed. Think about the question whether at the end of the year people will still want sewage - maybe destruction-in-place has been introduced. Roads may have gone because everyone's flying by then. Ten times more cables need to be laid out than expected. Now imagine you didn't really know what to do in the first place. All your precious large phase X final reports are useless,and lead to ever more costs and failures. Imagine what will happen when even the physical building projects often go over budget and past schedule.

Another problem is that very often in the computer world there is hardly anybody in the top of the government organizations with enough knowledge and experience to be able to really grasp what the computer companies are offering, and whether this is OK. The Cap Gemini's of the world just claim what they offer is necessary, and who can refute it? Certainly not the government people that were trained in exactly the same environments.

Lacking true insight, and lacking methods that have change as a basic principle - it's no wonder these projects fail. Just adopting Nazi methods demanding that things have to be 100% controlled will never guarantee success ...