Saturday, October 26, 2013

What Really Went Wrong with Healthcare.gov?

In a world where IBM's Watson has already started looking like a relic, its unbelievable that there are still places where building a website is well... rocket science. David Auerbach's brilliant CSI  (here and here) on slate has lessons for any project team:

So we had (at least) two sets of contracted developers, apparently in isolation from each other, working on two pieces of a system that had to run together perfectly. Anyone in software engineering will tell you that cross-group coordination is one of the hardest things to get right, and also one of the most crucial, because while programmers are great at testing their own code, testing that their code works with everybody else’s code is much more difficult.

Look at it another way: Even if scale testing is done, that involves seeing what happens when a site is overrun. The poor, confusing error handling indicates that there was no ownership of the end-to-end experience—no one tasked with making sure everything worked together and at full capacity, not just in isolated tests. (I can’t even figure out who was supposed to own it.) No end-to-end ownership means that questions like “What is the user experience if the back-end gets overloaded or has such-and-such an error?” are never asked, because they cannot be answered by either group in isolation. Writing in Medium in defense of Development Seed, technologist and contractor CTO Adam Becker complains of “layers upon layers of contractors, a high ratio of project managers to programmers, and a severe lack of technical ownership.” Sounds right to me.

Let me explain how end-to-end testing works in integrating large systems owned by multiple vendors. Each vendor works out detailed specifications for how the systems should interact. These are made as clear as possible so that when something goes wrong—and it always does—you can point to the spec and say, “You weren’t supposed to do that and that’s why our component appeared to misbehave.” In order to meet the specs, each vendor simulates end-to-end testing by building a prototype of the larger system.

In the case of healthcare.gov, QSSI should have had test scaffolding that could simulate the functionality of what CGI Federal was building, and vice versa. Each vendor needed to know the broad definition of what the other was building, and it was their responsibility to make sure they knew it. Campbell and Slavitt’s refusal to acknowledge this basic fact is both frightening and mortifying, and accounts for their inability to give any clear answers as to exactly which portions of the system are failing. They don’t seem to understand the difference between acceptable and unacceptable bugs, and worse, they don’t seem to know that there is a difference.

Campbell said, “We were quite optimistic that our portion of the system would work when the system went live.” That is either a lie or a revelation of ghastly incompetence, because no competent programmer or manager would ever display a shred of optimism until full end-to-end testing had been done. To say that “our portion of the system would work” is akin to saying that you know a computer will work before you’ve hooked a monitor up to it, just because it turns on. There’s no half-credit in these cases.


No comments: