Nov 12

Failure Is An Option: The Way To High-Performance Innovation

The three keys to innovation are to seek out new ideas, test these ideas on a scale where failure is survivable, and constantly monitor these trials for feedback. This is according to Tim Hartford's new book, Adapt: Why Success Always Starts With Failure. Hartford argues that the world is too complex for top-down “big project” innovation based purely on expert judgment. The best path to innovation is to try a lot of ideas simultaneously (even if they contradict each other), build in robust feedback loops, and use the winning ideas to start a new round of trials.

This is not a new method of innovation; in fact it is the oldest method of innovation around – evolution. Nature is constantly creating variations of species and then selecting the species that best survive current conditions. What Hartford does is apply that concept to organizations to see if a similar process works in determining what companies succeed and which go out of business. The organizations that best survive a constantly changing business environment are the ones that combine incremental improvement along with the occasional long-shot idea to propel them into a better part of the business landscape ahead of their competitors

So, what does this have to do with government agencies? Hartford flatly states that this innovation method will not work in government agencies because of several barriers. First, there is not enough time for political appointees to fully see these experiments through before a new administration comes in office. Second, the process depends on a large number of failures for innovation but failure carries a high stigma in government. Third, it is difficult to clearly demonstrate that a policy innovation actually had an effect due to the lack of robust feedback loops in government.

I believe that Hartford's opinion about government innovation is way overstated. There have been numerous government projects that have been extremely innovative: the Hoover Dam, rural electrification, the Interstate Highway System, the Moon landings, the Space Shuttle, the Internet, and so on. Even so, when you examine how these agencies developed these projects you do see that these agencies tried many ideas and learned from these trials. NASA has an amazing knowledge management culture and DARPA's successful record of innovation is built on the concept of trying many long-shot ideas at once.

What holds government back from being even more innovative is the stigma of failure. Many agency cultures are too cautious because of the constant external scrutiny and the internal cultural practices of not sticking your neck out and just waiting out the latest change effort. In many cases, this caution is well-warranted. Many people depend on government agencies and thus agencies cannot fail in their primary mission of delivering Social Security checks, defending the nation, or enforcing laws and regulations

But failure to innovate will also lead to mission failure for agencies. In the sixth chapter of his book, Hartford describes how the 2008 economic meltdown was inevitable given the tight coupling of economic institutions and the failure of the government to prevent financial organizations from becoming too entangled. He argues that in any complex system, accidents will normally occur and that often our failure-prevention efforts will only increase the probability of failure. What is needed are the twin strategies of placing buffers between parts of the system and setting up feedback loops to warn us of emerging failure events.

Government has to constantly innovate so that it can continually deliver on its mission. This means that the culture has to change so that the agencies accept the small failures that teach to avoid the large failures that cripple the agency and harm the people it serves. Whether we call it “experimentation”, “pilot tests”, or some other euphemism, the better government is at innovation the better it can serve its citizens.

Disclaimer: The opinions in this posting are solely mine and do not reflect the views and opinions of my employers or any organizations I belong to.

Belfiore, M. (2009). The Department of Mad Scientists: How DARPA Is Remaking Our World, from the Internet to Artificial Limbs. Smithsonian.
Hartford, T. (2011). Adapt: Why Success Always Starts with Failure. Farrar, Straus and Giroux.

Leave a Reply