In December of 2012, Supply Chain Matters featured a commentary: A Billion Dollar Failed-Software Implementation-Why?.  Our commentary at the time was a response to a published New York Times article describing the failed attempts of the United States Air Force to implement a new and long overdue modernized logistics system.  The system was named Expeditionary Combat Support System (ECSS) and was targeted to implement commercial off-the-shelf ERP software. The Air Force eventually cancelled its six year long effort to implement the new software system after spending more than $1 billion dollars of taxpayer money and determining that it would take an additional billion to implement a workable system. At the time of our commentary, we were very frustrated at the news since the situation described was one of a classic systems failure.

Earlier this week, the United States Senate finally got around to completing its investigation of the Air Force system effort and issued a report that lambastes the U.S. Air Force for poorly managing the acquisition and attempted implementation of the ERP logistics management system. The Senate report, which can be easily downloaded, classifies the system failure “As a waste of $1.1 billion in taxpayer money, a loss of eight years of effort, the same old inadequate logistics system far inferior to the promise of ECSS, and a major setback to the Air Force’s attempt to transform how it does business.

The comedy and/or tragedy of this week’s Senate report is that it took roughly 18 months for the Senate to get to the investigation and set of recommendations. Its investigation places majority of blame on the Air Force without addressing in-depth, a government systems acquisition environment or an overall organizational culture that might have contributed to the failure. That alone is a testament to the bureaucracy and inefficiency of federal agencies when it comes to IT systems implementation. The high visibility of the initial failures of healthcare.gov is yet another testament to an inherent problem.

With Internet based information discovery being what is today, a simple Google search yields a lot more learning and perspectives relative to the specific ECSS debacle.

Here is a sample:

Sanjiv Karani in a commentary on the Cincom ERP site outlined the full scope of the effort:

The scope of the ECSS ERP project included: advanced planning and scheduling; material management, contracting and logistics finance; configuration and bill of material; repair and maintenance; product lifecycle management; customer relationship management; order management; distribution and transportation; decision support; facilities management; quality control; document management and budgeting.  ECSS would have replaced the capability of approximately 400 or so legacy IT logistics systems with Oracle’s integrated IT suite of modules comprised of product support & engineering, supply chain management, expeditionary logistics C2, and maintenance, repair and overhaul.”

Obviously, the systems scope was rather large.

The Federal Times web site provides a download of the ECSS Acquisition Incident Review Team Final Report. This comprehensive report outlines the background and contributing causes to the systems failure. Key highlights include:

The Air Force’s strategy was to acquire commercial off-the-shelf ERP software with a provision for “bolt-on” applications. A systems integrator was to be tasked with both implementations of standard ERP and bolt-on applications, along with the other program implementation needs. In late 2005, Oracle E-Business suite was elected along with two bolt-ons: Click Commerce for planning and Industrial and Financial Systems for maintenance process support. In 2009, a major restructuring of the program addressed the need for product lifecycle management and logistics financials. The Oracle contract was modified to include an Oracle only footprint including PLM and other needs. A restructured systems integrator contract with CSC outlined four releases with six pilots.

The planned first release dragged on into late 2010 prompting the Department of Defense to conduct a technical risk assessment. Both Air Force teams and CSC attempted to implement a “recovery plan” but the report indicates that it became apparent the plan was failing to meet its objectives. A series of Air Force leadership “deep dives” concluded that significant shortcomings in program progress had not been remedied.  A side note- we take that to mean that Air Force leadership was not getting what it perceived it wanted. Eventually by 2012, the Air Force recommended cancellation of the program when discovering that Release 1 was pushed back to fiscal year 2017.

The Air Force Incident Review Team described contributing causes to the failure as follows:

  • Governance- described as confusing and, at times, ineffectual governance structure evident throughout the program. Noted in the report: “There lacked coherent leadership guidance and coordination from process ‘owners” on how to seamlessly mesh and implement the intermingled methodologies , thereby driving needless delay, frustration, uncertainty, and labor burden on the program office.”  The above statement should resonate with our IT and systems consulting readers since active governance this is a fundamental tenant for success.

Other contributing factors were described as;

  • The lack of effective change management made worse with a lack of successful implementation progress, in effect, signaling that “the program was not worth supporting.”
  • Program leadership management churn that included six program manager changes in eight years along with five program executive officers over six years.
  • A lack of understanding by the Air Force of all of its data along with the “as-is’ or “to be” architectures, coupled with a transition plan for resolving differences between the two states. Even though the Air Force determined that no single, stand-alone commercial system could satisfy its needs, there was apparently no singular specific articulation of the “to-be” state.  The report notes that this was compounded by the addition of multiple subject matter experts with no single vision as to how a transition would occur.
  • An unrealistic development environment, one that did not mirror the reality of the required operational environments.

As noted in our prior 2012 Supply Chain Matters commentary, more disturbing is that multiple U.S. military and government agencies are in need of modern logistics processes and yet the track record for software and change management implementation efforts is marginal at best. The latest reports of the Senate investigation makes mention of an additional Defense Enterprise Accounting and Management System that is in a similar mess, upwards of $1.7 billion over budget and considerably behind schedule. Another report makes mention of the Air Force’s F-35 Joint Strike Fighter aircraft program, an aircraft that requires 24 million lines of code to operate which has experienced multiple development setbacks.

Private industry has garnered its share of previous painful learning regarding multi-million dollar big-bang software implementation projects that ultimately failed to see the light of original scope or intent. Some brought businesses to their knees, with painful consequences. Yet, the technology successes of Amazon, Google, Facebook, and Linked-In, to mention but a few, are testaments that technology is no longer the sole obstacle. That learning is available to any implementation team today, including U.S. government agencies.

We seriously doubt that the watered down recommendations from the Senate and other agencies will get to the heart of badly needed the government IT modernization efforts.  It is clearly time for U.S. government to seriously consider the adoption of business transformation practices of private industry that include:

  • Establishing clear vision of future state needs that are not, business as usual which favors continued existence of added complexity and bureaucracy?
  • Avoiding huge-scope big-bang systems efforts in favor of incremental steps of manageable projects
  • Senior agency leadership singularly accountable and responsible for program success.
  • The recruitment of IT and CIO leadership with proven private industry track records provided with the organizational clout and support to get things done.

Finally, the Congressional toxic political environment being what it is today, requires that political leaders stop thinking in the context of blame and retribution, especially when the process drags on for years.  The federal government needs both visionary agency leaders and legislators who can think outside the box of bureaucracy.

Whether any of this happens in our lifetimes is certainly food for thought and constructive debate.

 Bob Ferrari