February 24, 2021

by Steve Brothers

Government system failures during the rush to provide public benefits to alleviate the economic effects of the COVID-19 pandemic publicly exposed the mainframe knowledge crisis that also threatens financial institutions, healthcare providers, and many other organizations foundational to the world economy.

Several states discovered the knowledge-gap’s potentially devastating consequences when waves of unemployment-claims poured into their systems as COVID-19 ravaged the economy in early 2020. The states’ unemployment computer systems crashed trying to process the deluge of claims using mainframes and decades-old programming languages.

But it wasn’t the mainframes or the legacy programming languages that failed, despite what you may have read. It was the lack of available expert programmers necessary to maintain and update these systems to handle the voluminous claims.

According to The Verge’s article, “Unemployment checks are being held up by a coding language almost nobody knows,” Colorado employed exactly one full-time programmer to maintain the state’s COBOL system prior to the pandemic. Back then, Colorado processed roughly 2,000 unemployment claims per week. In March and April 2020, that number rocketed to as high as 104,572 claims per week.

Now governments, non-profits, and private organizations are reviewing their systems’ strategies to learn from these mistakes. If your business relies on legacy systems, you probably should keep reading – and schedule some time with your IT people.

Mainframes are cornerstones

Legacy mainframe systems and software bedrock many of our most trusted institutions, including government services, finance and banking, healthcare, and insurance. In a substantial number of cases the expert developers that created and maintained these systems and software are retiring without a supporting workforce to replace them.

Besides the people that make your business run, your software is potentially the most important resource your organization has. Internal applications likely drive your employees' capabilities and productivity. Customer-facing programs attract new customers, close business deals, and increase revenue. New applications and features can open new markets and opportunities.

To maintain and improve your critical applications, your software team relies on individual engineers that developed an expert understanding of your programs through years of experience. They know the applications and all the accumulated system changes and challenges.

When those experienced engineers depart your business, the developers that replace them must acquire the same application knowledge through training, mentorship, and on-the-job programming. This exercise introduces several material business risks.

Learning on the job

Developers new to software applications typically require 6-12 months of on-the-job learning to become productive, depending on the size of the source code base. To become proficient, programmers may need up to 3 years.

Without qualified software developers knowledgeable about your applications, you endanger your business's operations, reputation, and security. You also risk a significant decrease in your software teams' productivity and efficiency.

Consider the monumental task confronting newly hired or transferred IRS developers last March. Congress passed the CARES Act on March 25, 2020, and then Treasury Secretary Steven Mnuchin announced that individual stimulus checks would be mailed in early April.

To assist with the delivery of economic stimulus payments, the new developers were required to immediately start working with the agency's source code base, which, in 2019, was estimated at nearly 20 million lines of code and includes over 60 years of legislative and system changes. As of mid-May 2020, nearly 20 million people had not received their stimulus checks, and some recipients had problems throughout the year.

These engineers didn’t have 6-12 months to become productive. They had to hit the ground running on day one. And without the benefit of weeks or months of training and on-the-job learning, they didn’t have the application knowledge necessary to understand how even simple changes could affect entire applications.

And let's not forget the productivity loss due to the remaining IRS's experienced engineers for training, mentoring, and supervising the new recruits.

What’s your risk?

Your situation may not be as dire as what the IRS faced – for now. But how much time can you really give your new developers to learn the system, and how much productivity can you afford to lose while your experienced programmers train and supervise them?

How much do you trust the developers that have just started working on your critical applications?

How confident will you be when your CEO or Board of Directors asks for assurances that the next customer-facing application update will not result in outages and lost revenue – especially if the update was programmed by a developer you hired just weeks ago?

Your software is a critical part of your organization, especially if you rely on legacy mainframe systems. You must have a plan or tool that keeps the code running and bridges the gap between retiring and departing developers and the people that will replace them.

Steve Brothers is the President of Phase Change Software. You can reach him on LinkedIn or at sbrothers@phasechange.ai.