May 21, 2020

Education is vital to legacy applications’ future – podcast

May 21, 2020

By Todd Erickson

Educating young developers about the importance of legacy software applications, and building the tools needed to connect them with modern technologies are the keys to combining old-school reliability and new-school engineering say Bill and Eileen Hinshaw of COBOL Cowboys.

If you've followed the stories about the computer-system meltdowns brought about by the overwhelming demand for government financial assistance during the COVID-19 pandemic, then you've probably read about COBOL Cowboys.

The Hinshaw's founded the company to bring together experienced programmers and organizations that lack the expertise needed to fix and maintain their legacy applications.

When the system failures started, government officials were quick to blame their back-end mainframe applications, just as they did during the Y2K crisis. However, those same officials were forced to backtrack when Bill and others revealed that the mainframe applications were fine – it was the agency's infrastructure and front-end systems that caused the problems.

Bill and Eileen have done a number of interviews about the system failures, but now they want to talk about moving forward to ensure that these legacy systems are updated using modern technologies, so they don't get blamed for the next computer catastrophe.

That's where education and better tools come into play. Bill and Eileen say the good that's come from our current situation has been the increased public and industry awareness of how important legacy systems are to industries and companies around the world.

Their goal is to educate young software developers on the advantages of mainframe systems so more programmers will be interested in working with them and will replenish the declining workforce.

Learn more about the COBOL Cowboys and how critical mainframe applications are to the world in this Phase Change podcast.

Todd Erickson is a Technology Writer at Phase Change Software. You can reach him at terickson@phasechange.ai.

April 24, 2019

The cost of fixing COBOL bugs

April 24, 2019

by Greg Brueggeman

COBOL maintenance costs are rising as aging developers leave the workforce and well-trained replacements are in short supply. Learn about how pervasive COBOL-based applications remain, why the supporting developer workforce is shrinking every year, and scrutinize our estimates of how high these applications' maintenance costs are rising.

I began working with mainframe programming languages in 1990 while I was in college. I started with FORTRAN and assembly languages, and I knew about COBOL but wasn’t exposed to it until much later.

While FORTRAN and assembly languages now have their niche uses, COBOL is entrenched as a vital part of the global economy. And now the cost of maintaining the nearly 60-year-old language is rising precipitously.

A little history

The Common Business Oriented Language (COBOL) was developed as a stop-gap measure during the second Eisenhower administration to create a portable mainframe programming language for the Department of Defense. It was based on the FLOW-MATIC compiler, which was recognized as the first English language data-processor compiler and was designed by Rear Adm. Grace Hopper.

I predict that the last mainframe will be unplugged on March 15, 1996.
~Stewart Alsop II, InfoWorld magazine, 1991

The bad news is that some of these applications are nearing 50 years in age. And although COBOL remains vital to many critical systems, this prominence has not resulted in a stable supporting workforce. The population of experienced COBOL developers declines at least 5% every year because of retirement. The population’s average age is roughly 55 years old, and the language’s absolute uncoolness has motivated the majority of university computer science programs to drop their COBOL classes entirely. The trickle of newly trained COBOL programmers are coming from community colleges, technical schools, and programs run by private companies such as IBM.

The continuing importance of COBOL-based applications and the diminishing trained workforce is causing the cost of maintaining COBOL-based applications to rise.

How often do defects arise?

You might ask, “Well, if COBOL is so reliable, why do we need lots of programmers? Fewer bugs naturally means fewer engineers.”

That sound perfectly logical, however, in my experience, due to the lack of a sufficient supporting workforce and the spaghettified nature of today’s COBOL-based applications, they are not modified and supported as well as other applications programmed in more popular languages.

Think about all the patches, additions, and technical debt that have built-up since these applications began surfacing in 1960, as well how many developers contributed to those applications but are no longer around to address issues and answer questions.

Due to the uncertainty of downstream impact and to reduce the risk of breaking critical applications, I believe many COBOL engineers duplicate hundreds or thousands of lines-of-code when making changes or repairs because they do not completely comprehend the COBOL code in their system. As a result, changes are made, tested, and implemented in an environment of uncertainty and heightened risk.

And when the flaw is discovered, the repair cost includes:

  • Consultants or outside IT contractors
  • The development team’s time away from building new features and products
  • Salaries of internal personnel tasked with fixing the bug or supporting the project
  • Application downtime
  • Lost business opportunities

The Y2K scare is just one example. While it ended up being much less daunting than many estimates, roughly $320 billion was spent worldwide evaluating and fixing systems.

But the price of fixing COBOL code includes more than just repairs and downtime. An organization’s security, reputation and market position can be affected by one time-bomb defect. When an August 1, 2012, software failure caused Knight Capital LLC to create thousands of trades per second on the New York Stock Exchange, the company lost between $440-460 million in 45 minutes. By the next day, Knight’s stock had fallen 75% and within the year the company was acquired by a rival.

What is the cost of fixing COBOL code?

So, what is the actual cost of fixing COBOL code?

For the sake of simplicity, let's analyze the cost based on salaries and time spent supporting COBOL-based applications, but not developing new programs.

Here are our assumptions:

If we conservatively estimate the cost of maintaining COBOL-based applications by only including the developers working solely on maintenance, it’s roughly $1.02 billion per year. Here’s how we get there.

20,000 COBOL developers
x 42% solely doing maintenance
x $81,000 per year                           
= $680.4 million per year

In my experience, you have to add another 50% of the development cost for quality assurance (QA) testing, which would bring the total to $1.02 billion just in the U.S.

$680.4 million
+ $340.2 million for QA testing     
= $1.02 billion per year

If anything, these numbers are low because we didn’t consider the companies doing both maintenance and new development in COBOL.

The cost of fixing COBOL v. alternatives

Another approach to handling applications written in legacy programming languages is to modernize the code – translate the COBOL source code to a modern language such as Java.

Despite the astronomical maintenance costs, it doesn’t necessarily make sense to modernize COBOL-based applications. Hundreds of millions of dollars are spent each year on COBOL transformation projects, and more than a few have ultimately failed.

The risks associated with a COBOL transformation project are significant. Dave Brown, Systems Architect at The Bank of New York Mellon, said in 2012 that his bank had 343 million lines of COBOL code. Transforming a code base of that size and complexity would take years and hundreds of millions of dollars.

Just ask the executives at the Commonwealth Bank of Australia, which undertook a planned AU $580 million (US $413 million) legacy core banking technology transformation in April 2008. The project was finally completed in August 2013 at a final cost of AU $1.1 billion (US $783.871 million) – almost two years behind schedule and $370 million over budget. And this was a successful legacy system conversion.

The COBOL programming language is not destined for retirement. Organizations can plan on continuing to spend a high percentage of their IT budgets maintaining legacy systems, especially in the financial and government sectors. An April 2017 article by NextGov.com claims that the federal government spent roughly $90 billion in 2017 maintaining legacy systems, which was roughly 80% of its entire IT budget ($90 billion in 2017).

Until new technology comes along to enable organizations to better understand their COBOL-based applications, unravel their complex and spaghettified code, and extract the embedded technical, business, and regulatory knowledge buried within, this trend will continue. Whichever approach you take, right now the only solution is to throw time and money at the problem.

*Greg Brueggeman is the Director of Product Management at Phase Change Software. You can reach him at gbrueggeman@phasechange.ai.

Contact

651 Corporate Circle
Suite 209A
Golden, Colorado 80401
Phone: +1.303.586.8900
Email: info@phasechange.ai

© 2024 Phase Change Software, LLC