All Posts in Applied

April 24, 2019 - Comments Off on The cost of fixing COBOL bugs

The cost of fixing COBOL bugs

April 24, 2019

by Greg Brueggeman*

I began working with mainframe programming languages in 1990 while I was in college. I started with FORTRAN and assembly languages, and I knew about COBOL but wasn’t exposed to it until much later.

While FORTRAN and assembly languages now have their niche uses, COBOL is entrenched as a vital part of the global economy. And now the cost of maintaining the nearly 60-year-old language is rising precipitously.

A little history

The Common Business Oriented Language (COBOL) was developed as a stop-gap measure during the second Eisenhower administration to create a portable mainframe programming language for the Department of Defense. It was based on the FLOW-MATIC compiler, which was recognized as the first English language data-processor compiler and was designed by Rear Adm. Grace Hopper.

COBOL was adopted by the business world starting in 1960, and because of its simplicity and reliability, COBOL-based applications remain entrenched in mainframe-dependent industries such as government and finance. The Social Security Administration (SSA) and Internal Revenue Service (IRS) rely on approximately 110 million lines of COBOL code combined daily. An estimated $3 trillion a day and 90% of all ATM and in-person financial transactions are handled by COBOL-supported systems.

I predict that the last mainframe will be unplugged on March 15, 1996.
~Stewart Alsop II, InfoWorld magazine, 1991

The bad news is that some of these applications are nearing 50 years in age. And although COBOL remains vital to many critical systems, this prominence has not resulted in a stable supporting workforce. The population of experienced COBOL developers declines at least 5% every year because of retirement. The population’s average age is roughly 55 years old, and the language’s absolute uncoolness has motivated the majority of university computer science programs to drop their COBOL classes entirely. The trickle of newly trained COBOL programmers are coming from community colleges, technical schools, and programs run by private companies such as IBM.

The continuing importance of COBOL-based applications and the diminishing trained workforce is causing the cost of maintaining COBOL-based applications to rise.

How often do defects arise?

You might ask, “Well, if COBOL is so reliable, why do we need lots of programmers? Fewer bugs naturally means fewer engineers.”

That sound perfectly logical, however, in my experience, due to the lack of a sufficient supporting workforce and the spaghettified nature of today’s COBOL-based applications, they are not modified and supported as well as other applications programmed in more popular languages.

Think about all the patches, additions, and technical debt that have built-up since these applications began surfacing in 1960, as well how many developers contributed to those applications but are no longer around to address issues and answer questions.

Due to the uncertainty of downstream impact and to reduce the risk of breaking critical applications, I believe many COBOL engineers duplicate hundreds or thousands of lines-of-code when making changes or repairs because they do not completely comprehend the COBOL code in their system. As a result, changes are made, tested, and implemented in an environment of uncertainty and heightened risk.

And when the flaw is discovered, the repair cost includes:

  • Consultants or outside IT contractors
  • The development team’s time away from building new features and products
  • Salaries of internal personnel tasked with fixing the bug or supporting the project
  • Application downtime
  • Lost business opportunities

The Y2K scare is just one example. While it ended up being much less daunting than many estimates, roughly $320 billion was spent worldwide evaluating and fixing systems.

But the price of fixing COBOL code includes more than just repairs and downtime. An organization’s security, reputation and market position can be affected by one time-bomb defect. When an August 1, 2012, software failure caused Knight Capital LLC to create thousands of trades per second on the New York Stock Exchange, the company lost between $440-460 million in 45 minutes. By the next day, Knight’s stock had fallen 75% and within the year the company was acquired by a rival.

What is the cost of fixing COBOL code?

So, what is the actual cost of fixing COBOL code?

For the sake of simplicity, let's analyze the cost based on salaries and time spent supporting COBOL-based applications, but not developing new programs.

Here are our assumptions:

If we conservatively estimate the cost of maintaining COBOL-based applications by only including the developers working solely on maintenance, it’s roughly $1.02 billion per year. Here’s how we get there.

20,000 COBOL developers
x 42% solely doing maintenance
x $81,000 per year                           
= $680.4 million per year

In my experience, you have to add another 50% of the development cost for quality assurance (QA) testing, which would bring the total to $1.02 billion just in the U.S.

$680.4 million
+ $340.2 million for QA testing     
= $1.02 billion per year

If anything, these numbers are low because we didn’t consider the companies doing both maintenance and new development in COBOL.

The cost of fixing COBOL v. alternatives

Another approach to handling applications written in legacy programming languages is to modernize the code – translate the COBOL source code to a modern language such as Java.

Despite the astronomical maintenance costs, it doesn’t necessarily make sense to modernize COBOL-based applications. Hundreds of millions of dollars are spent each year on COBOL transformation projects, and more than a few have ultimately failed.

The risks associated with a COBOL transformation project are significant. Dave Brown, Systems Architect at The Bank of New York Mellon, said in 2012 that his bank had 343 million lines of COBOL code. Transforming a code base of that size and complexity would take years and hundreds of millions of dollars.

Just ask the executives at the Commonwealth Bank of Australia, which undertook a planned AU $580 million (US $413 million) legacy core banking technology transformation in April 2008. The project was finally completed in August 2013 at a final cost of AU $1.1 billion (US $783.871 million) – almost two years behind schedule and $370 million over budget. And this was a successful legacy system conversion.

The COBOL programming language is not destined for retirement. Organizations can plan on continuing to spend a high percentage of their IT budgets maintaining legacy systems, especially in the financial and government sectors. An April 2017 article by claims that the federal government spent roughly $90 billion in 2017 maintaining legacy systems, which was roughly 80% of its entire IT budget ($90 billion in 2017).

Until new technology comes along to enable organizations to better understand their COBOL-based applications, unravel their complex and spaghettified code, and extract the embedded technical, business, and regulatory knowledge buried within, this trend will continue. Whichever approach you take, right now the only solution is to throw time and money at the problem.

*Greg Brueggeman is the Director of Product Management at Phase Change Software. You can reach him at

February 11, 2019 - Comments Off on Phase Change CEO Steve Bucuvalas featured on the InfluenceNow! podcast

Phase Change CEO Steve Bucuvalas featured on the InfluenceNow! podcast

February 7, 2019

by Todd Erickson1

Phase Change’s Inventor, Founder, and CEO, Steve Bucuvalas, was featured in the January 31, 2019, episode of the InfluenceNow! podcast, hosted by Justin Craft2.

The InfluenceNow! podcast highlights startups, exceptional business influencers, and ideas from a variety of industries that influence the world.

Steve and Justin discussed how Phase Change and the technology behind Mia, the first cognitive agent for software development, became a reality.

The interview begins with Steve describing his career leading technology and artificial intelligence (AI) groups in financial services and insurance companies, and his subsequent entrepreneurial career starting and selling two different companies. He tells the story of how a single conversation with the buyer of his second company led to his interest in applying AI technology to the problem of software-development productivity.

At the closing, the buyer said to me, 'What's wrong with you guys in software? AI has changed financial services extraordinarily - increased our productivity 100 times,' which is accurate. 'Why can’t you do that with your own industry?'

That moment led Steve to research the barriers to applying AI to software development, and the development of the human-centric principles that led to the creation of the Mia cognitive agent.

The podcast continues with Steve and Justin discussing why organizations that rely on applications written in the Common Business-oriented Language (COBOL) programming language are Phase Change’s first target market.

COBOL is this 40-50 year-old language that has atrocious legacy problems. Because the code has been around [so long], it runs 85% of the world’s financial transactions and [there’s] 220 billion lines of [active COBOL] code. The programmers are all in their 60’s and they all want to retire, but they keep getting incentives to work a few more years because no one wants to learn COBOL. In fact, some of the kids in computer science [college courses] have never heard of it.

Justin and Steve conclude the interview discussing the productivity gains realized by Mia and Phase Change’s technology, and when it will be generally available.

To learn more about how Steve and Phase Change Software will radically improve software productivity, watch the podcast video below or listen to the audio podcast.

1Todd Erickson is a tech writer with Phase Change Software. You can reach him at
2Justin Craft is the Founder and CEO of Cast Influence, a Denver, Colorado,-based turnkey marketing agency. Phase Change Software is a client of Cast Influence.

August 4, 2017 - Comments Off on Phase Change enables market adaptability through impact analysis

Phase Change enables market adaptability through impact analysis

August 4, 2017

by Todd Erickson

Gary Brach, Ken Hei, and Brad Cleavenger discuss how Phase Change's assistive AI removes the doubt associated with changing software applications.

Changing software is difficult and expensive, and it can be a major stumbling block to business innovation.

Phase Change's assistive AI will enable software teams to quickly and fearlessly address market opportunities by rapidly assessing the scope and viability of proposed software modifications, and then efficiently making changes without adding the technical debt that reduces system performance and application life span.

Todd Erickson is a tech writer with Phase Change. You can reach him at

July 10, 2017 - Comments Off on Phase Change will bridge application knowledge silos

Phase Change will bridge application knowledge silos

July 10, 2017

by Todd Erickson

Members of Phase Change's management team address how our technology will bring together an organization's siloed application knowledge to enable faster responses to market demands.

It's a paradox. Your most successful applications get larger and more complex with updates, upgrades, and new features until they become difficult to change and adapt. Now they are hard-to-manage legacy systems that cost ever more time and money to remain valuable.

One of the main reasons applications become difficult to maintain is that knowledge silos emerge – where various people in development and other departments understand small portions of the code, but no one person knows the entire code base.

Then when you bring people together to develop new features that will address market demands or opportunities, each contributor only knows his or her portion of the application code, each person has his or her own mental model of the code, and all of that knowledge is difficult to share.

Learn how Phase Change's assistive AI agent will bridge knowledge silos by understanding the entire code base, presenting a complete and accurate model, and collaborating with engineers and stakeholders.

Todd Erickson is a tech writer with Phase Change. You can reach him at

June 26, 2017 - Comments Off on Understanding code is the key to software development

Understanding code is the key to software development

June 26, 2017

By Elizabeth Richards and Todd Erickson

Discover how software developers are like archaeologists, and why understanding source code involves a lot more than digging.

Software engineers are often called developers. However, according to a number of experienced programmers, they spend the great majority of their time (78%) simply searching and understanding existing software, and the rest of their time (22%) actually modifying legacy software or developing new applications.

Programmers are forced to spend so much time finding and comprehending code because current tools and techniques are not technically savvy, and they are too focused on the searching process. They don't help developers understand how the code works together within the systems they serve.

In fact, a recent blog post compared the source-code searching process to archeology. It's a reasonable analogy given that the tools developers use are only slightly more advanced than rudimentary shovels and brushes.

Why is software development – an activity that's driving incredible technological change – so far behind the curve in building tools that help programmers comprehend the code they work on?


Searching for ancient relics and specific lines of code are both complex processes.

An archeologist doesn't use a bulldozer and dig in random locations. She researches excavation sites and uses advanced technology, such as satellite imagery to find optimum exploration locales and ground-penetrating radar for mapping. Then she carefully and methodically removes topsoil while analyzing and recording each artifact down to the smallest pottery shard.

Every relic she unearths builds her knowledge of the site, and the people and culture she's investigating. For example, the archaeologist may assemble a handful of pottery shards into a serving dish, which she studies alongside other artifacts to better understand an ancient culture's family meal rituals. She can easily share the dish with other scholars and store it for future analysis.

Each discovered artifact may also modify how she approaches the rest of the dig.


In software development, Professor Vaclav Rajlich asserts that the processes of searching and understanding code require two phases called concept location and impact analysis. Concept location involves finding the lines of code to be modified and the relevant but disjoined source surrounding it.

Impact analysis examines how a proposed modification will impact the entire application, including performance, stability, intent, and secondary consequences in distant modules. Poor or incomplete impact analysis can lead to more bugs.

In essence, a developer spends his time in active knowledge construction, building an integrated mental model so he understands how a system is constructed and its purpose and intended results. Only then can he be confident enough to make changes.

Errors logs, debuggers, and grepping assist developers in finding specific lines of code – his shards of pottery. But the developer must reconstruct the code into the mental models necessary for understanding the system. And these models remain locked away in the developer's mind, making them difficult to share and retrieve over time.

The greatest shovel ever invented

But it doesn’t have to be that way. Phase Change’s technology will transform how developers search and understand source code and applications. A programmer will no longer have to manually and laboriously search and play connect-the-dots to build mental models.

We are developing assistive artificial intelligence (AI) that automatically analyzes source code and understands the human intent behind it, creating immersive application-visualizations that resemble a programmer’s mental models. These visualizations can be stored, retrieved, and shared similar to how an archaeologist saves and shares the artifacts she assembles.

A developer will collaborate with our AI agent using natural language to effortlessly locate source and move quickly beyond simply identifying concept locations to performing comprehensive impact analysis. He will be aware of every effect his modifications bring about, and work confidently with a rich understanding of the code and the application.

Because the goal isn't to search, it's to understand.

learn more about our technology



Elizabeth Richards is Phase Change's director of business operations. You can reach her at
Todd Erickson is a tech writer with Phase Change. You can reach him at

May 25, 2017 - Comments Off on Why is Cobol cool again? – blog

Why is Cobol cool again? – blog

May 25, 2017

by Todd Erickson and Elizabeth Richards

Discover why the recent spotlight on Cobol systems and the shortage of qualified Cobol programmers isn’t due to a lack of qualified engineers, it's due to a lack of knowledge.

Reuters and The New Stack recently published articles about Cobol, an often-overlooked programming language that was developed before John F. Kennedy became the 35th President of the United States.

At Phase Change, we pay attention to legacy systems and their challenges. So, why was a mainframe language developed in 1959 suddenly the topic of multiple news articles?

The U.S. government developed the common business oriented language (Cobol) in conjunction with Rear Admiral Grace Hopper and a coalition of industry and higher-education envoys. It's simplicity and portability have stood the test of time, and are the main reasons why 50-year-old Cobol applications continue to play a critical role in finance, banking, and government operations. That plus the inertia that characterizes large, critical systems.

Organizations like the Department of Veteran Affairs and large financial companies, such as Bank of New York Mellon and Barclays PLC, are examples of the types of institutions that rely on Cobol applications for nearly $3 trillion worth of daily transactions. But they’ve used Cobol for decades, so, that doesn't explain the recent attention.

It's because the engineers that maintain Cobol-based systems are leaving the workforce, there aren't qualified developers available to replace them, and these institutions are freaking out. The Cobol brain drain is threatening the organizations that economies are built upon.

Brain drain refers to how departing software engineers leave with all of their system and domain knowledge supposedly locked away in their brains. That knowledge is thought to be lost from the organization forever.

The average age of a Cobol programmer is somewhere between 45 and 60 years old and they are retiring. The problem is that few programmers are interested in replacing them, and the availability of Cobol training resources has dropped precipitously because it's just not a cool language anymore.

We won't repeat all of the statistics that show how much Cobol code is still in use and how important those systems are. Read the Reuters and The New Stack articles, which both mirror a series of comprehensive feature articles published by ComputerWorld in 2012. The metrics and themes haven’t changed much.

You can also follow the "official" Cobol Twitter account, @morecobol (spoiler: it's clever).

Basically, these companies have three options to deal with Cobol brain drain, and all involve high risks. First, they can simply replace their Cobol systems with systems built on more modern programming languages. That project took the Commonwealth Bank of Australia 5 years and $749.9 million, which was 30% over budget. The risk associated with implementing such a massive new system has kept most financial institutions from doing it.

Second, they can engage consultants like the Cobol Cowboys, or hire and train new programmers to support their Cobol systems. This option also involves a great amount of risk because companies have to find engineers that have the skills and interest to support Cobol applications, and then hope they can unravel the layers of modifications and system integrations that accrue with five decades of maintenance, usually with little documentation.

Third, they can completely stop modifying core systems that nobody understands, but are too critical to risk changing or replacing. The USDA faced that choice.

It's not a people problem

But from our perspective, the issue is not a human-resources problem. The companies that rely on Cobol-based systems don't lack the right people, they lack the right knowledge.

If the new engineers assigned to work on Cobol-based applications could access the departing developers' system and domain knowledge, or better yet, all of the programming and domain knowledge imbued into the system from prior engineers, imagine how much easier it would be for them to comprehend these complex systems. It would be like having a personal mentor always available — even while the previous engineers are off enjoying retirement.

That's why this is a knowledge problem and not a people problem.

It's a huge opportunity for someone that can reach all of that trapped knowledge and make it easily comprehensible.

Exploiting the knowledge left behind

Phase Change's objective aim is to use our assistive AI technology to unlock all of the trapped programming and domain knowledge – as well as the human intent behind it – inside software applications, no matter which programming languages were used to create them, and make it easy to access that knowledge with natural-language interaction.

Engineers and stakeholders will literally talk to their software applications to reveal the hidden encoded knowledge they require to comprehend the overwhelming scale and complexity resulting from decades of modifications and system mergers, and hundreds of contributing developers.

Unlocking the encoded knowledge that's trapped in Cobol system will give these large institutions the knowledge they need to make informed decisions about their legacy systems.

learn more about our technology



Todd Erickson is a tech writer with Phase Change. You can reach him at
Elizabeth Richards is Phase Change's director of business operations. You can reach her at

May 8, 2017 - Comments Off on Phase Change creates scale-free software development – video

Phase Change creates scale-free software development – video

May 8, 2017

Learn how Phase Change's assistive AI creates scale-free software engineering and enables the development team to swiftly respond to market demands.

As software systems grow in size and complexity, they can easily become incomprehensible for individual engineers. They simply get too large and sophisticated for one person to fully understand. As more people are required to comprehend and maintain complex systems, the organization's ability to modify those systems and respond to changing market dynamics diminishes.

Watch President Gary Brach, Director of Engineering Ken Hei, and Senior Software Architect Brad Cleavenger, discuss how system scale affects the ability to modify applications and meet market demands, and how Phase change's assistive AI minimizes scale issues to create scale-free software.

April 24, 2017 - Comments Off on Why Phase Change will fundamentally change software development – video

Why Phase Change will fundamentally change software development – video

April 24, 2017

Gary Brach, Ken Hei, and Brad Cleavenger discuss how Phase Change's assistive AI technology will fundamentally change how software is developed so organizations can quickly and confidently respond to changing market dynamics.

While transformative advances in automation, communications' networking, and computer processing in the last 20 years have vastly improved business operations, the same cannot be said for software development.

The process of developing the applications that now run our daily lives hasn't significantly changed since the 1970s.

Sure, we've developed better tools and better ways of communicating with one another during the development process – such agile development techniques – but the underlying software development activities are the same.

This lack of substantial improvement makes it difficult for organizations to quickly respond to changing market dynamics.

However, the future of software development is bright. Organizations will soon be able to quickly and confidently respond to changing market dynamics.

Phase Change's technology will fundamentally transform how software is developed by introducing our assistive AI into the process – enabling organizations to quickly respond to market changes and opportunities.

Watch the following video below to learn why Gary Brach, Ken Hei, and Brad Cleavenger believe Phase Change's technology will fundamentally change the software development process.

April 10, 2017 - Comments Off on Prevent software application knowledge from walking out the door – blog

Prevent software application knowledge from walking out the door – blog

April 10, 2017

by Todd Erickson, Tech Writer

Brain drain is a serious problem facing organizations that use software applications to run their businesses. Learn how you can seal the drain and retain all of the knowledge trapped in your applications.

At the end of every workday, your software development teams walk out the door with all of their knowledge leaving with them. Some of them don’t come back, and that loss of information and expertise, or brain drain, is a growing business problem, especially with IT industry turnover rates hovering between 20-30% annually.

Consider how much knowledge your organization loses when key members of your development team retire or join other companies. Not only do you lose development expertise, but the knowledge your engineers have regarding how your software applications work, such as:

  • How the system is architected
  • The subject-matter expertise used to implement functionality
  • The business considerations that drove product and feature designs
  • How third-party and external systems are integrated

The plight of developing and supporting older and large-scale applications is exacerbated when companies have to scramble to replace retiring software engineers with unqualified replacements. Multiple reports suggest that 10,000 Baby Boomers walk out the corporate door in the U.S. for good every day.

Many of these retirees are the software engineers that developed and maintain the many systems that still run on Cobol and other mainframe programming languages. The impact of losing thousands of mainframe engineers and their vast programming and business knowledge will be widespread. The 240 billion lines of Cobol code running today power approximately 85 percent of all daily business transactions worldwide.

Most organizations don't have the processes in place to capture their employees' business and system intelligence before they leave for good.

It’s especially difficult for engineers. Today’s software tools don't allow them to easily convey their expertise to others – or enable developers, business managers, and executives to easily discover and utilize any previously shared knowledge.

What can you do?

You might be surprised to discover that your engineers’ domain and system knowledge already resides in one other place outside their minds – your software. While creating the code, development teams pour their organization, programming, and business intelligence into your applications.

Imagine what you could do if your organization's technical and business stakeholders had access to all of the knowledge and human intent embedded in your software applications. Imagine asking your software application how it works and having it answer you back.

How can you unlock all of that untapped knowledge?

Liberate encoded knowledge

Phase Change Software is creating AI-assistive technology that unlocks the encoded knowledge embedded in your software applications.

Our assistive AI understands your software and turns it into formal units of knowledge. In essence, software is transformed into data.

Our AI assistant will liberate your software's hidden knowledge and help it understand itself. Our natural language processing (NLP) techniques will enable your technical and business stakeholders to easily interact with applications.

You will soon be able to literally have a conversation with your software, and have it teach you its encoded programming, business, and domain knowledge.

learn more about our technology



Todd Erickson is a tech writer with Phase Change Software. You can reach him at

January 5, 2017 - Comments Off on How Phase Change’s AI impacts release management — video

How Phase Change’s AI impacts release management — video

January 5, 2017

Phase Change President Gary Brach leads a practical discussion with Ken Hei, director of engineering, and Brad Cleavenger, senior software architect, about how Phase Change's technology will transform release management.