Featured Image

October 6, 2021

COBOL defects’ paper coauthored by Phase Change scientists wins IEEE distinguished paper award

tsantalis tweet congratulating paper winnersA technical paper co-authored by current and former Phase Change research scientists, and presented at the 2021 annual ICSME event, won a Distinguished Paper Award from the IEEE Computer Society Technical Council on Software Engineering (TCSE).

The paper, "Contemporary COBOL: Developers' Perspectives on Defects and Defect Location," was co-authored by current Phase Change Senior Research Scientist Rahul Pandita, former Senior Research Scientist Aleksander Chakarov, and former intern Agnieszka Ciborowska. It was presented at the 37th annual International Conference on Software Maintenance and Evolution (ICSME) 2021 in Luxembourg City, Great Duchy of Luxembourg, September 27 - October 1.

The authors presented results from surveys of COBOL and more modern programming languages regarding defects and defect-location strategies. While the software industry has made substantial advances in maintaining programs written in modern languages, mainframe programs have received limited attention. Meanwhile, mainframe systems face a critical shortage of experienced developers and replacement developers face significant difficulties even during routine maintenance tasks.

pandita tweet announcing awardPandita, who has already co-authored a number of published papers, said that this award is particularly gratifying because all of the authors were working together at Phase Change when it was written, and that he hopes it is just the first of many more like it.

This is the fourth published technical paper co-authored by Phase Change scientists, the third to be presented at scientific conferences, and the second to win a distinguished paper award.

Published by Phase Change Staff in Community, Company, Science
Tags: cobol, defect location, defects, distinguished award, icsme, ieee, rahul pandita, technical paper, technical papers
Featured Image

July 20, 2020

Researchers earn distinguished paper award with Phase Change help

July 20, 2020

by Todd Erickson

A team of Oregon State University scientists partnered with Phase Change Research Scientist Rahul Pandita to study how cognitive biases affect software developers' everyday behavior. The resulting academic paper, "A Tale from the Trenches: Cognitive Biases and Software Development," was recently recognized by ICSE 2020 as an ACM SIGSOFT Distinguished Paper.

According to OpenResearch.org and ACM SIGSOFT, only 2% of all ICSE submissions earn Distinguished Paper Awards.

OSU researchers Nicholas Nelson and Anita Sarma enjoying time in Phase Change's offices.

"Bias is an essential tool for human cognition," said Rahul Pandita. "The presence of bias must not be automatically equated to something negative. In fact, some biases are extremely helpful in navigating the complexities of day to day life. The key is to understand how these biases operate. In the case of routine software development activities, such nuanced understanding allows us to develop effective intelligence augmentation (IA) technology to amplify the benefits of such biases and counter the detrimental effects."

The scientists conducted a two-part study from 2017-2018. Part one focused on observing Phase Change developers performing routine development tasks. They observed Phase Change developers at our offices for a week in March 2018.

“Getting to see the 'behind the scene' workings of this agile, innovative team was a great way of understanding how startups work," said Anita Sarma, an Associate Professor at Oregon State.

Part two involved triangulating their findings by interviewing developers from three other companies about how they perceive and deal with the observed biases found in Part One.

Research Scientists Anita Sarma, Nicholas Nelson, Souti Chattopadhyay, and Christopher Sanchez, along with research interns Audrey Au and Natalie Morales, co-wrote the paper with Pandita.

The research results were presented at ICSE 2020, the 42nd annual International Conference on Software Engineering, July 6-11 in Seoul, South Korea, and virtually from June 27-July 19. All of the Distinguished Papers were announced during a July 10 awards ceremony and appeared on other slides shown throughout the conference.

Todd Erickson is a Technology Writer at Phase Change Software. You can reach him at [email protected].

Published by Phase Change Staff in Community, Science
Tags: community, ICSE, rahul pandita, research, technical papers
Featured Image

February 4, 2019

IEEE magazine publishes Phase Change research scientist co-authored paper

Phase Change research scientist Rahul Pandita’s co-written paper, “A Conceptual Framework for Engineering Chatbots,” was recently published in the November-December 2018 issue of IEEE Internet Computing.

The industry magazine is published bi-monthly by the Institute for Electrical and Electronics Engineers (IEEE) Computer Society for evaluating and reviewing Internet-based computer applications and enabling technologies. It focuses on technologies and applications that enable practitioners to utilize Internet-based applications and tools, instead of having to build their own.

The paper

The use of chatbots as virtual assistants is becoming more widespread as companies strive to increase community engagement online and on social-media platforms.

The problem is that most commercially available bots are engineered with If-This-Then-That (IFTTT) frameworks from the 1980s. These decades-old frameworks often create inflexible chatbots that are difficult to maintain.

The bots can be monolithic and may mix dialog-managing rules with business-execution logic and response-generation rules. And when these chatbots must interact with third-party services to orchestrate workflows, the orchestration logic becomes entwined with the IFTTT rules.

Additionally, IFTTT tends to be order sensitive. As chatbots’ capabilities increase, their implementation rules grow more complex, and even simple modifications can require substantial effort.

The paper, “A Conceptual Framework for Engineering Chatbots,“ outlines a high-level conceptual framework founded upon agent-oriented abstractions – goals, plans, and commitments.

It theorizes that well-studied abstractions of goals and commitments from the area of artificial intelligence (AI) and multiagent systems allow for more flexible chatbots. Goals capture an agent’s intentions, and commitments capture meaningful business relationships between agents.

The paper describes how employing goals and commitments can enable a model chatbot that can be verified at design time or runtime, offers flexible enactments, and provides a basis for judging correctness.

Authors

In addition to Pandita, the paper is written by:

It is available free online for IEEE members, and can be purchased through the IEEE Xplore Digital Library.

^The figure represented in the featured image and the IEEE Internet Computing magazine cover are copyrighted by the Institute of Electrical and Electronics Engineers Inc..

Published by Phase Change Staff in Community, Technology
Tags: chatbots, natural language, natural language processing, NLP, published paper, rahul pandita, technical paper
Featured Image

July 17, 2018

Phase Change research scientist publishes technical papers in prominent research journals

July 16, 2018

by Rahul Pandita and Todd Erickson

Phase Change research scientist Dr. Rahul Pandita recently had two co-written papers published in well-known research journals. The first paper, “Are vulnerabilities discovered and resolved like other defects?,” was published in the June 2018 volume of the Empirical Software Engineering: An International Journal and presented as a Journal First Paper at the 40th International Conference on Software Engineering (ICSE) in Gothenburg, Sweden.

The paper was co-written with Patrick Morrison, Dr. Xusheng Xiao, Dr. Ram Chillarege, and Dr. Laurie Williams. Patrick Morrison is a Ph.D. candidate in the Computer Science Department at North Carolina State University. Dr. Xusheng Xiao is an assistant professor in the Department of Electrical Engineering and Computer Science at Case Western University.

Dr. Ram Chillarege is the founder and president of Chillarege Inc. Dr. Laurie Williams is a professor, and the department head, at the North Carolina State University Department of Computer Science.

The paper

The goal of the project’s research was to determine if security defects (referred to as vulnerabilities in the paper) are discovered and resolved by different software-development practices in comparison to non-security defects. If true, technical leaders could use the distinction to drive security-specific software development process improvements.

The research consisted of extending Orthogonal Defect Classification (ODC), which is a well-established scheme for classifying software defects, to study process-related differences between vulnerabilities and non-security defects, and thereby creating ODC + Vulnerabilities (ODC+V). This new classification was applied to 583 vulnerabilities and 583 defects across 133 releases of three open-source projects – the Firefox web browser, phpMyAdmin, and Google’s Chrome web browser.

The study found that compared with non-security defects, vulnerabilities are found much later in the development cycle and are more likely to be resolved through changes to conditional logic. The results indicate opportunities may exist for more efficient vulnerability detection and resolution.

The paper was accepted by the 40th International Conference on Software Engineering (ICSE) that was held in Gothenburg Sweden, between May 27 and June 3, as part of the *ICSE 2018* Journal First Papers track. Dr. Williams presented it on May 31, 2018.

But wait, there’s more

The second paper, “Mapping the field of software life cycle security measures,” is scheduled to be published in the October 2018 issue of Information and Software Technology. It was co-written with Patrick Morrison, Dr. Laurie Williams, and David Moye, a program site lead with Aelius Exploration Technologies LLC.

The authors suspected that a catalog of software-development life cycle security metrics could assist practitioners in choosing appropriate metrics, and researchers in identifying opportunities for security measurement refinement.

They conducted a systematic mapping study, beginning with 4,818 papers and focusing on 71 papers reporting on 324 unique security metrics. For each metric, the researchers identified the subject being measured, how the metric had been validated, and how the metric was used. Then they categorized the metrics and included examples of metrics for each category.

The research found that approximately 85% of the security metrics studied were proposed and evaluated solely by their authors, leaving room for replication and confirmation through field studies. Approximately 60% of the metrics were empirically evaluated by their authors or others.

They concluded that the primary application of security metrics to the software development lifecycle is studying the relationship between properties of source code and reported vulnerabilities. This suggests that researchers need to refine vulnerability measurements and give greater attention to metrics for the requirement, design, and testing phases of development.

Rahul Pandita is a senior research scientist at Phase Change. He earned his Ph.D. in computer science from North Carolina State University. You can reach him at [email protected].

Todd Erickson is a tech writer at Phase Change. You can reach him at [email protected].

Published by Phase Change Staff in Community, Science
Tags: ICSE, journal, published paper, rahul pandita, technical paper
Featured Image

March 8, 2018

Phase Change scientists publish paper on lessons learned implementing a natural-language chat interface – blog

Our research scientists recently published a workshop paper on the lessons learned implementing the company's natural-language chat interface. This post summarizes the key lessons learned and identifies the open questions we faced during our initial implementation.

Phase Change is developing a ground-breaking cognitive platform and an AI-based collaborative agent, called Mia, that will dramatically improve software development productivity and efficiency. Mia utilizes natural-language processing (NLP) chatbot capabilities so new users can use the technology immediately with little or no training.

towards jarvis, lessons learned implementing NL chat interface paper

The paper, Towards J.A.R.V.I.S. for Software Engineering: Lessons Learned in Implementing a Natural Language Chat Interface, was co-written by research scientists Rahul Pandita, Aleksander Chakarov, Hugolin Bergier, and inventor and company founder Steve Bucuvalas. The full paper text is available here.

The paper

Virtual assistants have demonstrated the potential to significantly improve information technology workers' digital experiences. Mia will help software developers radically improve program comprehension. Then we will gradually expand its capabilities to include program composition and verification.

Here are a few things we learned during the first iteration of the Mia chat interface implementation.

Reuse components to quickly prototype

Instead of building everything from scratch, consider reusing existing frameworks and libraries to quickly prototype and get feedback.

Gradually migrate from rule-based to statistical approaches

With the ever-increasing popularity and efficacy of statistical approaches, teams are often tempted to implement them without enough data to design an optimal work environment.

We have noticed that recent advances in transfer learning require only a small amount of data to begin reaping the benefits of statistical approaches. However, rule-based approaches still allow prototypes to get up-and-running with only a small amount of set-up time.

A rule-based-approach also allowed us to collect more data for a better understanding of the chatbot requirements, and future positioning to effectively leverage statistical approaches.

Adopt recommendation systems

In our testing phase, we learned that although users appreciated honesty when our chatbot did not understand a request, they didn't take it well (to put it mildly) when the chatbot did not provide a way to remedy the situation.

There can be many causes for the chatbot failing to understand a request. For instance, the request might actually fall outside the chatbot's capabilities, or, in our case, one class of incomprehensible requests were due to implementation limitations.

While we can't do much about the former, building a recommendation system for the later class of requests almost always proves beneficial and vastly improves user experience.

For example, the noise in a speech-to-text (STT) component is a major cause of incomprehensible requests. In our fictional banking system, we've created software that allows pets to interact with ATMs, and a Mia user might form a query to discover all of the uses cases in which the actor "pet" participates.

If the user says: "filter by actor pet," we could expect the following transcript from the STT component, which, unfortunately, caused the subsequent pipeline components to misfire:

  • filter boy actor pet
  • filter by act or pet
  • filter by act or pad
  • filter by a store pet
  • filter by actor pass
  • filter by active pet
  • filter by actor Pat

While users will most likely be more deliberate in their subsequent interactions with the STT component, we noticed that these errors are commonplace and very negatively affected user experience.

To remedy the situation, we used a light-weight, string-similarity-based method to provide recommendations. Subsequent observations indicated that users almost always liked recommendations - except when they were too vague.

To avoid annoying users, we came up with two heuristics. First, we provided no more than three recommendations. Second, to be considered as a candidate query for recommendation, the query's similarity measure had to score higher than an empirically determined threshold with respect to incoming requests.

Over time users stop using fully formed sentences

The novelty of using a natural language interface quickly wears off. We observed that most users began sessions by forming requests with proper English sentences, but the conversation was quickly reduced to keyword utterances. Chatbot designers should plan for this eventuality. 😉

Actually, I find this quite fascinating and the natural evolution of conversation. I think of this phenomena as mirroring our natural conversations. When we first meet someone new, we are deliberate in our conversation. However, overtime, conversations are more informal. But that is a topic for future posts.
~Rahul Pandita, Phase Change research scientist
Subliminal priming

In formal conversation study, the entrainment effect is informally defined as the convergence of the vocabulary of conversation participants over a period of time to achieve effective communication. We stumbled on this effect when we observed that users employed an affected accent to get better mileage out of the STT component.

In psychology and cognitive science, subliminal priming is the phenomenon of eliciting a specific motor or cognitive response from a subject without explicitly asking for it.

We decided to see if subliminal priming would expedite entrainment. We began playing back a normalized version of a query with the query responses. That simple change led users to quickly converge to our chatbot vocabulary.

Consider the frequencies of following user request variations in our system:

Query # of users by
Test Subjects
list computations with a negative balance 30
filter for computations where output concept Balance is less than 0 17
filter by balance Less Than 0 16
filter by output concept balance is less than 0 09
show computations where output concept balance is less than 0 01
filter by output balance less than 0 224

By playing back "our system found following instances where output concept balance is less than 0," to each of these request responses, we observed that users began using the phrase "output balance less than 0," more, as shown in the frequency counts.

For the keen-eyed, notice that the repeated proper phrase, "filter by output concept balance is less than 0" is used less. However, remember that over time, users stop using fully formed sentences.

We also observed that talking with affected American or British accents works. This may be a product of an unbalanced training set used during creation of the speech-to-text models. That's why fairness testing is important. But that is yet another topic for future posts.
~Rahul Pandita
Data-driven prioritization

We also realized the benefits of leveraging data to prioritize engineering tasks as opposed to going with your gut.

A pipeline design is often a used for chatbot realization. Like most pipeline designs, the efficacy of the final product is a function of how well the individual components work in tandem within the pipeline. Thus, optimizing the design involves iteratively tuning and fixing various individual components.

So how does one decide which components to tune first? This is where data-driven prioritization can really help. For instance, in our setting, a light-weight error analysis helped on more than one occasion to identify the components we needed to focus on.

I only imagine that data-driven prioritization will become more useful in the future as we experiment with statistical approaches that often have a pipeline design.
~Rahul Pandita

We hope that our observations will be helpful for those embarking on the journey to build virtual assistants. We would love to hear your experiences.

Published by Phase Change Staff in Science
Tags: aleksander chakarov, chat interface, chatbot, hugolin bergier, jarvis, mia, natural language chat interface, natural language processing, NL, NLP, query, rahul pandita, speech to text, steve bucuvalas, subliminal priming, technical paper
Featured Image

March 29, 2017

Hosting microservices: cost-effective hardware options – blog

April 29, 2017

by Rahul Pandita and Todd Erickson

When we moved from being primarily focused on innovation to also developing a demo platform, our developers began to work with very different frameworks and libraries. As our interactions with more libraries and frameworks grew, we faced dev-setup issues with our monolithic architecture, including:

  • Installing and supporting multiple IDE environments within the single framework. Our developers were installing and maintaining libraries and frameworks locally that they would never need for their current tasks.
  • Software versioning. It's a project manager's nightmare keeping everyone in different teams on the same software versions.

We began to consider moving to a microservices platform, which would allow us to isolate our developers' working environments and segregate libraries and software applications.

Industry literature and Rahul's personal experience at North Carolina State University pointed to a shift away from monolithic architecture to a microservices architecture because it's more nimble, increases developer productivity, and would address our scaling and operational frustrations.

However, moving to a microservices architecture made us address the platform's own issues, namely, how do we access these services – using in-house servers or through third-party hosted platforms?

We first considered moving straight to cloud services through well-known providers such as Google Cloud, Amazon S3, and Microsoft Azure. Cloud computing rates have dropped dramatically, making hosted virtual-computing attractive.

However, at the time, we were still exploring microservices as an option and were not fully committed. Also, we still have to do a lot of homework before transitioning to the cloud. When we added security and intellectual property (IP) concerns to the mix, we decided on an in-house solution for the time being.

This blog post is about our process of determining which servers we would use to host the microservices.

Here we go

To quickly get up-and-running, we repurposed four older and idle Apple Mac Pro towers that were initially purchased for departed summer interns. We reformatted the towers and installed Ubuntu Server 16 LTS to make the future transition to the cloud easier because most cloud platforms support some version of Linux (Ubuntu) out-of-the-box.

The towers featured:

  • Intel Xeon 5150 2.66 GHz dual-core processors with 4 MB cache
  • 4 GB PC2-5300 667 MHz DIMM
  • Nvidia GeForce 7300 GT 256 MB graphics cards
  • 256 GB Serial ATA 7200 RPM hard drives

These towers were fairly old – the Xeon 5150 processors were released in June 2006. We started with them to prove out the approach and quickly determine the benefits without investing a lot of money up front.

Moving to a microservices model immediately solved many of our issues. First and foremost, it allowed us to separate our development environments into individual services.

For example, our AI engine for logic queries could work independent of our program-analysis engine and our text-mining work. This was incredibly helpful because, for example, developers working on program analysis who did not directly dealt with the AI engine didn't have to install and maintain AI-specific libraries, and vice-versa for AI developers and program analysis tools.

Now, each team simply interacts with an endpoint, which immediately improved our productivity. More on this revelation in a future post.

As we continued to implement the microservices platform, we were pretty happy with the results. Then our servers started showing signs of their technological age – performance lags, reliability issues, limited upgradeability, and increasing power consumption. The limited amount of DIMM, limited cost-effective upgrade capabilities, and constant OS crashes hampered our efforts.

Next step

For the next "phase" of our microservices evolution, we decided to acquire performant hardware specifically geared for hosting microservices.

Phase Change is a small startup with limited funding, so we had to purchase equipment that would meet our needs within a budget. Like many ‘cool’ startups, we are a Mac shop, so we naturally gravitated towards using Mac mini servers. We were already using Mac minis for file hosting, and there are plenty of websites detailing how to use them.

After conducting random Google searches extensive online research, we decided our best option was not the Mac mini with OS X Server, but the original Mac mini model. The Mac mini with OS X Server features an Intel Core i7 processor and dual 1 TB Serial ATA drives, but Apple stopped offering the mini with OS X Server in October 2014.

So, we considered the next best thing, mid-level original Mac minis that included:

  • Intel i5 3230M 2.6-3.2 GHz processors with 3 MB cache
  • Intel Iris Pro 5100 HD graphics cards
  • 8 GB 1600 MHz LPDDR3 memory
  • 1 TB 5400 RPM hard drives
  • 1000 Base-T Gigabit Ethernet support

The Mac mini form factor – 7.7 inches width by 1.4 inches height and 7.7 inches depth – and power consumption – 85 W maximum continuous power – were also appealing. The retail base price is $699. The cost-effective modern processors and increased memory were the most important factors in our consideration, and the tiny little Macs would integrate well into our 'cool' Mac company environment.

We were all set to move on the Mac minis until we found Russell Ivanovic's blog post, "My next Mac mini," which revealed that the Mac mini product line hasn't been updated since October 2014 – over 2.4 years, but Apple is still selling them at new-computer pricing. So much for the minis. Aargh!

Luckily, we didn't have to start at square one this time around, because Ivanovic's blog post revealed what he bought instead of the mini – an Intel NUC Kit mini PC.

Intel NUC KitWe asked Siri to do the math crunched the numbers and found that the NUC was a reasonable Mac-mini replacement. The Intel NUC Kits are mini PCs engineered for video gaming and intensive workloads. The base models include processors, graphics cards, system memory, space for permanent storage devices, peripheral connectivity ports, and expansion capabilities, but we upgraded our NUC6i7KYKs to include:

  • Intel Core i7 6770HQ 4.0-4.65 GHz quad-core processors with 8 MB LC cache
  • Intel Iris Pro 580 graphics cards
  • Crucial 16 GB (8 GB x 2) DDR4 SODIMM 1066 MHz RAM
  • Samsung 850 EVO 250 GB SATA III internal SSDs
  • 1000 Base-T Gigabit Ethernet support

The following table presents technical comparisons between the old Mac towers, the Mac mini, and the Intel NUC Kit.

Mac Pro Tower Mac mini Intel NUC Kit NUC6i7KYK Comments
Base Price $200-$300 $699 $569 Mac tower has been discontinued but you can still buy preowned hardware. We chose the mid-level Mac mini ($699) for comparison fairness.
Processor Intel Xeon 5150
2.66 GHz dual-core		 Intel Core i5 3230M
2.6-3.5 GHz
dual-core		 Intel Core i7 6700K
4.0-4.65 GHz
quad core		 Processor comparisons
Xeon 5150 v. i5 3230M
Xeon 5150 v. i7 6700K
i5 3230M v. i7 6700K
You can update the Mac mini to an i7 processor for $300.
Graphics card Nvidia GeForce
7300 GT		 Intel Iris Pro
5100 HD		 Intel Iris Pro 580 Graphics card comparisons
Nvidia GeForce 7300 GT v. Intel Iris Pro HD 5100
Nvidia GeForce 7300 GT v. Intel Iris Pro 580
Intel Iris Pro had 5100 v. Intel Iris Pro 580
Apple hasn't officially released info on the Mac mini's exact graphics chipset, so we used specs fromEveryMac.com for comparisons.
RAM 4 GB PC2-5300
667 MHz		 8 GB LPDDR3
1600 MHz		 Crucial 16 GB SODIMM DDR3L
1066 MHz		 Out-of-the-box NUC Kits do not include RAM. We installed 16 GB DDRL3 SODIMMs in our NUC Kits for $108 each.
The Mac mini is upgradeable to 16 GB for $200.
Storage 256 GB
Serial ATA
7200 RPM		 1 TB
Serial ATA
5400 RPM		 Samsung 850 EVO
1 TB
SATA III
internal SSD		 Out-of-the-box NUC Kits do not include internal storage. We installed 250 GB SSDs ($109 each) for a good performance/capacity mix, but use a 1 TB SSD here for comparison fairness.
You can upgrade the Mac mini to a 1 TB Mac Fusion Drive (1 TB Serial ATA 5400 RPM + 24 GB SSD) for $200.
Comparison Purchase Prices (per unit) $200-$300 $1,399 $997 Mac mini upgrades: Intel Core i7 processor ($300); 16 GB LPDDR memory ($200); 1 TB Fusion Drive ($200)
NUC Kit Config Upgrades: 16 GB DDR3L memory ($108); 1 TB SSDs ($320)

Download the table in PDF

Our NUC Kits total price ended up being $786 per unit with the 16 GB SODIMM DDR3L RAM and 256 GB SSDs. If we had opted for 1 TB SSDs to match the standard capacity in the mid-level Mac mini, our price would have jumped to $997 per unit.

We chose the Intel NUC Kits over the Mac minis because of the NUC Kits' updated technology and overall better performance for the price. Putting together and installing Ubuntu Server 16 LTS on the NUCs was very straightforward.

Both units are fully configured and have been in full production operation for a few weeks. We haven’t encountered any issues. I'll divulge more on how they perform over time with different microservices and workloads in future blog posts.

P.S. We still looooove Mac towers and we are currently using them as test beds. That will also be the subject of a future blog post.

Rahul Pandita is a senior research scientist at Phase Change. He earned his Ph.D. in computer science from North Carolina State University. You can reach him at [email protected].

Todd Erickson is a tech writer at Phase Change. His experience includes content marketing, technology journalism, and law. You can reach him at [email protected].

Published by Phase Change Staff in Technology
Tags: blog, community, microservices, rahul pandita, software development, todd erickson

Contact

651 Corporate Circle
Suite 209A
Golden, Colorado 80401
Phone: +1.303.586.8900
Email: [email protected]

© 2024 Phase Change Software, LLC