Delivering Safety & Cybersecurity Critical Software Faster and Cheaper
September 29, 2022
How four leading safety-critical systems developers have used TrustInSoft Analyzer to eliminate pesky bugs, guarantee cybersecurity, improve customer relationships, and reduce verification costs
Introduction
In Part 1 of this series , we discussed why software for safety-critical systems is hitting an “affordability wall,” why reliance on wireless connectivity in those systems is exacerbating the problem, and one way the cost of critical software could be dramatically reduced.
In Part 2 , we examined why industries like aerospace and nuclear energy generation are moving away from traditional software testing and toward exhaustive static analysis and formal verification to solve the problems just mentioned.
In this final installment, we present four case studies showing how TrustInSoft customers have profited from using exhaustive static analysis in their development of safety-critical software.
This post is Part 3 of a 3-part series derived from TrustInSoft’s latest white paper, “Delivering Safety-Critical Software Faster and Cheaper: How Exhaustive Static Analysis Guarantees Correctness and Security while Reducing Cost and Schedule.” To obtain a FREE copy, CLICK HERE.
Tier 1 aerospace supplier cuts robustness testing costs by 75%
One of our customers is a tier 1 aerospace supplier with broad expertise in safety-critical and mission-critical embedded software. They develop software modules for major aircraft OEMs and have years of experience in certification to DO-178C.
A few years ago, they were contracted to develop for commercial transport aircrafts a module which is not classified as safety-critical, but is cybersecurity-critical as it connects the plane’s aircrew communications and in-flight entertainment systems to the internet.
Any vulnerability in the module, therefore, could allow hackers to access the plane’s flight control system, just as the FBI alleged Chris Roberts had done. Its code needed to be flawless.
An experienced aerospace supplier, the company already had a proven process for assuring certification of safety-critical software. Cybersecurity, however, was another matter. The cybersecurity issue being more recent, processes were in place but still evolving.
Because they as well as their customer and their certification authorities had a strong understanding of their safety assurance procedures, they initially considered adapting those same procedures to cybersecurity assurance.
This supplier soon realized, however, that using those procedures would entail a very high volume of robustness testing. They feared the process would become too demanding and expensive. They wanted to compare their results with an alternative solution to see if they could reduce cost and schedule while assuring even greater robustness.
They had another concern as well: cybersecurity can be considered even more demanding than safety. While safety-assurance robustness testing might give one the confidence that there is a less than 10-12 chance a software bug will result in a safety-critical failure, a hacker will be using search algorithms to actively hunt for those “10-12” cases, hoping to exploit them.
After a thorough search, the company chose TrustInSoft Analyzer for their comparison.
They set up two verification efforts in parallel. On one side, they applied their standard V&V process. Four engineers carried out their standard unit testing effort for one month.
On the other side, they used exhaustive static analysis. One engineer worked the same month finding errors with TrustInSoft Analyzer.
After that one month, they found that the verification coverage and the number of coding errors uncovered and eliminated using TrustInSoft Analyzer were far superior to what they had been able to achieve using their standard V&V procedures for safety-critical software.
In other words, TrustInSoft Analyzer allowed this supplier to cut their robustness testing costs by 75% while achieving better coverage and allowing the company to show they had done all the necessary testing.
Because TrustInSoft Analyzer guarantees that no bugs remained in the software, it brings a level of confidence to the end customer on quality of the software which cannot be reached any other way.
Tier 1 defense contractor streamlines customer acceptance
Another of our clients is a well-known name in aerospace and defense and had a contract to develop a package of cryptographic software.The code required the highest level of cybersecurity assurance.
As mentioned earlier, cybersecurity norms are constantly evolving and this supplier was looking for a solution that could support adequate cybersecurity assurance processes and procedures requested by their customer.
The firm decided to seek a new solution—one that could provide proof of cybersecurity. They chose TrustInSoft Analyzer because it promised a mathematical guarantee that all vulnerabilities—all defects that could act as entry points for hackers—had been eliminated from their code.
Thanks to TrustInSoft Analyzer, this defense contractor was able to achieve an extremely high level of confidence in their software. TrustInSoft Analyzer’s mathematical guarantee allowed the company to easily demonstrate that high level of confidence to their customer, which, in turn, gave the customer a very high level of confidence in the code they received.
As the company had hoped, TrustInSoft Analyzer’s unequivocal, mathematical guarantee shifted the acceptance conversation from the verification process to the actual software product. This guarantee shortened the discussions over verification and certification and smoothened the supplier’s relationship with their customer. Ultimately, TrustInSoft Analyzer streamlined the company’s entire development process, reducing their verification and validation timeline and speeding the delivery of their product to the field.
ATC system integrator expands test coverage while reducing costs
This customer, an air traffic control system (ATC) integrator, has a well-defined verification process following the traditional V-diagram model. They are similar to the suppliers in our previous examples in that regard.
Unfortunately, that process was becoming unaffordable due to the rising complexity of the software they develop. The firm wanted to lower the cost of its error detection and correction process.
In a typical development program, this company’s functional-level software testing will reveal hundreds if not thousands of bugs. About one-third of those bugs will be undefined behaviors—pure coding errors that were not the result of errors in the requirements or software architecture. Bugs discovered during functional testing cost the company four times the effort required to correct them during the development phase.
The subsequent step of their verification process, system testing, would typically yield about a third of the number of bugs discovered at the functional test level. Again, one-third of these are pure coding errors. Due to the ATC industry’s intense certification process, bugs discovered at the system level require four times the effort required to correct them during functional testing. That’s sixteen times the effort required to correct them during the development phase.
The company reports that the actual engineering time spent correcting each bug—regardless of the process phase in which the bug is discovered—is a maximum of two hours. The rest is overhead.
This ATC system integrator’s objective is to find bugs earlier in their development process with TrustInSoft Analyzer and thus reduce their burden at the system test level.
TrustInSoft helped them to increase their level of test coverage within an environment that replicates their target system. Even better, they can detect the vast majority of their bugs early in their development process and eliminate most of the overhead tasks their engineers previously had to perform,significantly lowering the company’s verification costs.
A potential customer puts us to the test
Even in safety-critical software—despite having the most rigorous standards, processes, and procedures in place—subtle bugs can occasionally find their way past conventional testing and into the field.
It’s a constant fear among critical software companies who rely on traditional testing: despite proven processes and the utmost diligence, a bug slips past their testers and causes a catastrophe. Even after doing everything possible in the time available, software developers and testers in these industries may wonder—at least in the back of their minds—if they’ve done enough.
This is exactly what happened to one aerospace supplier who came to TrustInSoft. An undetected bug in one of their software modules went to the field and was eventually discovered by their customer. Once alerted to its presence, it took the company months to find, understand, and correct the bug.
The cost to the company was enormous. Not only did this supplier have to absorb a massive, unplanned engineering effort. The incident also had a major impact on the company’s reputation and its relationship with their customer.
Later, when the company was considering TrustInSoft Analyzer for purchase, they presented us with a challenge. They gave us the original, defective code as a test saying, “If you can find the bug, we’ll buy your tool.”
In response to this challenge, one TrustInSoft engineer took two to three days to set up and launch the analysis of the customer’s code with TrustInSoft Analyzer. TrustInSoft Analyzer quickly found the bug… along with others not previously discovered.
Every one of these newly discovered bugs was extremely subtle. They would all have been very difficult to find through conventional testing. With TrustInSoft Analyzer, though, one engineer was able to find all of them in just a couple of days.
For more information…
These are just four customers among the many who have streamlined their verification process and trimmed costs and schedules using TrustInSoft Analyzer. To learn more about TrustInSoft Analyzer, visit our product page: https://trust-in-soft.com/product/trustinsoft-analyzer/.
If you have enjoyed this post and would like to have the entire series in a more portable format, be sure to download our new white paper, “Delivering Safety-Critical Software Faster and Cheaper: How Exhaustive Static Analysis Guarantees Correctness and Security while Reducing Cost and Schedule.” You can download your free copy here .
This post is Part 3 of a 3-part series derived from TrustInSoft’s latest white paper, “Delivering Safety-Critical Software Faster and Cheaper: How Exhaustive Static Analysis Guarantees Correctness and Security while Reducing Cost and Schedule.” To obtain a FREE copy, CLICK HERE.