Managing Requirements Tracking, Implementation and Sign-off for Embedded Systems

INTRODUCTION

This document describes the issues faced when building hardware and software systems where the success of the project is dependant on requirements being fully supported and tested. Where the cost of failure is high there is a greater necessity for a robust requirements sign‐off capability. This particularly applies to systems where the financial cost of recalling a failing product is prohibitive and/or there is a high safety factor which is typical of embedded systems.

The following represents an approach to achieving the above through a combination of a software solution, asureSign™ and associated best practice as defined by Test and Verification Solutions Ltd (TVS).

CURRENT INDUSTRY PRACTICE

Currently best practice in requirements tracing stops at test definition. From that point the industry provides only a partially automated approach and most software developers settle for a manual one. There is no tool that will automatically track the results of tests as they apply to requirements. Quite often companies export tests from their requirements management tool into a spreadsheet (or similar) and then record test results in the spreadsheet. They do not hold the tests results for more than a few days and as changes are made the test results are soon out‐of‐date. What is really required is the ability to automatically record test results against the requirements they satisfy, to generate management reports from those results generate and to keep those results for future reference.

The hardware industry has developed numerous techniques to help verify design correctness, such as pseudo random testing, functional coverage, assertions, formal verification etc. But the usage of all these approaches also brings other problems. Quite often a particular requirement will be verified by a collection of approaches, and with hundreds (or thousands) of tests, functional cover points, properties etc complicated by the fact that some tests or cover points could target more than one requirement, it quickly becomes non‐trivial to determine how well a particular requirement is progressing. Different tools also require different approaches to analyse their status, often having independent people in charge of them with independent flows. This makes it harder to see the full picture and to understand how an individual’s activity fits in relation to the project. Most companies address this by taking a number of days at the end of a project (when the pressure is greatest) to manually map all these approaches to project requirements. Inevitably this is time consuming and only provides valuable information at the end of a project. It also usually results in the identification of verification holes, leading to more work and another round of manual signoff. Finding late bugs often has the same effect.

The software world developed an array of tools for analysing source code and testing executable code associated with newly developed programmes and applications. Source control software provides a mechanism for defining versions of software and an associated history. Requirements management software enables the definition and tracing of user requirements. Bug tracking provides a simple process of listing and describing bugs and their status. However, these do not address the issue of ensuring that requirements had tests defined against them and that these tests were successfully completed.

THE asureSign APPROACH

The asureSign approach was identified during a UK]wide survey into verification carried out by TVS on behalf of the National Microelectronics Institute. The approach is equally relevant to both hardware and software development in solving the above challenges. It is derived from the needs expressed by Infineon Technologies AG and XMOS Ltd; both companies are designers and manufacturers of advanced semiconductor products and their associated software. XMOS undertook early development of a solution to problems stated above and used it to sign off both their hardware and software products. That solution was subsequently taken over by TVS (under agreement with XMOS) and Infineon has contracted with TVS for development of a software product, and have been a primary contributor in determining the functionality.

asureSign addresses the gap between the capture and tracing of functional requirements available in industry leading products (such as DOORS. from IBM and Reqtify. from Geensoft) and the testing capability provided by a whole host of solutions. The industry has to date not addressed the need to ensure that for every functional requirement there is, i) a test for each requirement, ii) what tests have been specified but not written and iii) what tests have been written but not run. There has also been no simple and cohesive method for tracking over time how a project is developing with respect to every feature and requirement, and how these features relate to the tests that are used to measure their progress

asureSign has been developed to address these industry shortcomings. It provides a solution that both supports management in delivering higher quality products and developers/testers in achieving more complete and robust development and testing procedures in support of their system development.

asureSign uses the flexibility of a relational database to track how a project is developing over time for every feature and requirement, and how these features relate to the tests that are used to measure their progress. The product tracks not only tests, but an array of information that can be used to measure progress: coverage, memory leaks, performance, etc. The asureSign database enables management to ask more complex questions, from very high to the very low level:

• reports on a range of key criteria associated with the sign‐off of your functional requirements including who ran the various tests and on what servers.
• reporting of incomplete test specification and the presence of the tests specified, written and run on a project wide basis
• how are my requirements progressing over ti me, with visibility in to historical results and trends and how they relate to requirements
• improved decision making due to the high quality, real-time information on the project status

The database also provides the ability to access real]time information on the progress of a test run, and the advantage of aggregated statistics about the tests: How long to they take to run?How much computing power do I need to run or verify a particular requirement ?

For developers and testing teams asureSign provides a structured and logical means to improve control;

• visibility on early stages of test and verification and the automatic tracking of test resulcoverage (structural and functional) for a wide variety of verification and testing functio
• bug tracking. When a test fails your bug fixing system will be updated
• plotting of historic results e.g. Was this passing before? If so, then what version of the source code was it using and what changed between those two runs of the tests?
• automatically relate test results to the code that generated them

Figure 1: This shows the change over time for the correlation between requirements that have tests specified, written, run and passed

asureSign has a number of opportunities to provide automated links to products addressing key areas of support for other parts of the system development llifecycle.

• Requirements Management: By linking to established products users can ensure that requirements are fully tested.
• Configuration Management: By llinking to a source control system asureSign can provide a history of which versions were tested at what time.
• Bug Fixing: By linking to bug fixing systems users can ensure that their bug fixing system will be updated with the latest test results.

SUMMARY

asureSign provides embedded system developers and management with a controlled environment for managing the implementation, tracking and sign-off of requirements. Development teams can be certain that requirements have been implemented through the development and testing of associated programs.