ESC: Setting up inspection for software quality

However, there is no obvious data that shows embedded software will benefit more or less from software inspection. But casual observation does seem to suggest that designers of embedded projects employ formal inspection to a smaller degree than, say, PC developers. This could be because of the historically s maller teams that have been used to develop embedded software and firmware. In such environments, individuals have complete control and responsibility for a subsystem. However, as embedded platforms grow and the applications increase in size and lines of code, software inspection will be recognized as a necessary tool to achieve desired levels of quality.

The formal inspection is the only mechanism that provides all the value stated. But there's another contribution that inspections make: the amount of data that can be collected around the inspection process. The following metrics can and should be collected: effort to prepare for the inspection, number and types of defects found and the cause of the defect (design, interface, construction). The formal inspection, as the name implies, has a more rigorous preparation process than the other two types.

Review elements can be broken down into three groupings: errors and vulnerabilities, architecture and organization, and consistency to style. So me of these groups are more critical than others.

The five inspection elements that must be considered in any formal software inspection process, whether embedded or not, include some common mistakes:

• Improper scope of variables, especially the use of global variables and access functions.
  

• Poor or ineffective use of comments.
  

• Dangerous or subtle pointer implementation.
  

• Mixed data-type interactions.
  

• Function structure.
  

Beyond these, other inspection elements that are important specifically in the case of embedded systems design include:

• Processor performance.
  

• Use of resources (RAM, ROM, Stack and HEAP).
  

• Exception handling.
  

• Interrupt service routines.
  

• Hardware abstraction.
  

• Instrumentation.
  

• Mixed mode programming.
  

• Testability.
  

It is important for at least some of the reviewers to have knowledg e of the platform at which the source code is targeted. Hardware is generally abstracted in most embedded applications, but even so, someone has to write the interface routines and these routines must be inspected. In addition, knowledge of the system's throughput limitations can help inspection groups determine if a specific block of code is using more bandwidth than it should.

Regardless of the length of code blocks, their use of system resources must be critically analyzed. Inspectors should ensure that even short loops are running at the appropriate priority level and within the correct timing loop or scheduled task. Scope of variables has been mentioned, but only from a persistence and data-hiding perspective.

In an embedded system, a designer must also decide the correct type of memory in which to locate a variable and the inspection team must verify this decision. The desired persistence of the value must be considered. If the variable requires some sort of static storage, the develo per must decide whether it should be in volatile RAM (internal or external, fast or slow), or nonvolatile RAM (such as battery-backed RAM, electrically erasable RAM or flash). A variable could also be stored in ROM if it is constant during the production run time of the application, but its ultimate value is not known at compile time. The initialization of each variable must also be approved.

Temporary, local and otherwise nonpersistent variables should also be analyzed. These are usually stack or heap based, but may also be register based. Sometimes this will not be known until the compilation is complete, but the worst case should be assumed.

Memory-mapped registers also require attention by inspection teams. These variables are especially vulnerable to optimizations made by a compiler. If a hardware register is read multiple times during the execution of a block of memory, the compiler may assume that the value at that location does not change outside of local control.

Exceptions are generally used heavily in embedded products, and each implementation of an exception handler is a little different. This variability is dictated by the differences in the way each microcontroller deals with exceptions. Exceptions run the show; they direct good and bad behavior. Controlled by the exception handler are traps and other software interrupts as well as some hardware interrupts like serial devices, timers and analog to digital converters. The exception handling must be analyzed critically and any logic that causes an exception or reacts to one must be inspected with extra care. This includes the abnormal exceptions like divide by zero, address errors, illegal instructions and so on, as well as the normal exceptions already mentioned.

Interrupt service routines (ISR) are an invaluable element of embedded systems development, but they also add an additional level of complexity that must be managed. ISRs can be scary solely because, seemingly at random, they change the flow of execution o f a microcontroller. Because of this behavior, logic that can be interrupted must ensure that its priority is set to an appropriate level. If there is a period during which it would be "bad" for flow to be interrupted, then the appropriate priority level, or interrupt masking, must be activated. The setting of priority levels and interrupt masking should be an obvious element of embedded software inspection.

ISRs, like any logic block, can be too large. The results of exceptionally long service routines can be disastrous. Interrupts can be missed, the microprocessor can become overloaded, normal execution may halt while exceptions are processed and so on. There are several techniques that can be used to shorten ISRs: spawning a task, setting a flag for additional processing to happen in the normal logic flow, optimizing code and more. The inspection should not try to redesign the function or correct the vulnerability. The inspection is solely responsible for pointing out the problem and trusting that the engineer will seek assistance if he or she feels taxed by the problem.

Even short service routines can be problematic if they are not constructed in a re-entrant manner and there is a chance that the same routine can be called before it has had a chance to complete. Static and global variables accessed and changed by the service routine run the risk of participating in a race condition — a dastardly little varmint that is difficult to debug.

Instrumentation is commonly attached to the target under test during validation and debugging. Sometimes emulators even replace (in- or out-circuit) the ROM device so that breakpoints and bit patches can be achieved during debugging. It is accepted that external tools, like instrumentation, should not change the way software is developed; however, this often happens. Extra routines must be written to interact with the test machinery; variables and functions sometimes have to be structured in a less than ideal manner to deal with instrumentation idiosyncrasies.

Mixed-mode programming refers to any scenario where multiple programming languages are used together. This could be as simple as a C function calling an assembly routine, but other examples include inline assembly, inline microcode (for example, microprocessor machine code or TPU instructions) or other numeric, usually hexadecimal, data that has to be placed in a certain spot of memory. This placement can be accomplished through software execution, loading an array during initialization, or by the linker command file. The linker command file can define a location in ROM and then the piece of logic containing the microcode will be placed in the memory map at that location by a linker-specific set of instructions.

Ultimately, even with software inspection, the software has to be tested. It will likely go through multiple stages of testing: unit testing, implementation testing, systems testing, perhaps even regression testing. Testing should be on the mind of the developer durin g construction and it should be on the minds of the reviewers during software inspection, so logic can be constructed that facilitates easier testing. Shorter functions with low amounts of nesting are much easier to unit test. The amount of nesting and, therefore, the number of branches through the logic contributes to a value known a cyclomatic complexity. Lower cyclomatic complexity is easier to test, to understand and to modify safely. Thus, some software groups set a maximum cyclomatic complexity as a metric for software compliance.

This article is based on excerpts taken from ESC class # 461, Embedded software inspection overview.