The Power of Developing Hardware and Software in Parallel

How Virtual Prototypes Reduce Risk in a High-Stakes Game

Tom De Schutter, Product Marketing Manager, Synopsys, Inc.

Introduction

The culture-shifting rise of mobile devices in the last decade has impacted everything except the embedded software development process. Intelligent, smart-connected mobile devices¹ have not only eclipsed yesterday’s desktops in popularity but now exceed them in raw computing power. One company that ruled the technology market in 2000 has seen its stock plunge 75%, dragged down by “having an aging PC business hanging like an albatross around its neck.”² The global smartphone business, by contrast, is expected to grow from US $69.8 billion in 2010 to US $150.3 billion by 2014.³

Thanks to the proliferation of smartphones, tablets and more, embedded systems, so-called for their habitat inside other systems, have been flushed out of hiding and into the hands of billions of consumers around the world. The basic characteristics of embedded systems—rigorous size and power constraints, precise allocation of system resources and real-time or near real-time execution—are the best platform match for the pocket-size multifunction devices that dominate the consumer experience today, BUT.

The “but” is a big one. The complexity of hardware, software and HW/SW integration that arises from the convergence of so much functionality in such small devices has driven both hardware and software innovation at almost break-neck speed, while the development methodology that brings hardware and software together in market-ready devices lags behind. Sequential development, with software development waiting for available hardware, is still the prevailing industry norm. But sequential development often fails to deliver quality products within the version cycles and seasonal market windows that rule the consumer space today. A 2010 EE Times market study of embedded device development reported that 56% of projects were finished late. Such delays are non-trivial in a volatile marketplace.

With time, cost and quality as key concerns across the device supply chain, both semiconductor companies and OEMs are turning to virtual prototyping to shorten development cycles and buy time to better integrate and more thoroughly test new products. Virtual prototypes, which represent fully functional software models of systems under development and execute unmodified production code, have become mainstream tools for earlier and faster software development, allowing teams to develop, integrate, test and analyze software far in advance of hardware availability.

Figure 1: Virtual hardware lets software teams get to work even before RTL

Exploring trends in the highly competitive mobile device market and the highly regulated automotive industry, this paper will discuss the challenges device development teams face today and how virtual prototyping can help meet them.

Convergence, Complexity and Competition

Camera, video, telephone, music, email, 3D gaming, social networking, GPS, Internet browsing, voice and gesture recognition, cloud portal, data communications center, content creation as well as consumption—as consumers today, we expect our smartphones to do what used to be the work of a dozen separate products. As the lines between business, entertainment and social applications blur, we ask them to securely host both our personal and professional data. Because we do our buying and our banking on these devices, we expect them to be impregnably secure—open architecture notwithstanding. We expect them to respond instantly to our touch and move between applications with no glitches or delays. We demand plenteous features, long battery life and ever increasing multi-standard communication speeds. How well a device mediates between performance and power management has become a determiner of how well it performs in the marketplace.

And what a fickle, fast-paced marketplace it is. The tables below, displaying 1Q12 market data for both global mobile and smartphones, illustrate the volumes of the business and how quickly the fortunes of the players in it can rise and fall. A look at the last column, reporting year-over-year change in market share can be read as proof of the power of one release to make or break a corporation.

Table 1: Top Five Worldwide Total Mobile Phone Vendors, Shipments, and Market Share, Q1 2012 (Units in Millions)

Table 2: Top Five Worldwide Smartphone Vendors, Shipments, and Market Share, Q1 2012 (Units in Millions)

Everything we as consumers want turns into requirements device manufacturers must capture and satisfy, at the hardware or the software level or most often, at both, exerting pressure on every link in the device supply chain.

Hardware Complexity

Embedded devices have always relied on custom hardware. A broad array of use cases–robot arm controllers, set-top boxes, the military, network routers, residential gateways, microcontrollers for assorted automotive functions, to name a few—gives rise to an equally broad array of hardware platforms and architectures. This absence of standardization adds both time and risk to even the simplest, single-function device development project.

Convergence drives innovation

Moore’s law observes that the number of transistors on integrated circuits doubles approximately every two years, an estimated velocity that may have been surpassed with the emergence of today’s System(s) on Chips, or SoCs. This relentless advancement in silicon process technology has driven swift innovation in the semiconductor industry, focusing first on the multicore architectures that enable multiprocessing, then forcing a shift from Application-Specific Integrated Circuits (ASIC) to SoC designs.

Figure 2: Diagram of an SoC

An SoC, consisting of at least one, or typically more application processors (CPUs), one or more digital signal processors (DSPs), data processing engines (DPEs) for high resolution graphics, interfaces for high-speed cellular communication, Bluetooth, WLAN, and other peripherals4 is several orders of magnitude more timeconsuming and costly to design, fabricate test and integrate than independent ASICs. It not only loads a lot of software onto one chip, but also requires considerable R&D investment upfront that’s only recovered if and when devices built on it ship in volume and win respectable market share.

Who’s responsible for the software?

The time and expense of creating a complex SoC joins with the industry business model to give semiconductor vendors an urgent stake in helping their customers get to market faster. OEMs want a “complete” product so they can just start innovating; semiconductor companies want a fast ROI; hence today, semiconductor companies find themselves in the software business, taking increasing responsibility for the software that mediates, manages and differentiates the device at every level, from power and memory management to system security to must-have applications.

The semiconductor company that provides the best software reference stacks for particular device types —set-top boxes, for example, or machine to machine applications—gains traction in those market verticals. OEMs expect new SoCs to be pre-validated with popular operating systems and to ship with a Board Support Package (BSP) to jumpstart software development across their ecosystem. Thus semiconductor companies are now taking on software development tasks that were traditionally performed by embedded software vendors. Recent acquisition of embedded software companies by semiconductor vendors—Monta Vista software by Cavium Networks, Wind River Systems by Intel, for example—underscore how valuable software has become not just to differentiate end products but to give chip companies a competitive edge with OEMs.

The willingness of rival silicon companies to make common cause around open source initiatives like the Yocto Project, which seeks to standardize tooling and SDK generation for embedded Linux across multiple architectures and Linaro, a non-profit organization that consolidates and optimizes open source Linux software and tools for the ARM architecture, testifies to both the difficulty and the urgency of developing quality software for today’s complex devices and the need for application portability across device platforms.

Software Complexity

In mobile devices, the first software development task is to create boot ROM code to bridge the gap between hardware and software. Boot ROM code is the code that first executes when you push the “on” button on your mobile phone. It initiates key security functions and prepares the SoC to get the operating system up and running. This includes initialization of all the subsystems and peripherals like the USB port, for instance, setting up the memory controller and more. Boot ROM is the piece of software that has to be working before other software can be integrated, before the huge post-silicon validation suite can be loaded to test every aspect of the SoC, before the OSes can be brought up, before development teams can start climbing the software stack.

Delays here can jeopardize a software development schedule even before it’s gotten underway; and, given the complexity of the systems involved, delays do occur. Horror stories of months lost getting the test suite to run are as common as they are costly. Sequential dependencies between hardware and software bring-up at the beginning can shorten the time available for integration, test and validation at the end. Not only the schedule but the quality of the product is put at risk.

The State of the Stack

The figure below is a simple representation of a device software stack today.

Figure 3: The Device Software Stack Today. Tasks outlined in red have become the domain of the chip company

As the inverted pyramid suggests, software content grows exponentially in both size and complexity as development progresses. Each traditional phase of embedded software development—from “low level” engineering like hardware enablement and validation through device driver development, operating system bring up and the middleware that supports a universe of applications—adds millions of lines of code. Both the sheer size of the code load and the sophisticated work software is required to perform in today’s devices introduces opportunities for error and demands rigorous test and validation at both component and system level. This work is both time and labor intensive.

Where there used to be clear separation between chip and operating system development tasks, silicon companies now take ownership of a broader swath of the software stack in order to pass time savings along to their customers. In part, this shift reflects the growing popularity of open source operating systems like Linux which is not natively optimized for embedded development and the Android platform, which is. Even more important is the way in which hardware complexity has driven change—and responsibility—up the software stack. Today’s board support packages must not only enable highly sophisticated, brand specific SoCs, they’re also the site of some 50% of power management code. Driver development and integration is tightly aligned with the BSP. As the importance of these elements has skyrocketed, so has the volume and sophistication of the software development semis are expected to perform. Some would argue that the software tasks being assumed by silicon vendors are among the most challenging in device development today.

If our diagram dramatizes the challenge, it also hints at a solution. Even as software content increases, the dependency between hardware and software decreases as development proceeds up the stack. A hardware simulation good enough to let software development start before physical boards are in hand becomes a magic bullet. Because of their responsibilities up and down the stack, semiconductor vendors are embracing virtual prototypes as a way to hit their market targets.

The Multicore Challenge

Multicore programming, debug and test are demanding new disciplines that many engineering schools don’t teach yet, although multiprocessing is now the accepted means of load balancing power and memory resources in complex systems and the number of CPUs and Data Processing Engines (DPEs) is expected to continue rising for the next decade.⁵According to VDC’s 2011 embedded market survey, 31.6% of embedded engineers cite no experience working with/programming multicore and/or multiprocessor designs⁶, yet every new mobile device on the market incorporates multiple processors to meet its workload. Silicon vendors are actively shopping for engineers with multicore programming skills, but talent with the requisite deep understanding of hardware and software capabilities is hard to find. The most enlightened companies facilitate communication between hardware and software teams throughout design and development, which helps to optimize the successful orchestration of multiple cores. Unfortunately, this is still the exception and not the rule.

Power and Performance: Concurrent Problems

A mobile device with lots of bells and whistles and no battery staying power is doomed to disappoint consumers. To develop a solution to the concurrent and interdependent problems of power management and optimal performance requires sophisticated multicore programming skills and deep knowledge of hardware capabilities. The software programmer addresses not what the device is capable of doing, but how it is used. The addition of the relatively new Power Management Unit (PMU) to hardware platforms aims to solve the problem, but the PMU must be properly threaded into the OS, drivers and applications in order to do its job, a complex software development task. What makes this issue so challenging is that even though performance impacts power and the reverse, they are not the same problem. It takes two equally accomplished dancers to perform this delicate and complicated waltz.

Security: A Growing Concern

The many aspects of security and the inherent vulnerabilities of open software architectures also add complexity to mobile device design. A number of promising approaches contribute to security without making it absolute. Activating authentication through the SoC and using this to create security zones within the operating system is one such strategy. The hypervisor, a virtual hardware platform that sits above the BSP and manages the execution of multiple guest operating systems is also being used to implement security. Using Virtual Machines (VMs) to separate personal and business data on a single device helps to secure sensitive business information. While these and other initiatives help ease consumer anxiety, the realistic goal of most security features is to drive the cost of hacking them high enough to discourage the activity.⁷ Like the rest of us, criminals want a positive ROI.

The Shape of the Team: Tipping toward Software

Changes in the makeup of development teams, too, reflect a rise in the perceived value of software. Embedded industry veterans remember a day when software was something of a poor relation, getting slapped on at the tail end of development. Today, according to industry analyst VDC, the average device engineering team has 33 members, 40% of them software engineers.⁸ Software now accounts for nearly half the cost of new product development—47% among VDC survey respondents in 2011. Most telling, the 12-18 months it takes to develop, debug and test embedded software is now equal to the time needed to implement a complex new hardware design.

It’s important to note that on teams employing traditional embedded development methodology, working on the same team does not mean working collaboratively.

As soon as the architecture is defined, hardware and software specs are created and hardware and software teams work independently until software and chip meet late in the development cycle. In most companies today, the hardware/software divide is infrequently crossed. Citizens of one kingdom are not fluent in the language of the other, nor do they see the world in the same way. The hardware mind is necessarily detail oriented, while software developers tend to be more abstract thinkers. Most often, these necessary minds work sequentially, in isolation, even though the problems they look to solve are really multidisciplinary.

Let’s Do the Math

Given realistic time estimates from the veterans of complex embedded projects, here’s how much time is needed to develop and test a complex consumer device today.

12-18 months hardware development
12-18 months software development
6-12 months system integration, verification and validation
For a grand total of: 36-48 months device development and test effort

The reality is that those same teams report that their companies typically allocate 15-17 months duration for device development projects.

⁹

Whether engineering managers are guilty of bad math or wishful thinking, the disparity in these numbers suggests that something is seriously broken. Given this level of risk, it’s a miracle when a functioning, energy efficient product makes it to the marketplace on time.

Two conclusions become inescapable:

Sequential development of embedded hardware and software won’t meet product market windows
New tools and new methodologies must be adopted as a response to burgeoning complexity

The Shift Left: Using Virtual Prototypes to De-risk Device Development

First, let’s look to the not so distant past, when embedded software development was slower paced, sequential and dependent on the availability of physical boards. Those boards passed into the hands of software development teams like sacred artifacts proclaiming, Let there be software! Until they did, nothing happened on the software side of creation. Of course, boards cost quite a bit and were rather unwieldy, so software teams had to wait their turn to have access to one. That in itself could take a long time.

The process looked something like this.

Until recently, embedded software development commenced only after hardware became available

When the software content of an embedded system was relatively small, it was safe to wait for hardware to become available to start developing the software. Teams that wanted to get started sooner used RTL to represent the hardware, but that process was painfully slow. For thirty years, give or take, this was the status quo. Then things got complicated.

The risks that growing complexity exert on device development schedules have led forward-looking semiconductor companies and OEMs to embrace new approaches to developing embedded software. While the basic tasks—develop, move to target, debug—remain the same, the use of virtual prototypes lets software teams get to work long before hardware becomes available. When hardware and software teams work in parallel, it creates feedback loops that reveal and make it possible to correct problems throughout product development.

This is the “Shift Left”: a move from hardware and software development occurring sequentially and in isolation to hardware and software teams working in lock-step, communicating regularly, and continuously integrating hardware and software. This not only keeps problems from compounding, but also expedites the work that remains once boards become available. The Shift Left is a methodology innovation that’s both necessitated and abetted by technology advances. Virtual prototyping is the tool that makes it possible.

It looks something like this.

With the “shift left,” hardware and software teams work in parallel.

Some Benefits of Shifting Left

Long before chips come back from fab and before boards are built and debugged, software teams receive a fast, fully functional software model of the system. The model exactly replicates the target system, giving software developers full visibility into and control of the multicore platform during debugging. Because they’re virtual, all system components are accessible and debug and test activities are non-intrusive.

With virtual prototypes, a common golden reference model is easily and inexpensively shared among globally distributed hardware and software teams. Curating the system becomes a software discipline. Prototypes change and evolve during development, but they never break or wear out.

Breaking dependencies between hardware and software lets software teams choreograph the complex dance among cores required to perform high bandwidth tasks without touching or, heaven forbid, breaking the hardware. Programming to optimize performance while managing energy consumption is an emerging skillset, some would call it an art form, so being able to practice without negative impacts has real value. Virtual prototypes give software teams the time and tools they need to develop and test multiple usage scenarios— scenarios that are deterministic and repeatable until the best power management solution is achieved.

Time, Money and Parallel Development

Shifting left both saves money and increases revenue. Shorter development cycles reduce overhead and get products to market faster. Better tested products reduce the risk of recall. Most companies adopting virtual prototyping solutions can expect to experience economic benefits similar to those shown in the chart below. In this example, earlier development and production starts result in a $50 milliion revenue gain over the production life cycle.¹¹

What exactly is a virtual prototype? What can it do?

A virtual prototype is a fast, fully functional software model of a system under development that executes unmodified production code. It is both deterministic and scriptable, and combines fast processor models, called instruction-set simulators, and register-accurate models of both on-chip peripherals and off-chip devices, to assemble a binary compatible model of the actual SoC(s) and the system as a whole.

This chart compares net profit and break even experienced from use of virtual prototypes, depicting usage with and without virtual development kits (VDKs). Figures reflect the use of Synopsys products.

In embedded software development, it acts as the target system, allowing teams to debug and analyze software in advance of hardware availability using their familiar software development tools, and enables hardware/software integration in the absence of physical boards. It provides improved visibility and control where these are limited in physical systems, and is non-intrusive, making it possible to insert diagnostic probes in the hardware without changing the hardware. Scripting makes it possible to program “what if” scenarios and perform regression testing pre-silicon.

Users and purveyors of virtual prototypes agree that nearly 100% of embedded software development can be completed in advance of board availability. In fact, software development can begin before the virtual prototype itself is complete. The ability to grow incrementally as development progresses is a virtue of virtual prototypes, at each step modeling and providing access to just what’s needed to perform the tasks at hand. The prototype doesn’t need to be complete to be usable but instead evolves as development proceeds.

Teeing up the ecosystem

While accelerating development and improving software quality are key motivators for adopting virtual prototyping as a tool, users are beginning to leverage their prototypes long after the develop-debug-integratetest cycle is complete. OEMs are dependent on the availability of drivers and middleware to start developing their device-differentiating IP. Today, semiconductor companies are providing prototypes (as well as Virtual Development Kits with sample software, tools, virtual prototype and installer) to their OEM customers to give them a head start on device software development. OEMs in turn share protoypes with top tier service providers to win sockets and with 3rd party software vendors where interoperability is essential.

Jumpstarting next-generation products

A virtual prototype is not a one-off investment. It can be revised and re-used on the next generation of a chip or a device, with two important effects. The first is to help ensure a smooth transition with no loss of quality or customer satisfaction. The other is to give product designers a whole system view as well as a virtual sandbox for envisioning and trying out changes and innovations with minimal R&D investment.

Who uses virtual prototypes today?

Given their position in the device supply chain, it’s not surprising that semiconductor vendors are enthusiastic adopters of virtual prototypes. Software teams at semi companies are de facto innovation centers today, the hot spot where complex programming challenges come to get solved. These growing software teams are charged with making complex SoCs work, preferably out of the box, for their OEM customers. Under this kind of pressure, failure to innovate becomes failure to thrive.

Case Study: Altera

Altera used virtual prototyping to create a virtual target of their new Cyclone® V and Arria® V SoC FPGA devices. The Altera® SoC FPGA Virtual Target is a fast functional simulation of a dual-core ARM® Cortex™-A9 MPCore™ embedded processor development system.

This complete prototyping tool, which models a real development board, runs on a PC and enables customers to boot the Linux operating system out of the box. Designed to be binary- and register-compatible with the real hardware that it simulates, the Virtual Target enables the development of device-specific, production software that can run unmodified on real hardware.

Altera’s software development teams and partners have used the Virtual Target to port operating systems such as Linux and VxWorks as well as to develop complex device drivers. The teams found that not only did the Virtual Target enable much earlier software development, but also provided higher debug productivity.

In the Case of Cars

According to The Economist¹², today’s vehicles typically have somewhere between 30 and 100 electronic control units (ECUs) and at least 10 million lines of code. Software content accounts for 40% of the value of automobiles today, while a recent JD Power and Associates study of 2011 auto quality confirms that software problems are car owners’ most common complaint.¹³ With dozens of small computers on board governing virtually every subsystem, vehicle performance and safety are closely linked to software quality.

In cases where human life is at stake, penalties for failure escalate and international regulations kick in; an automotive product recall can damage a company’s reputation, and revenues, for a very long time. The need to validate vehicular software, as much as the mechanical engineering complexity and R&D expense involved, slows the automotive industry’s model development to a 7 to 8 year cycle. The high unit price and “consumer” nature of the purchase make for a buy cycle of 10 years or a minimum of 150,000 miles.

Virtual prototypes have become an important quality-enhancing technology for the automotive industry. By accelerating software bring up and hardware/software integration, virtual prototypes can improve quality and lower cost. But their greatest value is unique to the automotive industry—the ability to simulate and test potentially deadly corner cases in software without smashing real cars or risking test drivers, to inject faults and regression-test outcomes, to integrate hardware and software early in development and put those integrations through their strenuous paces.

Used in concert with electro-mechanical test harnesses, virtual prototyping can help auto manufacturers produce safe and fuel efficient vehicles—a step toward reducing recalls and loss of revenue.

In Conclusion: Trends in Parallel Development

Whether device teams use open source emulators or one of the several commercial solutions now available, they are embracing virtual prototypes, making this one of the fastest growing segments of the embedded tools market. As this¹⁴ graph of tools currently in use shows, virtual prototyping, simulation and system modeling tools are becoming common in the embedded toolbox. Adoption is growing in every market where time, cost and quality are threatened by complexity—in mobile and wireless, consumer, industrial and networking, companies are going virtual to survive.

This in turn drives innovation in the tools available for parallel hardware and software development. Large libraries of market- and device-appropriate models are beginning to reduce the time and cost involved in prototype creation. Virtual Development Kits (VDKs) bring together the elements needed to expedite development—Models and Transaction Level Model (TLM) creation tools, debugging and analysis tools, optimized operating systems and reference software stacks for particular device types. These elements are packaged with an installer to make the VDK easy to get up and running.

As virtual prototyping tools proliferate up and down the device supply chain, so expertise accumulates. First, the prototypes themselves mature with use and incorporate models of the IP of their owners across product generations, increasing their value over time. Second, the industry as a whole accumulates a valuable store of best practices and tribal technical knowledge that facilitates innovation. When old ways fall short, new tools arise to solve new problems. Virtual prototypes are that kind of impasse-smashing tool.