Verification IP Qualification and Usage Methodology for Protocol-Centric SoC Design

by  Michael T. Horne, Verifica LLC, Portland, OR

Verification in the Nanometer Era

With the continued progression of chip geometries to ever smaller sizes, designers are finding themselves with a wealth of available gates in which to create their latest designs. New generation devices like smartphones and network appliances are blazing trails in the 90 and 65 nanometer processes with massive gate counts and unprecedented integration of IP blocks (Figure 1). Product differentiation in the form of advanced features and novel configurations of common functions is becoming the only viable competitive advantage. With design-from-scratch totally out of the question, designers now build these systems with off the shelf IP blocks that are pre-designed and verified, helping them meet their goals of differentiation, cost control, and time to market.

Figure 1. Standards-based design. Design and verification is simplified when systems are centered on industry standard protocols and assembled with offthe- shelf IP. Shown is the NetSilicon NS9750 network processor.

The ability to assemble systems with Implementation IP, or IIP, is predicated on a reliable way to connect them together. To simplify interconnection, designers naturally gravitate to IIP blocks that utilize industry standard busses and protocols. The most popular IIP today are processors like ARM, MIPS and Power-PC, and the many standardsbased on-chip and off-chip communications protocols like PCI Express, AMBA bus, and USB 2.0. While leveraging IIP dramatically reduces system implementation time, verifying the system continues to grow in complexity and effort. Quality IIP comes pre-verified, but responsibility for verifying the correct operation of the system assembled from the IIP still falls on the designer. Simply put, engineers must ensure that their designs will meet the industry bus and protocol standards that they are built upon. To help combat the exploding verification problem created by increased system complexity, designers are turning to Verification IP, or VIP, for help. VIP blocks are well-tested simulation models of industry standard busses and protocols that generate and respond to stimulus and check protocol rule adherence. As part of a modern system verification methodology, VIP reduce system verification time and improve quality -- just like Implementation IP.

How VIP Shorten the Design Cycle

Modern chip development methodologies typically follow a progression of turning simple conceptual product ideas into rather complex implementations. Surprisingly, many design teams still think of the verification task as something to be done after the first wave of RTL coding is completed. However, the most successful methodologies today ensure that verification is properly tended to in each phase of the development process.

Figure 2. Using VIP to explore SoC architectures. Early architecture exploration can be accomplished by creating high-level behavioral models of the SoC’s core blocks, leveraging VIP to quickly implement block interfaces and generate interesting bus traffic.

Phase 1 – Architecture Exploration

Scalable design methodologies typically begin with the exploration of the product architecture where implementation tradeoffs are made while attempting to meet the product functionality goals. To validate assumptions about system performance like data throughput, required memory structure and protocol stack flows, architects can quickly create models of major components of the system to simulate system operation at the transaction level. Using efficiently implemented VIP and Vera or C models of core blocks, architects can rapidly gain confidence in the system structure and performance at this crucial stage of development (Figure 2).

Phase 2 – Block Implementation

As the chip architecture firms up, an implementation strategy is defined and blocks are handed off to individual designers for implementation. Practical market considerations require large blocks to be purchased off-theshelf, like processor IP, or reused from previous designs. However, some logic is usually created from scratch, like control blocks and other specialized logic. Functional verification of these blocks is simplified by using VIP to ensure their compatibility with the bus and protocol standards of the system. For example, say a designer will create a block that accepts Ethernet packets, manipulates the data at layer 3, and makes the data available on an AMBA slave interface (Figure 3). By leveraging Ethernet and AMBA VIPs to create interesting protocol-compliant traffic, the designer can quickly and accurately verify his design for proper data handing and compliance to the industry standards. Attempting instead to create a test environment from scratch would take dramatically longer and greatly lower confidence that the design block will properly meet the protocol standards.

Figure 3. Block verification using VIP. Verifying an RTL block is simplified when VIP are used to create interesting protocol-compliant traffic to ensure proper block and interface operation.

Phase 3 – System Integration and Verification

Full system verification commonly starts after individual blocks are created and pass block-level tests; however, more powerful design methodologies start system verification at the architecture definition stage. By leveraging VIP to create high-level models of blocks in the system, interesting system-level testcases can be created early in the development cycle (Figure 4). Then, as individual blocks are implemented and integrated into the system, RTL or core models can be swapped in and out as needed to test for proper system-level operation. By leveraging VIP, this block-based system verification method allows system testcases to be written and debugged early, substantially shortening the product development cycle.

Figure 4. System-level verification with VIP. System-level verification using a block-swap methodology maximizes verification efficiency. Behavioral models created with VIPs (phase 1) can be swapped in-and-out with RTL blocks to confirm proper RTL operation in the system.

VIP Options: Roll-Your-Own, or Off-The-Shelf?

Simply put, VIP deliver the biggest productivity gain possible when verifying large SoCs and systems designed around industry standard protocols. “Considering that verification consumes 50 percent to 80 percent of the total development effort, verification reuse brings tremendous benefits to the verification team,” says Steve Ye, a senior engineer in the IP Reuse Design & Development group of Agere Systems1. The age-old method of creating a verification environment from scratch is no longer a practical or safe option. Even when project development schedules are not an issue, the cost of a chip respin or a product recall due to a poorly verified design can be huge – if not fatal. When it comes to today’s protocol-based designs, the question isn’t whether to use VIP, but rather how best to leverage it.

There are two practical options for obtaining VIP when assembling a verification environment:

1. Roll-your-own VIP. You can tap into VIP building blocks previously created by internal project teams or CAD groups, or choose to create your own VIP from scratch with reusability in mind. Most projects that roll their own VIP typically start with code harvested from a past project. The complete verification environment is assembled from these locally created VIP, and testbenches are written to leverage these blocks.
2. Off-the-shelf VIP. You can leverage off-the-shelf VIP previously created and tested by EDA vendors like Synopsys, Denali and Verisity, or bundled with Implementation IP cores by IP companies and fabs like ARM, LSI Logic and TSMC. The better off-the-shelf VIP are optimized for performance and have a consistent user interface which simplifies the interconnection and configuration of blocks, thereby reducing the overall testbench writing effort. Many include assertion-based checkers as well.

It is not unusual for an SoC verification environment to be constructed using a combination of the above 2 options, particularly if off-the-shelf VIP are not available for key protocols used in the SoC. However, nowadays virtually all common and many leading edge protocols or bus standards have well supported and tested VIP available on the open market, reducing the need for designers to create common VIP from scratch.

“It's a worrying state of affairs when the verification IP is almost as complex as the design IP itself, but that's the stage we've reached,” says Clive Maxfield, a noted author and columnist for EE Times2. Still, a surprising number of engineers feel the need to roll their own VIP or create most of their verification infrastructure from scratch. Development teams who choose this route usually find that the effort required to create quality VIP is staggering – and often late in the product development cycle. When choosing to create VIP from scratch, several formidable challenges always emerge:

• A deep understanding of the protocol is required. When creating your own VIP, you need to learn the details of the protocol or bus standard to know what to test for compliance. Many protocols like AXI, USB 2.0 and PCI Express are quite complex, requiring a substantial investment in time spent learning. Some protocols like PCI Express have published lists of protocol conformance tests numbering in the hundreds – resulting in the daunting task of reading volumes of supplemental information and interpreting how suggested tests should be created.

• Supporting constrained random test methods is crucial. Writing code to test the full state-space of most designs is prohibitive these days, particularly with large, complex protocol-centric systems. To manage this explosion in verification complexity, many teams are using constrained random verification methodologies and languages that support them like Vera, SystemVerilog and Specman. Creating VIP for these modern environments requires deep knowledge in constrained random test techniques and a substantial investment in support infrastructure.

• Harvesting code is never as easy as it seems. Creating VIP by harvesting code blocks used on past projects can be costly if the code is poorly documented, or if access to the original designer is not possible. Once modified, verification blocks that were tested and assumed correct for past designs must be re-verified to ensure their proper operation. Harvesting legacy code blocks for VIP is usually false economy: in most cases it takes more time to harvest the code than it does starting from scratch.

•  Gaining model confidence is expensive. Kathy Werner, Director of Design Reuse at Freescale Semiconductor, sums it up well in a single sentence: “It requires a lot of resources and overhead to do a full test on IP”3. Any engineer who wants to create VIP from scratch should honestly ask himself the question, “do I really want to be my own beta customer, and do I have the time to do it?” Testing a newly created VIP to ensure conformance to the specification is highly complex and requires a massive number of simulation cycles. Building confidence and trust into a VIP requires rigorous testing and the ability to confirm its proper operation on dozens of typical designs. Confirming that all corner cases are covered and that the developer has properly interpreted the specification is crucial; the consequences of trusting an unproven VIP model can be disastrous for the project using the model.

• Support never comes for free. Once development of a VIP is completed, someone must take on the task of supporting the model, perform bug fixes and update the model to incremental revisions of the protocol specification. The recurring cost of supporting an internally developed VIP can easily add 20 to 30% to the initial development costs, and a failure to support it results in a quick drop in model use by future design teams – thereby eliminating much of the perceived value of developing VIP internally.

• Developing VIP internally is rarely strategic. When one is tempted to develop VIP from scratch, it pays to ask the question, “What business am I really in?” Very few businesses would agree that internally developed and maintained VIP is crucial to their product success in the market. It gets back to the value proposition for your customer; customers pay for products that get to market early and with differentiable product features. Customers are simply not interested in paying for the time and cost incurred in developing and maintaining VIP internally.

Considering the raft of challenges faced when creating VIP from scratch, design teams and CAD departments are now gladly handing the development and support load off to commercial VIP developers. Commercially available VIP embed a tremendous amount of knowledge and capability that can be easily tapped into by virtually all developers of protocol-centric designs. When you take the total effort of developing and maintaining VIP into consideration, off-the-shelf VIP are an incredible value and the only practical way to get to market quickly.

VIP Quality: History Repeats Itself

To ensure that VIP deliver on the promise of accelerated time-to-market and reduced cost of SoC verification, the quality of the VIP and the supplier support must be superb. Unfortunately for VIP buyers, this is not always the case. George Bernard Shaw once said, “If history repeats itself, and the unexpected always happens, how incapable must Man be of learning from experience.” Sadly, some commercial suppliers of VIP failed to learn from past industry experiences with quality and support in the Implementation IP market.

In the late 1990s, design reuse was gaining momentum as a way to dramatically shorten product design cycles. The mantra of the time was “why create a design from scratch when you can buy it off the shelf?” Companies like ARM and MIPS were gaining momentum by delivering quality Implementation IP to the market. Around that time, a number of smaller, lesser known companies jumped on the bandwagon and introduced new IIP offerings to the market. While design teams were initially enthusiastic about the plethora of cores they could tap, problems began to surface for some users of IIP. Engineers found that, in many cases, the IIP they licensed from their vendors were poorly verified. They discovered that many of the blocks were difficult to integrate. In many cases, chips came back from fabrication non-functional or with other quirky corner case problems. An IIP quality crisis quickly ensued, and engineers pulled back from using off-the-shelf IIP.

In the years that followed, engineers became savvy in evaluating IIP for use in their designs. Convinced that IIP still had potential to save them a great deal of design time, they began to demand higher quality from IIP providers. When evaluating cores, they demanded to see the verification plan the vendor used to verify their cores. They wanted to know if the core was developed from scratch, or harvested from old designs and repackaged. They asked for proof that the cores had been taped out successfully. With time, the quality crisis subsided and now well-designed, tested and supported IIP are readily available from reliable suppliers.

Unfortunately, Verification IP has gone through a similar quality crisis over the past few years. As verification teams jumped on the verification reuse bandwagon as a proven way to shorten their ever-increasing verification schedules, new suppliers of VIP began to flood the market with a broad selection of VIP for industry standard protocols. Once again, verification teams found that much of the VIP available from small, obscure vendors were of poor quality and questionable origin. Many small providers were nothing more than independent consultants who lacked any real ability to support large customers. With the economic shake out of 2001-2003, quality VIP offerings began to consolidate under a few technology leaders like Synopsys and Verisity who now set the standard in quality, title availability and support.

The Four Dimensions of VIP Quality

When tightly integrated into a project or corporate design methodology, VIP dramatically simplify the verification task and greatly reduce the burden on tool and methodology support staff. However, VIP must be carefully selected and evaluated to ensure that projects can reap the substantial benefits that VIP can deliver. Many development teams have a natural tendency to focus on the features of a VIP while not spending enough time evaluating the quality and support. The consequences of selecting poorly designed or supported VIP go well beyond the simple cost of the VIP; problem VIP can affect the project schedule or even cause a chip re-spin or a product recall.

With some investigative work, it is possible to do a reasonable apples-to-apples comparison of VIP from disparate vendors. Fortunately for those evaluating VIP, industry organizations like VSIA have extracted the lessons learned from the past IIP quality crisis and published utilities that VIP users can tap for selecting quality VIP. While VSIA’s original focus was on IIP quality, they have taken care to include VIP under the umbrella of their Quality IP, or QIP, metric development4. Their collective work can be summarized as defining the necessary criteria for grading candidate IP on 4 critical dimensions of quality. Applied specifically to VIP, they are:

1. Authoring Process, or how the VIP was created.
2. Verification, or how VIP operation was verified to meet the specification.
3. Maturity, or how many times the VIP has been used in real designs.
4. Vendor Capability, or how well the vendor supports VIP users.

By focusing on these 4 key aspects of quality when evaluating VIP offerings from both internal and commercial sources, engineers and CAD managers can obtain a reasonable level of confidence in the outcome of their selection process.

Evaluating VIP: Walking Through the Process

When evaluating VIP from multiple vendors, it pays to follow a step-by-step procedure that helps reduce the total data evaluation load and time spent in evaluation. It is tempting to simply select a VIP from a single vendor and evaluate it by trying to integrate it into your environment; however, given the consequences of possibly finding a fatal problem with the VIP later in the design process, it is prudent to follow a methodical process to evaluate several VIP simultaneously. Doing so will yield the best decision in the least amount of time.

Figure 5. The Three Phases of VIP Evaluation. Candidate VIP are best evaluated by collecting common data from VIP vendors, comparing the data for capabilities that matter to the end user, then performing a final test integration to confirm the findings

The process of VIP evaluation can be reduced to 3 phases: Collecting data on the VIP; comparing the data; and running a real-world test (Figure 5).

Phase 1: Collecting the Checklist Data

To properly compare VIP from a disparate group of vendors, you need to ask for more than just a datasheet. To simplify the evaluation process, you need a checklist inhand when asking for data from all vendors, including internal VIP developers. If a vendor does not provide meaningful documentation or reasonable responses at this phase of the evaluation, odds are that their offering lacks the quality you will need, saving you the time of evaluating them further. In all cases, vendors who are serious about earning your business will be very responsive and generous with supplying the required information.

To aid your data collection, an evaluation checklist covering the four dimensions of VIP quality is shown in Appendix A. While not exhaustive, the checklist covers most of the key data you will need to do a reasonable comparison of available VIP for a given protocol. Whether the vendors are commercial EDA companies, consulting firms or internal VIP developers, all sources should be asked to provide the data listed in the checklist

Phase 2: Comparing the Data

By asking for specific data items from all vendors, you have an opportunity to compare their relative strengths and weaknesses under a reasonably consistent set of criteria. VIP feature sets, level of protocol support, documentation quality and after-sales support are all criteria that are best compared on an apples-to-apples basis whenever possible. Data collected from the vendors is perhaps best compared side-by-side in a spreadsheet (for yes/no and quantitative questions), as it is done with the VSIA QIP spreadsheet. However, some questions may require a qualitative analysis. For example, while you should expect all VIP vendors to deliver a Functional Specification, you will need to pore over the documents to ensure that the level of detail, quality of documentation, and design discipline is exhibited.

While it is tempting to compare VIP based upon only the technical feature set you currently need, it pays to “future proof” your decision by considering features that you may likely need to verify future designs. In some cases, such as with newly emerging standards, the feature sets of available VIP may be limited; in such cases, look closely at data collected relating to Vendor Capability and be sure to drill down on the vendor’s plans for supporting the evolving standard.

In some cases you may be comparing from a field of one – which simply means you’re faced with a make-buy decision which depends upon how well the single vendor offering meets the criteria of a quality VIP. While it is better to roll your own VIP than suffer the pain and risk of using a low quality VIP, you must objectively assess your own ability to create an alternative VIP when faced with a field-of-one decision.

While it is preferable that the VIP evaluation be fully objective, the simple fact remains that the vendors may supply data in differing formats or make statements that are difficult to compare. Nevertheless, when attention is paid to collecting detailed information from the vendors on all four dimensions of quality, your experienced design and verification engineers will be able to sift through the data and formulate a reasonably objective comparison.

Phase 3: Running a Real World Test

One or two VIP are usually identified as the top candidates from the selection process. To back up your evaluation, it pays to run a simple integration test of the top candidate VIP in your current environment, or in a test environment supplied by the vendor. The ordering of this phase is important; do it last. It’s tempting to jump immediately to this phase of the evaluation before taking a hard look at the candidate VIP in the first two phases (data collection and evaluation). It’s most efficient to perform the real world test after you’ve narrowed the candidate pool to only those VIP that meet the minimum quality and performance standards you must have to be successful.

The principal goal of this phase is to confirm that your findings in the first two phases are accurate. Critical evaluation points include ease of use, quality of the documentation when used in practical tasks, and how fast you can get to a confident thumbs-up/thumbs-down result for the design under verification. If you use a vendorsupplied example test environment, be sure that it is structured the way you would likely use the VIP; vendors will obviously provide example environments that show off the best features of their VIP and hide the flaws.

Summary

Engineers can achieve a tremendous jump in verification productivity by tapping into VIP when architecting SoCs, implementing their blocks, and performing system integration. But to realize that productivity boost, engineers and tool managers need to carefully evaluate the VIP offering from the principal vendors to avoid the false economy that accompanies VIP of poor quality and dubious capability. By following a structured methodology for evaluating VIP along the four dimensions of VIP quality – Authoring Process, Verification, Maturity and Vendor Capability – engineers can ensure that the selected VIP will deliver on the promise of shorter verification cycle time and greatly improved product quality.

Appendix A
VIP Evaluation Checklist

1. Authoring Process. Knowing how a VIP was created tells you a lot about the quality and maturity of the VIP, and the commitment of the vendor to delivering high quality products.
1.1 Documentation. Specifications and development plans used in the VIP development process can give you a clear understanding of the vendor’s attention to detail and completeness. Vendors should be able to provide copies or excerpts of their development documentation under NDA.
1.1.1	Requirements Specification	This document defines at a high level the key features of the VIP, which portions of the standard that are implemented, and what usage features are supported such as languages, environments, etc. This spec is sometimes combined with the Functional Specification.
1.1.2	Functional Specification	This document details every aspect of the VIP functionality including a listing of every command, parameter, error message, etc. This spec is sometimes combined with the Requirements Specification.
1.1.3	Test Plan	This document explicitly identifies how each of the items listed in the Functional Specification are tested.
1.1.4	Users Manual	This document provides detailed information on VIP usage including a detailed listing of commands, parameters, etc. The Users Manual should also document example usage in all supported languages. 1.2 Development Methods. How a vendor develops its VIP directly affects the VIP quality.
1.2.1	How are the development documents linked?	All development documents should be in-sync with each other. Out-of-sync documents reveal a lax development process.
1.2.2	Did the technology evolve piece-meal or was it designed with intent?	VIPs should be designed with a platform concept to ensure simplicity of support and ease of updating. VIPs that evolve from inferior base technology could be abandoned by the Vendor when support costs become unwieldy.
1.2.3	Was the VIP harvested?	VIP cobbled together from code blocks used on past projects or consulting engagements usually provide inferior functionality.
1.2.4	In what language is the VIP implemented?	VIP created from testbench languages typically provide the best feature set and are the most powerful.
1.2.5	Do they use revision control and bug tracking software?	The use of software development tools helps make the vendor more efficient and able to address bugs – resulting in shorter development and model support time
1.2.6	Do they perform portability testing?	Is the VIP tested across all supported simulators, operating systems, and languages for consistent operation?
1.3 Project Staffing. How a vendor staffs VIP development provides insight into the long-term commitment of the developer.
1.3.1	Where is the VIP developed?	The physical location of the developers, whether they are all in one place, and their time zone distance from your team can affect problem resolution time.
1.3.2	Is development performed inhouse or outsourced?	Outsourced development can imply a lack of commitment by the vendor and introduce delays in resolving model problems.
1.3.3	How many engineers are dedicated to VIP development?	Serious VIP vendors invest substantial resources in VIP development for generating new models and evolving model features.
1.4 Functionality and Portability. Not all VIP are created equal, as functionality and portability differences can dramatically impact your total coding effort and future flexibility.
1.4.1	How much of the protocol is supported?	Depth of protocol support can vary greatly across models.
1.4.2	Is full constrained random test supported?	Full support for random test methodologies ensures maximum usage flexibility and model evolution.
1.4.3	Is it partitioned logically?	Models that are partitioned into blocks that reflect how the protocol is typically used provide maximum usage flexibility.
1.4.4	How many languages are supported?	VIP restricted to use in a single language limits your ability to migrate usage to other languages and platforms in the future.
1.4.5	Are functionality, usage model and results consistent across all languages?	VIP that support multiple languages should have a consistent usage model and provide identical operation and results.
1.4.6	Is it compliant with emerging methodology standards?	VIP that support emerging standard methodology guidelines like RVM and VMM save verification implementation time.
1.4.7	Are all VIP available in a common Library?	Mature VIP vendors tend to include all of their models in a single library for customer ease of use and financial value.
1.4.8	Do they deliver a full VIP package?	A complete VIP release will include a fully documented Users Manual with examples in the languages you use.
2. Verification. Ensuring that a VIP conforms to the published specification is both crucial and non-trivial. VIP users need solid confidence that the VIP will perform to spec, or they simply won’t use it.
2.1	Is code coverage measured on the VIP?	Vendors should be able to meet modest coverage goals of 95% line coverage (with clear reasoning for the remaining 5%) and branch coverage greater than 80%.
2.2	Do they verify against a golden reference?	Some standards bodies provide golden reference models that can be used to validate the VIP’s core functionality to spec.
2.3	Do they perform portability testing?	Is the VIP tested across all supported simulators, operating systems and languages for consistent operation?
3. Maturity. The stability and trustworthiness of a VIP will be a direct function of how many times the VIP has been used in the real world.
3.1	Has the VIP been used on a taped-out design?	While some Vendors with new VIP offerings will provide early access to their models, mature VIP on the market will have a proven track record of 5 or more successful tape-outs.
3.2	Are customer references readily available?	Get at least 3 solid references from the vendor and call them. Check on how it was used, problems found, responsiveness to support requests, etc. You can find other references by checking the VerificatinGuild5, DeepChip6 or Usenet7 community forums.
4. Vendor Capability. Since most vendors prefer to license their VIP using time-based models, you must confirm that a vendor is financially stable, committed to delivering quality VIP and capable of promptly supporting you for the duration of the license period.
4.1	How long has the vendor been in the business? What is their primary focus?	VIP vendors who have focused for some time in this space, building up a real core competence in VIP, are more likely to deliver mature product and be more responsive.
4.2	How often are models updated?	A regular release schedule of updates is critical for newer protocols as they evolve over time. However, rapid releases (e.g. monthly) can indicate immature technology.
4.3	How many engineers are dedicated to VIP support?	The vendor should dedicate a reasonable number of full-time staff members in supporting customers.
4.4	Where are the support staff located?	Confirm the location of the support staff and evaluate whether a timezone difference will be a problem for you.
4.5	What is their service record like?	Try and obtain metrics from a vendor regarding issue response time, average time to resolution, etc.

1. Steve Ye, “Best Practices for a Reusable Verification Environment,” EE Times, 12 Jul 2004.
2. Clive Maxfield, “Max Bytes” editor’s column, EE Times, 16 Sep 2004.
3. Kathy Werner quoted by Peggy Aycinena, “QIP & the Art of Motorcycle Maintenance,” EDA Weekly, 18 Aug 2004.
4. The VSI Alliance published their QIP Quality Metric Spreadsheet in mid-2004. More information can be found at http://www.vsi.org.
5. The Verification Guild is a users’ forum focused on verification methodology and usage questions. Managed by Janick Bergeron, a noted verification expert, it is accessible online at http://www.verificationguild.com.
6. DeepChip is the online archive of John Cooley’s ESNUG moderated email forum. It is available online at http://www.deepchip.com
7. Usenet Newsgroups are open forum discussion groups that cover specific language and technology issues, and can be found online at http://groups.google.com. Principal groups of interest include comp.lang.verilog and comp.lang.vhdl.