Graph based test case generation for TLM functional verification
Introduction
Conventional RTL simulation is no longer fast enough for verification of today’s complex system-on-a-chip (SoC) designs. Comprehensive verification of such sophisticated designs needs not only verification of the hardware implementation, but also verification of the embedded software. To achieve the necessary simulation performance, we must move to a higher, more abstract level of modeling. Transaction level modeling (TLM) [2], [8] in SystemC [1] achieves 100x improvements in the simulation speed and enables architectural analysis and hardware/software co-verification.
Highly dependable construction of system level designs and proving their correctness are particularly important and arduous. Simulation is a practical and attractive method to achieve this goal. Although TLM expedites the verification of a hardware design [10], [11], [15], the problem of having high coverage test cases remains unsettled at this level of abstraction.
Admittedly, testbench development for SoC designs [7] requires even more effort than the design itself. The complexity of the code requires the use of a sophisticated software development environment.
To prove the correctness of a design by simulation, we must utilize all cases in the design input domain exhaustively. However, as this is not practical, we rather execute a test target program with test-data which are elements selected from a subset, according to some conditions on the input domain. These conditions are called test cases. Test cases play an important role in determining the quality of a design. They must be written carefully and if their number is not adequate, it is likely that some bugs will remain undetected in the design. In addition, test case overlaps would increase the verification cost.
To perform the verification in a minimum period of time, we need some metrics to detail the conditions. These metrics are called coverage metrics. At the evaluation step, by examining the coverage, we can determine how much of the design has been verified. By clearly setting up the coverage metrics before we start verifying a design, we can generate test cases that will satisfy these metrics.
Taylor and Kelly proposed the concept of structural testing of concurrent programs [3] in software domain. They defined the concurrent state graph as a model of a concurrent program in software. Some EDA tools such as Specman elite [9] of Cadence provide test generation with assertion coverage. They offer a user defined constrained random simulation for RTL designs. Jindal and Jain in [5] describe a methodology for verification of TL models by using RTL Testbenches. Habibi et al. in [4], [12] proposed a method based on FSM generation for TLM verification. However, they did not offer a methodology for generating test cases for it. Bombieri et al. [6] proposed a technique for reusing TLM testbenches at RT level with respect to both fault coverage and assertion coverage. Nevertheless, they used already written TLM testbenches and did not explain how these testbenches are obtained. To the best of our knowledge, our previous work [15] is the first in generating test cases for functional verification of TL models. However, in that work we assumed that the TL model does not have any wait statement or synchronization statement. In this paper, we extend our graph to cover these kinds of statements and apply our method to a second case study.
The motivation behind this work is to investigate some techniques to provide high-quality test cases for the verification of an SoC in TLM domain. To show that these test cases are satisfactory, we define some coverage metrics. We introduce the Control-Transaction Graph (CTG) which is a directed graph to generate the appropriate test cases and coverage metrics.
In this paper we propose a technique for generating test cases, which satisfies some defined coverage metrics, based on the SystemC code. We use the Control-Transaction Graph (CTG) as a model for Transaction Level designs. The CTG describes design modules as Control Graphs and Transactions between modules. We generate test cases based on the CTG. Some coverage metrics are defined based on this graph (i.e. CTG) to check the quality of test cases and to control the verification flow.
The rest of this paper is organized as follows: the next section represents the Control-Transaction Graph (CTG) as a model for transaction level designs. Section 3 describes our technique for generating test cases by using CTG. Section 4 defines coverage metrics for Transaction Level Models. Section 5 describes the algorithms which generate test cases on the CTG. Section 6 gives an outline of our test case generation tool. In Section 7 we apply our method on two real SystemC models as case studies. Finally, the last Section concludes the paper.
Section snippets
Representing a TL model by CTG
In this section we introduce the CTG describing the behavior of a transaction level model. A CTG includes two parts: Control Graphs and Transactions. Each Control Graph represents a module independently and transactions show the relations between modules.
Test cases for transaction level models
To generate test cases from a CTG, we initially consider test cases for a single module on a Control Graph.
Coverage metrics
In Section 3 we defined DePaths as test cases for a TL model. To generate paths, a trunk path in a CG and an exchangeable subpath in the trunk path are used. In presence of a loop in a CG, there will be an infinitive number of generated paths. Coverage metrics specify not only termination conditions of test case generation but also the reliability of simulation. If we verify a design based on a specific coverage metric, we expect to detect some types of errors in the design. We adopted simple
Algorithms
We introduce two algorithms to generate test cases from a CTG. The first one describes path generation and the second one DePath generation.
TLM test case generation tool
We have developed the TLM test case generation tool to implement the algorithms of Section 5. The tool generates DePaths from the source code written in SystemC. The tool consists of five steps as shown in Fig. 2. These steps are as follows:
- 1.
Generating Control Graphs: all Control Graphs of all sc_modules in SystemC as well as transaction table are generated.
- 2.
Making Control-Transaction Graph: the whole CTG consisting of Control Graphs and transaction table is produced. The Control Graphs made in
Case studies
Two case studies from real AMBA and JPEG designs are used to show the effectiveness of our approach. The Advanced Microcontroller Bus Architecture (AMBA) specification [13] defines an on-chip communications standard for designing high-performance embedded microcontrollers. To evaluate our test case generation methodology, we consider the Advanced High-performance Bus (AHB), one of the buses defined within the AMBA specification and JPEG encoder which is a method of compression commonly used for
Conclusion
We introduced a Control-Transaction Graph to describe the principle of test case generation for simulation of a Transaction Level Model. A Control Graph is the control flow graph of a module in a design and transactions represent the interactions such as synchronizations between the modules. Based on the concept of Control Graph and Transactions we set up a TL model with a CTG to generate test cases. The number of overlapped or insufficient test cases decreased since the test cases generated by
Acknowledgements
We would like to thank the TLM group within the CAD research center at the school of ECE, University of Tehran for the concept of “event rate” and “number of events” used in Table 1, Table 2.
References (15)
- Open SystemC Initiative:...
- et al.
Transaction level modeling: an overview
Proc. of CODES+ISSS
(2003) - et al.
Structural testing of concurrent programs
Proc. of Workshop on Software Testing
(1986) Efficient assertion based verification using TLM
Proc. of DATE Conference
(2006)- et al.
Verification of transaction-level SystemC models using RTL Testbenches
Proc. of MEMOCODE Conference
(2003) - et al.
On the evaluation of transactor-based verification for reusing TLM assertions and testbenches at RTL
Proc. of DATE Conference
(2006) - S. Paricha, Transaction level modeling of SoC with SystemC 2.0, Synopsys User Group Conference,...
Cited by (2)
A systematic approach to configurable functional verification of HW IP blocks at transaction level
2012, Computers and Electrical EngineeringCitation Excerpt :TBs are generated using the SystemC verification library while the coverage metric is not specified. Kakoee et al. ([13]) propose a transaction-level verification of the communication between a system of IPs using ABV. A new coverage metric is proposed accompanied by a TB generation proposal.
Modeling of transaction level systemc modules and transactional channels with petri nets
2008, 2008 2nd International Conference on Signals, Circuits and Systems, SCS 2008