automated testing of embeded system

AUTOMATED TESTING
OF
EMBEDDED SYSTEMS
 ABSTRACT

Embedded systems are part of crucial real-time applications and are widely used in complex, time crucial projects like rocket launching and nuclear power development projects. So whatever be the design complexity and sophistication, the designed systems should be put to use with utmost guaranteed performance unless which huge losses can incur.

Embedded system is simply a combination of hardware and software that performs the component of a larger system. An embedded system employs a combination of hardware & software (a “computational engine”) to perform a specific function; works in a reactive and time-constrained environment. A software component called as Real-Time Operating Systems makes the hardware counterpart work accordingly.

This paper having considered various issues on testing strategies concludes with a bias on implementing the sophisticated automated testing tools into the testing phase of various design projects. The choice of testing strategy manifests with the designer and hence care must be taken in choosing the appropriate strategy for testing the design.

The more appropriate the automated testing strategy, the more powerful will be the system.
CONTENTS:~

 INTRODUCTION
 DESIGN OF EMBEDDED SYSTEMS
 WHY SO MUCH FOCUS ON TESTING
o ANCHOR THE INTERFACE FIRST
o TYPE OF INTERFACES
 GENERALIZED TESTING TECHNIQUES
o STATIC TESTING
o STRUCTURAL (“WHITE BOX”) TESTING
o BEHAVIORAL (“BLACK BOX”) TESTING
 WHICH IS BETTER: MANUAL OR AUTOMATED?
o WHEN TO FOCUS ON MANUAL TESTING
o WHEN TEST AUTOMATION MAKES SENSE
 TEST AUTOMATION
o RUNNING A TEST
o FUNCTIONALITY OF AUTOMATED TEST BENCH
o TESTING WITH AN ORACLE: AN ALTERNATE STRATEGY
 EMBEDDED SOFTWARE
o STRATEGIES FOR TESTING EMBEDDED SOFTWARE
 AUTOMATED TESTING TECHNIQUES FOR EMBEDDED SOFTWARE
o TEST SCRIPTING
o FRAMEWORKS
o INDIVIDUAL TEST PROGRAMS
o CHOOSING A RIGHT STRATEGY
 REAL-TIME CONTEMPORARY TOOLS FOR TEST AUTOMATION
o TESSY
o VERTAF
 CONCLUSION
 REFERNCES


 INTRODUCTION
Embedded system is simply a combination of hardware and software that performs the component of a larger system. An embedded system employs a combination of hardware & software (a “computational engine”) to perform a specific function; works in a reactive and time-constrained environment. A software component called as Real-Time Operating Systems makes the hardware counterpart work accordingly.








 DESIGN OF EMBEDDED SYSTEMS

An embedded system typically comprises the hardware, embedded RTOS, device drivers, communication stacks and embedded application software.





The development of embedded systems is a crucial area of responsibility in industrial practice. Embedded system functionality should meet sophisticated real time constraints unlike ordinary control system design. This adds a new dimension to the design process of a system.
 WHY SO MUCH FOCUS ON TESTING?
Though testing is a complementary aspect of design process, still why so much is to be focused on testing an embedded system? Embedded systems are part of crucial real-time applications and are widely used in complex, time crucial projects like rocket launching and nuclear power development projects. So whatever be the design complexity and sophistication, the designed systems should be put to use with utmost guaranteed performance unless which huge losses can incur.
o ANCHOR THE INTERFACE FIRST
One of the most important things with the embedded testing is going to be the interface we choose. The interface will define what testing you are able to do. So wherever we put this interface and how we configure it will be very important. It’s going to limit and focus the amount of testing or its direction. Defining the interface is very important to the testing effort. With embedded systems, this is important because if the focus is wrong, we will start with a wrong view of the system and the testing that we try to implement will be very unproductive. Probably we will not be able to accomplish what we wanted to do and still spend a lot of time implementing the wrong test and hardware.
Here we briefly discuss the aspects on which interface is chosen.
o TYPE OF INTERFACE
TYPE 0 interface:
The Type 0 interface is only the microprocessor board. The outside hardware does not exist. Here we are typically interested in verifying the actions and the responses on the board and to the software running on the board.
TYPE I interface:
The Type I is where the interface boards beyond the microprocessor exist, but all the outside signals are simulated.
TYPE II interface:
In a Type II system there is a single board in the system that is missing or is simulated.
This implies that the board provided to the system is virtual so that all sorts of exception handling can carried out which could not be possible otherwise.

 GENERALIZED TESTING TECHNIQUES
The choice of right technique is critical to achieve good test results. Some techniques involve analyzing the structural organization of the system while some involve analyzing the system’s behavior. Some tests require inside knowledge of how the system works while others require an understanding of what the system does.
Some of the generalized testing techniques are discussed below.
o STATIC TESTING
A static test is one that evaluates the quality of the system without the system actually running. This may seem very strange but still this can be regarded as a very basic test.
How can we run a test without actually running the system?
A static test looks at the system or artifacts of the system to see if we can find problems early. Regarding hardware, the architecture as well as the structural organization of components building the system is verified. When we turn to embedded software, desk-checking is carried out to review the code after writing it.
o STRUCTURAL (“WHITE BOX”) TESTING
Static testing is a powerful—but not a sufficient—test technique. At some point we must actually run the system and look for bugs in it. Tests that involve running the system under test are called dynamic tests. Structural tests are one major example of dynamic tests. Structural tests are based on how the system is built. Structural tests are most typically applied to individual components and interfaces, being particularly effective at discovering localized errors in control and data flows.
o BEHAVIORAL (“BLACK BOX”) TESTING
Behavioral tests, as the name “black box” implies, focus on what the system does, not how it does it. Behavioral testing most typically focuses on global issues of workflows, configurations, performance, and so forth. Because of the focus on how the system works, behavioral tests can cover many of the important usage profiles and quality risks.


 WHICH IS BETTER: MANUAL OR AUTOMATED?
Having seen the general testing techniques, we now turn towards the decision of choosing testing suite: Manual or Automated?
Automated Approach can be implemented on some tests with most efficiency while others demand manual. On the other hand, some kind of tests can’t be done manually while some test automations proves to be risky. Hence a right decision should be incorporated for successful testing.
o WHEN TO FOCUS ON MANUAL TESTING
High per-test or maintenance costs are one indicator that a test should be done manually. Another is the need for human judgment to assess the correctness of the result.
The following tests are good fit for manual testing:
• Installation, setup, operations and maintenance
• Configuration and compatibility check
• Error handling and recovery
• Localization test where human decision is significant
Besides, the following benefits can be derived from manual testing:
• Faster test development: Test methodologies can be development quickly and instantaneous with regard to contemporary conditions.
• Fewer resources: Manual testing can be deployed where fewer resources are available.
• Strong regression tests: Tests can be fragmented to the most fundamental level so that study of system artifacts is easy and consistent.
• New tests: Some systems demand different testing strategies under different test conditions. So new tests can be developed and used manually when and where it is required.
o WHEN TEST AUTOMATION MAKES SENSE
Characteristics of tests that can be automated are summarized below
• Regression and confirmation: Rerunning a test against a new release to ensure that behavior remains unbroken.
• Monkey/random: Tests that fire large amount or long sequences of data, transactions.
• Load, volume and capacity: For systems that support tremendous load.
• Performance and Reliability: This aims at looking for flaky behavior of the systems.
In addition, test automation can be exploited in following aspects:
• Repeatability: Tests which are to be repeated more number of times can be simply automated.
• Streamlining: Tests can be automated for smooth and efficient testing.
• Consistency: Precise and accurate testing can be performed with automation.
• Leverage: Tests can be focused on a specific goal.
• Scope: Test scope can be extended to various test conditions.
For any given test, manual or automated, first-order cost-benefit analysis should be carried out. Though this financial consideration is a secondary issue, still it needs to be emphasized from a good professional designer’s point of view.
 TEST AUTOMATION
We now focus on automated testing process of embedded systems. Most embedded systems need to meet real-time requirements. This adds a new dimension to the testing of such systems – not only the logical behavior but also the temporal behavior of these systems requires thorough testing. The critical phases in a typical testing process are discussed below.
o RUNNING A TEST
• TEST SETUP phase should incorporate the simulation of ambient conditions, initial data input/output values and embedded program state.
• RUNNING TEST EXERCISE should actually carry out the specified task defined in testing process.
• CAPTURING the test recorded data should accompany COMPARISON with the expected results so that the system can be judged for success/failure.




Fig. A TYPICAL TESTING MODEL


o FUNCTIONALITY OF A FULLY AUTOMATED TEST BENCH
Now that we are aware of the test phases, we can look over what an automated test tool performs. It should be able to
• run two or more specified test cases
• run a subset of automated test cases
• launch test without any intervention
• set-up and record relevant test environment automatically
• run test cases with provided input test parameters
• capture relevant results
• compare actual with expected results
• report the result of analysis
Apart from these, the automated test bench should be able to test the system with almost every possible input state under every possible operating condition. This helps us in providing exception handling for erroneous conditions which in turn minimizes errors in real-time operation.
o TESTING WITH AN ORACLE: AN ALTERNATE STRATEGY




Fig. TESTING WITH AN ORACLE
Systems under tests are vulnerable to damages and this may not be feasible in high-cost projects. Instead, the test oracle can be deployed to emulate the system to be tested. This test oracle can be a software or low cost hardware emulator. All the test conditions can be equally imposed on the test oracle so that the system behavior is equally tested, analyzed and reported.
Test Oracles adapt different strategies to verify test results. These strategies are compared and tabulated as follows.


Table: TEST ORACLE STRATEGIES FOR VERIFICATION

 EMBEDDED SOFTWARE

Embedded Software is one that is part of a larger system and performs some of the requirements of that system. It must be designed on a platform different from the platform on which the system is intended to be deployed. This component is also worth testing for much part of the embedded system functionality is incorporated within it.

o STRATEGIES FOR TESTING EMBEDDED SOFTWARE

• ‘No Oracle’ Approach is poor choice
• ‘True Oracle’ Approach may be used efficiently
• Software simulators , hardware & software emulators, cross compilers and debuggers can be potential consistency oracles
• Self referential approach is not usually adaptable.
• Heuristic approach may sometimes be required and used.
 AUTOMATED TESTING TECHNIQUES FOR EMBEDDED SOFTWARE
o TEST SCRIPTING
Scripting is generally a user language particularly aimed at testing the software independent of tool involved. These scripting tools can be tailored depending on the context and requirement. This technique doesn’t provide playback or recording mechanisms. The user should manually spy on the behavior of the system during or after test. Though the scripting can be done across different environments, it is environment dependent.
o FRAMEWORKS
Framework is the standardization of typical testing subroutines. This is an enhanced over simple programming tasks used for testing. It includes a subroutine library that houses common, complex, repeated actions that are deployed in testing process. It enforces discipline in the operating sense and efficiency. It can be evolved as needed and can leverage the prior work so that the framework fortifies. It could be efficiently used if maintenance cost can be afforded.
o INDIVIDUAL TEST PROGRAMS
These are simple test programs written explicitly focusing on a particular task of subsystem. They can evolve as independent tests and can be implemented right from the very beginning of design process. A complex test activity can be separated into fragments so that the testing process is simple and comprehensive.
Apart from these, strong environmental dependencies and high maintenance costs makes it unsuitable for certain projects.

o CHOOSING A RIGHT STRATEGY
Having known all the strategic aspects of automation, but still the decision of choosing a right strategy is vague and could be said that the choice is mere appropriate and can’t be exact. The following aspects must be considered for choosing an appropriate strategy
• Understand the development and platform environments
• Evaluate the automation options
• Identify the oracle strategy
• Consider costs and benefits between different strategic options

 REAL-TIME CONTEMPORARY TOOLS FOR TEST AUTOMATION

Current technology in designing real time software is still immature. Very rudimentary trail and error design techniques of developing everything from the scratch are in use.
Furthermore, the testing of embedded systems is more complex due to several specific technical characteristics such as the development in host-target environments, the intense interaction of the systems with the real application environment, and the limited resources of the target system. In order to facilitate systematic and largely automated testing in defiance of the complexity of real time systems powerful testing tools are required. Here we discuss on two evolved tools- TESSY and VERTAF.

o TESSY
TESSY has been extended in order to support the total testing life-cycle of real-time tasks. New components allow a thorough examination of the logical as well as the temporal behavior of the tasks. The logical behavior is tested by means of function oriented and structure-oriented testing methods; the testing of temporal behavior is automated by evolutionary testing. TESSY concentrates mainly on test case design,
test execution, monitoring, test evaluation, and test documentation. TESSY automates all test activities except the test case generation for examining logical program behavior. In order to automate the test execution, the required test drivers are generated, communication between host and target system is automatically built, the program code is instrumented and coverage analysis is performed, and the execution times on the target system are measured. Regression testing is also entirely automated by TESSY.

o VERTAF
A new application framework called Verifiable Embedded Real-Time Application Framework (VERTAF) is proposed for embedded real-time application development, with the aim of reducing design errors and increasing design productivity. It is a blend of three technologies- Object Oriented, Software component and Formal Verification technologies. It incorporates a model checking, which is defined as follows-Given a real-time system description S and temporal property specification φ, model checking answers if S satisfies φ.
Using VERTAF, a developer can devote more time and effort to the actual application tasks, instead of real-time system peculiarities. Even program verification can be accomplished automatically by VERTAF since it has integrated formal verification into its design process. VERTAF is modularized into five software components that can be used at different stages of application development namely, Implanter, Modeler, Scheduler, Verifier and Generator.
Given a software application to be designed, an engineer identifies and specifies the objects that are specific to the application using the Implanter component, in which a uniform object model called Autonomous Timed Object (ATO) is provided. The application objects are then transformed by Modeler component into uniform process models, each of which represents a real-time task. Scheduler checks the schedulability of the tasks and schedules them using a scheduling algorithm. Verifier proves the feasibility of the scheduled set of tasks by showing if they satisfy all given real-time and embedding constraints. Generator generates code based on decisions made by other components.

 CONCLUSION
This paper having considered various issues on testing strategies concludes with a bias on implementing the sophisticated automated testing tools into the testing phase of various design projects. The choice of testing strategy manifests with the designer and hence care must be taken in choosing the appropriate strategy for testing the design.S
The more appropriate the automated testing strategy, the more powerful will be the system.


REFERENCES:

 Encontre, Vincent; “Test, Observe, and Assess Embedded Applications During Development,” International Conference on Software Test Automation, Spring 2002.

 Hoffman, Douglas; “Using Test Oracles in Automation,” Pacific Northwest Software Quality Conference (PNSQC), 2001

 IEEE Std 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology

 Wegener, J. and Pitschinetz, R. (1994): TESSY- Yet another Computer-Aided Software Testing Tool? Proceedings of the European International Conference on Software Testing, Analysis & Review Euro STAR ’94, Bruxelles, Belgium.

 Grochtmann, M. and Grimm, K. (1993): Classification Trees for Partition Testing. Software Testing, Verification & Reliability, vol. 3, no. 2, pp. 63-82, Wiley.ry Testing.