In today’s article, we will shed some light on how to measure test code coverage.
Introduction to Test Coverage:
Code Coverage is a technique to measure how much the test covers the software and how much part of the software is not covered under the test. The tester is able to find out what features of the software are exercised by the code. Using the code coverage technique and number of bugs in the application we can build confidence upon the system on its quality and functioning. Test code coverage measures the minimum number of test cases that need to be executed to provide confidentiality in the system.
What is Test Code Coverage?
Code coverage measures how much code of the application is being exercised when the tests are run. This makes sure that the tests being run are actually testing the code of the application. The basic measure of Code Coverage is the “Coverage Item”, which can by anything we have been able to count and see whether it has been tested or not.
Measurement of Coverage can be determined by the following formula.
Coverage= Number of coverage items exercised / Total number of coverage items *100%.
It should be kept in mind that 100% code coverage does not mean that the application is 100% tested. An application with high code coverage means it has been more thoroughly tested and would contain fewer software bugs than an application with low code coverage. There are numerous ways to calculate code coverage like program subroutines and program statements called during the execution of the test suite.
What are the test code coverage criteria?
Coverage criteria are a method to measure how much code has been exercised by the test suite. The test suite must satisfy the rule of the coverage criteria. To Measure different Test Code Coverage understand the various Coverage criteria are as follows:
- Statement Coverage
- Decision Coverage
- Condition Coverage
- Path Coverage
Let us understand each of these in details
1. Statement Coverage: The statement coverage tells us whether each executable statement is covered. This technique is better than other techniques as the defects are evenly distributed over the entire code. Hence the percentage of executable covered can be found out using the percentage of defects found. Control statements like if-else, for and switch are covered only if the expression of the control statement is covered in the executable statements. The statement coverage does not work with some control structures. For If-Statement Structures, if ever the if the condition is false, that statement coverage method would fail. If there is no test case that causes the condition to be false, the statement coverage would state the code is fully covered. The loops that contain break statements are not covered by statement coverage. It does not tell whether the code reached its termination stage. For do-while loops, the code is taken as a nonbranching statement code by the statement coverage. There are many other shortcomings of the statement coverage. It does not work for logical operators like And and OR. It is also insensitive to the switch statements.
2. Decision Coverage: Decision coverage tells us whether the expressions in control structures came out to be true or false. The whole expression is taken as one true and one false regardless of whether it contains Logical AND and Logical OR operators. It also includes coverage of switch statement cases, exception handling, and all entry and exit points. The contact expressions which control the flow are ignored by decision coverage. Indecision coverage metrics all the Boolean expressions should evaluate to true and false even if it is not included in the control flow. But the exact definition of Boolean Expressions is not defined here. Some programming languages like C language can take both integer and Boolean together without requiring it to be declared Boolean. Generally, a Boolean operator is one with Boolean operand and Boolean output is taken as a Boolean expression. The decision coverage is also known as branch coverage, all-edge coverage. The decision coverage has many disadvantages like it ignores some Boolean expressions branches which occur because of short circuit operators.
3. Condition Coverage: It is quite similar to decision coverage but it is better than the decision coverage. It is more sensitive to control flow. In condition coverage, the conditions are taken independently of each other. It tells us about the true or false result of each condition. The condition is taken as the operand of the logical operator that does not contain a logical operator.
4. Multiple Condition Coverage: It tells us whether every possible combination of conditions occurs. The test cases for multiple condition coverage can be found out by logical operator truth table For languages like C, Java multiple condition coverage requires very thorough testing. This technique has some disadvantages as well. For any complex Boolean expressions, it is very cumbersome to find out the number of test cases required. Another disadvantage of this coverage is that the number of test cases varies among conditions that have the same complexity.
5. Path Coverage: Path Coverage tells us whether all the paths in each function have been followed. The path coverage determines the paths are possible combinations of logical operators. A path is a sequence of branches from the entry point to exit point. It is useful as it requires thorough testing. Path Coverage has disadvantages as the number of paths is exponential to the number of branches. Also due to the relationship of data many paths are not possible.
Test coverage metrics are less common than test case metrics and defect metrics. To produce a decision metric coverage, we need traceability from your tests back to the requirements or design elements they cover that is bidirectional between the test cases and each item on the test basis. Another type of metric Risk Coverage metric reports on risk coverage. For this coverage, we need meaningful and sufficiently fine-grained traceability between test cases and risk items. Another type of metric is the kind that looks at environmental and configuration coverage. This type can be particularly useful for shrink-wrap software and other mass-market software or systems that customers will use in many environments. We need to capture, for each important environmental factor, what option you selected for the test. Code coverage metric is almost never measured with a traceability table or database but rather with a tool that keeps track of the total code coverage achieved cumulatively as test cases are run.
These metrics are important because they allow a test manager to establish whether what needed to be tested will be or was tested. In the case of tests having been run against some requirements element, risk item, or configuration, we can then use traceability to trace the test result- pass, warn and fail or block- back to the underlying item you are trying to test, reporting its status.
These coverage metrics can serve as a surrogate metric for confidence. We can roughly measure the level of confidence we should have in a system by establishing to what extent we have covered it with tests and to what extent what we have covered actually works.
A mistake that the organizations make when they use coverage metrics as a surrogate metric for confidence is to rely on a single factor, like requirements coverage or code coverage, without considering other measures of coverage. Ideally, one should use multifactor coverage metrics-checking code coverage, requirements coverage, and risk coverage- before deciding that one should have a high level of confidence in a system. For mission-critical and safety-critical systems, using multifactor coverage measures is a smart approach.
Use of metrics:
We can use all of these metrics in a variety of forms; we can use a verbal narrative, detailed or summary numerical table, visual graphics. To determine the proper form to use in part by the purpose of the metric and the purpose of its use. As a test manager, it is important to remember that, if the recipients of these metrics are anyone other than self, the proper gathering, analysis, and reporting will depend on the audience, One should consider their goals, needs, and abilities. If we think that the metrics from the testing indicate some kind of problem, we have various options to resolve the problem and bring back the project under control. We can revise test road maps like quality risk analysis documents, test plans, or test priorities. For example, if we find that the project deadlines have changed and will not allow completion of the tests, we can reduce test coverage back on our risk analysis and adjust the downward priority of the associated risk items.
We can add test resources or overall test effort. We can decide to delay the release date to allow for more testing. This is seldom a popular option and is often only chosen when no other option exists. A common decision, that can be seen taking on many projects, is to change the test exit criteria if the release date or the other project priorities are threatened. While a test manager should outline the project team’s option as part of reporting test results, and the test manager often should provide the advice on which option to select, the actual selection and implementation of one or more of these options are typically outside the test managers remit.
⇓ Subscribe Us ⇓
If you are not regular reader of this website then highly recommends you to Sign up for our free email newsletter!! Sign up just providing your email address below:
Happy Testing!!!