We need test cases with 100% test coverage in order to test any software application. This will ensure that all possible combination of input and output parameters are well tested. Now a quick question, is that possible to achieve 100% test coverage without writing verbose test cases? Well, it’s possible if we practice “Pairwise Testing” or “All-Pairs testing” test design. Pairwise Testing can be defined as a “black box testing” test design technique that ensures almost hundred percent test coverage without writing verbose test cases and their execution.
With this test design technique, the testing team can achieve maximum test coverage ensuring high defect yield ratio and there is no need to test the application behavior against all possible combinations of input and output parameters. Since a number of test cases are less, therefore, test cases execution requires very less time and covers almost all possible scenarios.
In the black box testing approach, we test the application behavior for input parameters versus output results obtained. Again, the application output depends on input parameters, static values, state transitions and variables, output values, etc. Other approaches like boundary value analysis, equivalence partitioning, etc. are beneficial but these techniques do not provide maximum test coverage. Therefore “Pairwise Testing” approach is practiced to achieve maximum test coverage which we are going to discuss below in this article with an example.
Advantages of Pairwise Testing:
- It reduces the number of test cases execution.
- It increases the test coverage up to 100 percentages.
- It increases the defect or bug yield ratio.
- It takes less time to complete the execution of the test suite.
- It reduces the overall testing budget for a project.
Disadvantages of Pairwise Testing:
- If values selected for the variables are inappropriate, it is no longer beneficial.
- The highly probable combination can be missed while selecting the test data. This may result in a reduction of defect yield ratio.
- If variables combinations as test data are not understood correctly, such testing has no use at all.
Example of Pairwise Testing:
Suppose we have to test an application known as “Phone Ordering System” which has following input parameters along with their set of permissible values.
Automated “Public Library System”:
- Toronto public library offers books for reading which could be reserved and taken to home whereas some books can never be reserved based on its barcode but available for reading only in-store.
- Toronto public library is only for residents of Toronto. Other cities residents can only read the books but cannot reserve a book and take home.
- Books can be reserved online through website (online) and pick up at the library or directly in-house at the library.
- Books are available in two languages i.e. French and English.
- Books are available for categories as encyclopedia, fiction, and academics.
Rule I: First of all, we need to identify the variables and its permissible set of values.
- Member Residence: If library member is resident of Toronto of other cities of Ontario such as Hamilton, Mississauga, and Brampton. Total 4 cities in scope.
- Book Bar Code: Total 80,000 books are present in the library with associated status as reserve and non-reserve.
- Book Language: Books are either available in French or English. Total 2 Languages.
- Book Reserve Method: As discussed above there are total 2 ways such as online or in-house.
- Book Categories: There are total 3 categories of books as discussed above. These 3 categories are encyclopedia, fiction, and academics.
In order to test all possible valid combinations, here is the mathematics.
=> 4 X 80,000 X 2 X 2 X 3
=> 3,840,000 (Valid tests)
Also, if consider the negative tests and invalid test combinations, the total test number may bump up to some indefinite value.
Rule II: Smart Representative Simplification
- Let’s reduce book barcode variable into reserve and non-reserve, i.e. total 2 values.
- Let’s Member residence can be taken as Toronto and non-Toronto for rest of the cities, i.e. total 2 values.
- Also, try to use groups and boundary conditions even though data may not be non-discrete.
New valid combinations will look like as below:
=> 2 X 2 X 2 X 2 X 3
=> 48 (Valid tests)
Rule III: Arrangement of variables and corresponding values as input test data.
After arranging variables and values, test data table looks like this.
Phone Ordering System Specifications:
- Phone ordering system permits buying and selling of smartphones.
- Such trading is supported in cities say Bangalore and Mumbai.
- The system should have contract numbers with status as active or expired. It should permit the trade on these smartphones: Apple, Samsung, and LG.
- Booking of such devices can be done only In-store.
- Such an order can be booked during working hours.
Rule I: Identify the input variables required for testing and permissible set of values.
- Purchase Type: Permissible values are Buy and Sell. Total number of values is 2.
- Booking Type: Permissible value is In-store. Total number of value is 1.
- Deal Time: Permissible values are ‘Working hours’ and ‘Non-Working hours’. Total number of values is 2.
- Location: Permissible values are Bangalore and Mumbai. Total number of values is 2.
- Phone Brand: Permissible values are Apple, Samsung, and LG. Total number of values is 3.
- Contract Number: Permissible values are Active (6000) and Inactive (2000). Total number of values is 8000.
In order to test all possible valid combinations, here is the mathematics.
=> 2 X 2 X 3 X (6000 + 2000) X 1 X 2
=> 192,000. (Valid tests)
Also, if consider the negative tests and invalid test combinations, the total test number may bump up to some indefinite value.
Rule II: Smart Representative Simplification
Let’s reduce contract number variable into Active and Inactive, i.e. total 2 value. Also, try to use groups and boundary conditions even though data may not be non-discrete.
New valid combinations will look like as below:
=> 2 X 2 X 3 X 2 X 1 X 2
=> 48 (Valid tests)
Rule III: Arrangement of variables and corresponding values as input test data.
After arranging variables and values, test data table looks like this.
Sorting the columns of the table in descending order, as given below.
Rule IV: Arrange the table to prepare the test suite data as guided below:
Choose the column with a maximum number of permissible values first and place the values of variables as given below. It is actually multiple of first two column numbers (here 3 X 2 = 6). It will look like this.
Based on the first two column arranged in descending order, prepare the table as given below. Such that each book categories should be made available to both Toronto and non-Toronto residents.
Let’s try to place the next column ‘Book reserve method, such that it provides maximum test coverage, after placing online and in-house values it will look like this.
After placing online and in-house values, we can observe the pairs (Toronto, online) and (non-Toronto, in-house). Using such data these two cities could never be tested for in-house and online book reserve method respectively. The problem can be solved when we swap the online and in-house vales as indicated in the above picture and it will look like as given below in the picture.
Let’s place the values for next variable ‘Book Bar Code’ as reserve or non-reserve. Again we can identify pairs as (Toronto, reserve) and (non-Toronto, non-reserve). In order to increase the test coverage, we can ‘Book Bar Code’ values for 5th and 6th rows as demonstrated below.
After swapping values for ‘Book Bar Code’, the table will look like as below.
In a similar way, I have included values for variable ‘Book Language’ as French and English such that each of the row pairs is unique and covering all possible scenarios.
Let’s include the last variable ‘Book Language’ value. It has two value, therefore, we can place French and English as shown in the picture below.
Thus, we have left with only 6 valid combinations to test for all valid possible scenarios which we calculated as 48 before. This is what made possible by “Pairwise Testing” technique.
Conclusion:
“Pairwise Testing” requires initial effort to understand and select the possible input data combination. When this is selected efficiently it may result in high defect yield ratio and may provide 100 percent test coverage. It is one of the best design which results in high test efficiency in considerably less effort and time. There are number tools available in the market that can be used to achieve “Pairwise Testing” technique as follows.
| S. No. | Pairwise Tools Name | Vendor |
| 1 | ACTS [formerly: FireEye] | NIST |
| 2 | AETG | Telecordia |
| 3 | AllPairs | Satisfice |
| 4 | AllPairs | [McDowell] |
| 5 | AllPairs | MetaCommunications |
| 6 | ATD | AtYourSide Consulting |
| 7 | Bender RBT Inc. | BenderRBT |
| 8 | CaseMaker | Díaz & Hilterscheid |
| 9 | CATS (Constrained Array Test System) *) | [Sherwood] Bell Labs. |
| 10 | Combo-Test | The Australian eHealth Research Centre |
| 11 | CombTestWeb | Universidad de Castilla-La Mancha |
| 12 | CTE-XL | Berner & Mattner |
| 13 | CTS (Combinatorial Test Services) | IBM |
| 14 | DDA *) | [Colburn/Cohen/Turban] |
| 15 | EXACT *) | [Yan/Zhang] |
| 16 | Hexawise | Hexawise |
| 17 | IBM Functional Coverage Unified Solution | IBM |
| 18 | Intelligent Test Case Handler (replaces CTS) | IBM |
| 19 | IPO (PairTest) *) | [Tai/Lei] |
| 20 | IPO-s *) | [Calvagna/Gargantini] |
| 21 | Jenny | [Jenkins] |
| 22 | NTestCaseBuilder | [Murphy] |
| 23 | OA1 | k sharp technology |
| 24 | OATS (Orthogonal Array Test System) *) | [Phadke] ATT |
| 25 | OATSGen *) | Motorola |
| 26 | Pairwise Test Case Generator | TestersDesk |
| 27 | Pairwiser | Inductive AS |
| 28 | PICT | Microsoft Corp. |
| 29 | PictMaster | IWATSU System & Software |
| 30 | Pro-Test | SigmaZone |
| 31 | rdExpert | Phadke Associates, Inc. |
| 32 | ReduceArray2 | STSC, U.S. Air Force |
| 33 | SmartTest | Smartware Technologies Inc. |
| 34 | SpecExplorer | Microsoft Corp. |
| 35 | tcases | [Kimbrough] |
| 36 | TCG (Test Case Generator) *) | NASA |
| 37 | TConfig | [Williams] |
| 38 | Test Vector Generator | |
| 39 | TestCover | Testcover.com |
| 40 | VPTAG | [Robert Vanderwall] |
⇓ Subscribe Us ⇓
If you are not regular reader of this website then highly recommends you to Sign up for our free email newsletter!! Sign up just providing your email address below:
Happy Testing!!!