The two major issues concerning covering arrays are explicit construction of a covering array, and exact or approximate determination of the covering array numberthe minimum size of a covering array. An efficient strategy for covering array construction with fuzzy logicbased adaptive swarm optimization for software testing use article in expert systems with applications. Covering arrays cas are mathematical objects used as platform or structure to represent the interactions of factors for a given system. An important component of this project is the development of covering arrays for use in the design of small but effective test sets for combinatorial testing. Covering array sampling of input event sequences for automated gui testing xun yuan dept. A tway covering array includes all tway combinations of variable values, up to a specified level of t usually 26 for software testing. Equivalence class verification and oraclefree testing.
An efficient strategy for covering array construction with fuzzy logicbased adaptive swarm optimization for software testing use article in expert systems with applications july 2015 with 277. Random testing, in another hand, doesnt guarantee the coverage of all factors interaction. Pdf a hybrid artificial bee colony and harmony search. Abstractcovering arraysca are combinatorial objects that have been used succesfully to automate the generation of test cases for software testing. Combinatorial software testing covering arrays software and network testing we want to test a system. This test verifies the subset of the total number of truth assignments to the expressions.
Covering arrays have proven to be highly effective in detecting software bugs in what is known as combinatorial testing. Covering arrays have been studied for their applications to drug screening and software testing. Recent research activities have demonstrated the effective application of combinatorial optimization in different areas, especially in software testing. Combinatorial testing ct can detect failures triggered by interactions of parameters in the software under test sut with a covering array test suite generated by some sampling mechanisms. The tables of covering arrays that are accessible from this website are maintained as part of the automated combinatorial testing for software acts project at nist. Adaptive algorithms for locating faulty interactions. Covering arrays can be used for software interaction testing to detect faults. An important application of covering arrays is being used for both software and hardware testing. The key ingredient for this form of testing is known as a covering array, a mathematical object in which all tway combinations of parameter values are covered at least once 6, 7, 8. Logic coverage comes from any of the below mentioned sources. A covering array can be used to su ciently encompass the main system con gurations. Cas to software testing and discuss the future possible directions of the research. It appears recently that the interaction among the system factors represents a common source of faults. The comprehensive tool for software updates and installations streamtask is a browserbased, platformindependent application that supports software upgrade planning and execution.
Constructions, applications and generalizations carleton university, may 1416, 2006, ottawa, ontario, canada. Usage for example, command program 4x2 3x3 2x1 outputs a pairwise testing suite for the case of 2 4valued, 3 3valued, and 1 2valued parameters. Nist planning report 023, the economic impacts of inadequate infrastructure for software testing, 2002. Covering array library automated combinatorial testing. There are two approaches for combinatorial testing. The major contribution of algorithm ptsgga is that it uses a probabilistic approach to generate initial population of casmcas to improve the performance of ga. Covering array ca has been introduced as a representation of the combinations in one complete set. Apr 17, 2015 a second layer covering array of the equivalence class values is computed, and its values substituted for the equivalence class names in the primary array. Software to produce a covering array of given size on demand, called fireeye, is under production by the acts team. Covering arrays have been studied for their applications on software testing, hardware testing, drug screening, and in areas where. Covering array sampling of input event sequences for. Covering arrays are a useful mathematical tool for testing all possible tway interactions among the components of a system. Covering arrays help with this by optimally sampling from the input space, thus maximizing your chances of finding defects while minimizing timeresourcescost. Citeseerx experimental designs in software engineering.
In combinatorial testing, t is the strength of the coverage of interactions, k is the number of components degree, and v is the number of symbols for each component. Thus, covering arrays are used as an efficient way for picking tests for software 5,7,9. It is shown that this method can also detect certain classes of errors without a conventional test oracle, and an illustrative example is given. When testing a software component with k parameters, each of which must be tested with g values, the total number of possible test vectors is gk. The research in this area seems to be an active research direction for the coming years. A survey of combinatorial testing acm computing surveys. Any software package will have two natural parameters, the number of inputs and the number of values each input can take. Apr 16, 2020 orthogonal array testing technique is a statistical approach for testing pairwise interactions. Sequence covering array library many testing problems involve sequences of operations. May 24, 2016 although most combinatorial testing problems have varying numbers of values per variable, in some cases all variables have the same number of values and a precomputed array can be found. The algorithm constructs test suites for pairwise testing, that is, covering arrays of strength 2, based on an algebraic approach. Key method here, we present a novel strategy abccag artificial bee colonycovering array generator based on the artificial bee colony abc algorithm to generate covering an array and a mixed covering array for pairwise testing.
Combinatorial testing ct based on combinatorial design has been researched. A second layer covering array of the equivalence class values is computed, and its values substituted for the equivalence class names in the primary array. Covering arrays have been studied for their applications on software testing, hardware testing, drug screening, and. Expanding upon the concept of orthogonal arrays are covering arrays. It has been an active field of research in the last twenty years.
The proposed abccag strategy is implemented in a tool and experiments. This tutorial will give a quick introduction to what covering arrays are. N array with entries from a set of v symbols such that in any t. Software performance testing using covering arrays. A column expansion construction for optimal and nearoptimal mixed covering arrays.
Background material for the service including references and links for software risks, pairwise testing, pairwise test case generators, covering arrays and orthogonal arrays. For a survey of covering arrays in the binary v 2 case, see. It is particularly effective in finding errors associated with faulty. An efficient strategy for covering array construction with. Exhaustive testing is not possible in most systems. Covering arrays of higher strength from permutation. Apr 25, 2015 covering arrays are combinatorial objects that have several practical applications, specially in the design of experiments for software and hardware testing.
An efficient strategy for covering array construction with fuzzy logic. Event sequence testing automated combinatorial testing for. An efficient strategy for covering array construction. Logic corresponds to the internal structure of the code and this testing is adopted for safetycritical applications such as softwares used in aviation industry. Citeseerx document details isaac councill, lee giles, pradeep teregowda. Combinatorial software testing covering arrays pairwise testing covering array. A covering array \t\text ca n,k,g\, of size n, strength t, degree k, and order g, is a \k\times n\ array on g symbols such that every \t\times n\ subarray contains every \t\times 1\ column on g symbols at least once. Covering array ca and mixed covering array mca are often used to represent test sets. Software function properly, all input factors and their interactions of the software need to be tested i. Covering array, mixed covering array, interaction testing, testing processes, software testing, software system, parameters values, construction. It is important to test combinations of connected components, but also to test the order in which they could be.
Most of the defects which i have observed are caused due to interaction and integration. Although most combinatorial testing problems have varying numbers of values per variable, in some cases all variables have the same number of values and a precomputed array can be found. Orthogonal arrays can be applied in user interface testing, system testing, regression testing, configuration testi. This interaction or integration can be within different objects, elements, options in a screen of the application, or configuration setting in a file. A covering array is a mathematical object that can be used for software testing purposes see the hartman paper for another introduction. The key ingredient for this form of testing is known as a covering array, a mathematical object in which all tway combinations of parameter values are covered at. An efficient strategy for covering array construction with fuzzy logicbased adaptive swarm optimization for software testing use 1.
Covering array definition b a covering array is a tool you can use to optimize your testing to get enough information without using too many resources. Each test contains a set of values that represent a speci c setting for each. Ab a covering array of size n, degree k, order v and strength t is a k. Problems and algorithms for covering arrays sciencedirect. Covering arrays have been studied for their applications on software testing, hardware testing, drug. Covering array represents the test case selected under pairwise testing 5. Key method here, we present a novel strategy abccag artificial bee colony covering array generator based on the artificial bee colony abc algorithm to generate covering an array and a mixed covering array for pairwise testing.
Use of orthogonal arrays and design of experiments via. Several algorithms for constructing covering arrays suitable for software testing have been developed. Blackbox testing mobile applications using sequence covering. Testing is an indispensable part of the software development life cycle. Refining the inparameterorder strategy for constructing. Orthogonal array testing technique is a statistical approach for testing pairwise interactions. Zamli software engineering group, school of electrical and electronic engineering. Pdf covering arrays of strength four and software testing. It is particularly effective in finding errors associated with faulty logic within computer software systems. Generating mixedlevel covering arrays with 2 and test. Covering array generation, a discrete optimization problem, is the most popular research area in the field of ct.
Construction of variable strength covering array for. Orthogonal array testing is a black box testing technique that is a systematic, statistical way of software testing. The classical covering array problem is to minimize the number of columns b for given fixed values of t,k, and g 1,g. Jun 26, 2015 a covering array \t\text ca n,k,g\, of size n, strength t, degree k, and order g, is a \k\times n\ array on g symbols such that every \t\times n\ subarray contains every \t\times 1\ column on g symbols at least once. A major application of covering arrays is the task of software testing see the hartman paper. Covering array tutorial computer science and engineering.
Software often has many possible configurations, each of which could be faulty. It is important to test combinations of connected components, but also to test the order in which they could be connected. Combinatorial testing ct aims to generate small sets of test cases to uncover defects caused by those factors and their interactions. Covering arrays are used in software testing for accurate coverage of interactions, while doptimal and factorial designs measure the amount of interaction. Covering arrays of strength three arizona state university. Covering arrays have been studied for their applications on software testing, hardware testing, drug screening, and in areas where interactions of multiple parameters are to be tested. Equivalence class verification and oraclefree testing using. Covering arrays of strength four and software testing. For example, an embedded system may accept multiple sensor inputs and generate output to several communication links and effectors such as machine controls. A testing array is said to be a covering array cat,k,g 1,g 2,g k if for any tdistinct rows, the submatrix induced by the selected rows contains every member of the cartesian product of the t domain sets as a column vector. Blackbox testing mobile applications using sequence.
Adaptive algorithms for locating faulty interactions covering. Multiple studies have shown fault detection equal to exhaustive testing with a 20x to 700x reduction in test set size. This finding, referred to as the interaction rule, has important implications for software testing because it means that testing parameter combinations can provide more efficient fault detection than conventional methods. Siam journal on discrete mathematics siam society for. This new organizational channel allows you to communicate, test, monitor, track and document upgrades with accuracy and confidence. Nist covering array tables what is a covering array. It is used when the number of inputs to the system is relatively small, but too large to allow for exhaustive testing of every possible input to the systems. Array software is located in agawam massachusetts and serves hospitals and health care systems throughout the world. In this paper, combinatorial objects, namely covering array ca and mixed covering arrays mca, are used to represent test set. Initial work involved exhaustive generation of designs in order to compare covering arrays and doptimal designs in approximating full factorial designs. In every n x t subarray, each ttuple occurs at least.
Some use the inparameterorder, or ipo, strategy 7,9. Combinatorial testing ct aims to generate small sets of test cases to uncover defects caused by. The idea is that the covering array acts as a test suite containing a series of tests. Event sequence testing automated combinatorial testing. Combinatorial testing is a vital approach to detect interaction errors occurs because of interaction of several parameters. Locating errors using elas, covering arrays, and adaptive. As a complex logic system, software may suffer from different source of faults. Nist library of precomputed covering arrays arrays are available for t2 to t5, with 2 to 6 values per variable, and for t6 with 2 to 5 values per variable. Jul 06, 2004 a testing array is said to be a covering array cat,k,g 1,g 2,g k if for any tdistinct rows, the submatrix induced by the selected rows contains every member of the cartesian product of the t domain sets as a column vector.
A discrete particle swarm optimization for covering array. Construction of mixed covering arrays for pairwise testing. Construction of mixed covering arrays for pairwise. Combinatorial testing and covering arrays school of electrical. Those faults can be avoided by applying different testing processes. Covering array, mixed covering array, interaction testing, testing processes, software. This paper presents a hybrid algorithm that integrates artificial bee colony algorithm abc and harmony search algorithm hs to construct cas for testing all 2way interactions pairwise testing in software systems.
701 1031 1230 1293 1209 136 1007 31 669 191 1239 273 1087 270 12 1547 1308 946 897 101 786 1276 302 220 433 1168 496 774 1337 838 1544 1074 989 128 1393 24 1089 1214 1126 988 1127 1165 356 82