A Combinatorial Approach to Software Testing

The National Institute of Standards and Technology estimates that $60 billion dollars are spent each year due to inadequate software testing, despite the fact that 50-80% of development budgets go to testing. Testing every possible configuration is unrealistic for all but the smallest system, and so carefully designing a test strategy is essential for identifying and fixing errors. As systems become more complex and interconnected, errors become more difficult to detect.

A system which has thirty parameters has over a billion different configurations which must be tested to be 100% confident that the system is error-free. To reach 90% confidence, however, we need as few as 10 tests (and a few assumptions). Combinatorial testing applies the theory of combinatorial designs to produce series of tests which elicit errors (when they exist) efficiently and effectively. This area combines classical problems in pure math — including a centuries old false conjecture by Euler — with real-world applications.