Software testing is an important approach to increase code quality, and thus part of software engineering education. Mutation testing is an effective testing technique with increasing adoption in industry, measuring and improving how well a test suite can detect small changes to a program under test. This paper aims to study the use of mutation testing tools in an educational setting and understand students’ conceptual and technical challenges in applying mutation testing. It reports on two case studies of incorporating mutation testing into software engineering curricula.

The Scaffolding Study explores the impact of using different mutation testing tools directly or indirectly via a uniform interface provided by an educational infrastructure. We observe that scaffolding (indirect tool use) reduced technical difficulties and improved the consistency and quality of student submissions.

The Visualization Study explores the impact of different forms of output of a mutation testing tool. Specifically, it assesses to what extent visualizations support students in the tasks of reasoning about mutants and writing detecting tests for them. We observe that visualizations improved the average mutation scores, with lower-performing students seeing a boost in particular.

We further explore challenges around automatic assessment of mutation-testing assignments. For example, we observe that even with assignment scaffolding, between 18% and 21% of student submissions required manual modifications to successfully execute.