Killing Mutants with MuClipse


Ben Smith and Laurie Williams [Contact Authors]
CSC 712 - Software Engineering
Department of Computer Science
North Carolina State University

Back to Software Engineering Tutorials


0.0 Contents
1.0 What are Mutants?
2.0 Example
3.0 What is MuClipse?
4.0 Installing MuClipse
5.0 MuClipse with CoffeeMaker
6.0 Exercise
7.0 Resources

1.0 What Are Mutants?

With unit testing, code is divided into pieces and tests are performed on only each piece of functionality to make sure that it gives a predefined desired result. Using tools, like jUnit (see tutorial), the process of testing a given method or class to see that it always gives this result is automated. The result could be anything from the return type of the method to a line of code in the method failing.

Mutants are modifications of source code that have been modified by mutation operators. The primary concept behind mutation testing is that modifying the source code of a given unit (or mutating it) should cause the test that operates on that unit to fail.

Top | Contents

2.0 Example

Example 2.1: Method and Corresponding Unit Test


//original code under test
public class Example
{
	public boolean isLessThanThree(int number)
	{
		return (number < 3);
	}
}


//unit test
public class ExampleTest extends junit.framework.TestCase
{

	public void testLessThanThree()
	{
		assertTrue(isLessThanThree(2));
	}

	public void testLessThanThreeFail()
	{
		assertFalse(isLessThanThree(3));
	}

}

Example 2.2: Mutated Method

public class Example
{
	public boolean isLessThanThree(int number)
	{
		return (number > 3);
	}
}

Notice that with this mutation, the method testLessThanThree would fail because the mutated method now returns the boolean number > 3, which in this case would be false and the assert statement is checking to see that the desired result is true. The method testLessThanThreeFail would not fail, however, because 3 is not greater than 3 and the returned boolean would be false, which is what the assert statement is checking for.

Because testLessThanThree failed, this mutant is said to be killed; the mutation caused the test to fail, which means the test is adequately catching this kind of mistake.

Top | Contents

3.0 What is MuClipse?

MuJava is an Eclipse plugin which contains two runtime configurations. The first uses a given set of mutation operators on the given source to create mutants, which it stores in a folder within the same project. The second program runs a given test or set of tests on the original code and the generated mutants and compares the results. If no difference was detected, the mutant lives. If either the returned result is different, an Exception is thrown, or the mutant causes the JVM to go into an infinite loop, then the mutant is said to be killed.

After all the results have been tabulated, MuClipse returns a mutation score which is a percentage of all the generated mutants that have been killed. Mutation operators can be written and added as a developer thinks of them, and typically contain a Reader and a Writer. There are two types of mutation operators:

  • Traditional: Operate only at the method level, changing lines of code that fit a certain pattern (ie, switching operands, replacing + with -, etc).
  • Class-Level: Operate on the class level: changing keywords that dictate the type of class or the methods involved (ie, overloading a given method, changing a class to static, etc).

MuJava has to be loaded into your workspace and must be pointed to the project (or projects) on which it will perform mutation testing. The following instructions contain download links and a short example for setup and use of the MuJava (JMutation System) tool.

Top | Contents

4.0 Installing MuClipse

  1. From an open Eclipse workbench, go to Help -> Software Updates -> Find and Install...
  2. Select the bullet "Search for more updates to install". Click Next.
  3. On the dialog box that appears next, click "New Remote Site..." and enter the data into the fields as you see below.


    Figure: Configuring the Update Site

  4. After clicking OK, be sure that none of the other sites are checked and that "MuClipse" is checked. Click Finish.
  5. A new dialog box will appear with MuClipse as an option to update. Make sure that it is checked and click Next.
  6. On the next frame, select "I accept the terms of the license agreement" and click Next.
  7. A new frame will appear which has the MuClipse feature and the install location. NOTE: Be sure the install location is the default Eclipse directory! Click Finish.
  8. The following dialog box will appear:


    Figure: Verifying the Update Site

    Click Install.
  9. After the installation process is complete, Eclipse will ask you if you would like to restart the workbench. Click Yes.
Top | Contents

5.0 MuClipse with CoffeeMaker

For the exercise, you will run mutation testing on the CoffeeMaker project. Download and unzip this project into your workspace or open a workspace which already has it loaded.

extendedOJ

Neither the process of Generating Mutants nor the process of Running Test Cases will function correctly if extendedOJ is not on your buildpath. To put it on your buildpath, follow these steps:
  1. Download extendedOJ from the link above and save it into a directory within your project's root (we recommend /lib).
  2. Open Eclipse and open the workspace with your project in it. Right click on the extendedOJ file within Eclipse and go to Build Path -> Add to Build Path. As shown below:

  3. Figure 1: Adding extendedOJ to the buildpath

  4. Your project should rebuild.

Folder Structure

The following requirements for the directory of your structure must also be implemented to use MuJava.
  • You will need a folder within your project's root directory called result. No other name will work with this version of MuClipse.
  • Your unit tests must compile to their source folder. To set this up, right click unittests, go to Build Path -> Configure output folder, select the radio button "Specific output folder" and type in the same path as the source folder.

jUnit

IMPORTANT: Since you are going to be using jUnit test cases to perform your mutation testing, their setUp() method needs to have the declaration public (it is currently protected).

Generating Mutants

To generate mutants you will use the "MuClipse: Mutants" runtime configuration. Follow these steps to open and prepare your configuration:
  • IMPORTANT: Click on the project you are working with.
  • Go to the arrow to the left of run and select "Run ..." as shown below.


    Figure 2: Opening the runtime configuration

  • The Eclipse runtime configuration dialog box will appear. Select "MuClipse: Mutants" and click "New".

The Directories Tab

The following is an explanation of the fields on the "Directories" tab as shown below.


Figure 3: Directories

  • Project: The selected project (CoffeeMaker).
  • Mutants Output: Where the mutation results will go (result)
  • Classes Folder: The root of where your compiled Java binaries are (bin)
  • Testset Folder: The root of where your test case source files are (unittests)
  • Source Folder: The root of where your source code files are (src)
  • After you have correctly pointed all the folder fields to the correct locations, select the target class Recipe.java.
  • You are not done: see the mutants tab for more information.
  • The Mutants Tab


    Figure 4: Mutants

    Select the mutation operators COI and JTI. COI stands for "Conditional Operator Insertion" which, as its name suggests, places unary conditional operators (such as ! or > ) within boolean expressions. JTI stands for "Java this Insertion" which inserts the Java this keyword before references to variables when they are used within a method. These operators will produce some interesting results!

    Execution

    MuClipse will now create the mutants you have specified. Some operators will not produce any mutants. If they do not, try different operators. The console will show "Mutants have been created!" when the process is complete. If you would like to see your mutants without any test case information, proceed to the section "Viewing Results" below.

    Running Test Cases

    To generate mutants you will use the "MuClipse: Tests" runtime configuration. Follow these steps to open and prepare your configuration:
    • IMPORTANT: Click on the project you are working with.
    • Go to the arrow to the left of run and select "Run ..." as shown below.


      Figure 5: Opening the runtime configuration

    • The Eclipse runtime configuration dialog box will appear. Select "MuClipse: Tests" and click "New".

    The Directories Tab

    The following is an explanation (and values to set) for the fields on the "Directories" tab as shown below.


    Figure 6: Directories

    • Project: The selected project (CoffeeMaker)
    • Mutants Output: Where the mutation results will go (result)
    • Classes Folder: The root of where your compiled Java binaries are (bin)
    • Testset Folder: The root of where your test case source files are (unittests)
    • Source Folder: The root of where your source code files are (src)
  • After you have correctly pointed all the folder fields to the correct locations, select the RecipeTest unit test (not Inventory).
  • Since this class targets Recipe and we have mutated Recipe we will select that as our target class.
  • You are not done: see the test options tab for more information.
  • The Testing Options Tab


    Figure 7: Testing Options

    The testing options tab allows you to specify the following options:
    • The Timeout Period: This is the amount of time each testcase is given to run. If a testcase causes an infinite loop, MuClipse kills that thread and counts that as a failure. If a lot of testcases are failing, increase this value. If execution is taking too long, decrease this value. In this case, 4500ms should be fine.
    • Test Traditional/Class-Level Mutants: These are two different types of operators. Do not test an operator type which you did not create mutants for or which did not generate any mutants. We generated both class- and method-level mutants, so leave these boxes checked.

    Execution

    When you have finished specifying your options, click "Run." The console window will display output indicating the test results. The last thing in the output window should be your mutation score and killed/live numbers. When execution is complete, the console will display these statistics and the string "Writing to file..."

    Viewing Results

    • To view your mutants, right click on the result folder and go to MuClipse -> View Mutants and Results.
    • Each root node in the view of the tree that will appear is a class, underneath which are the Class-Level and Method-Level mutant nodes. The Method-Level mutants are seperated by method name.
    • The node names correspond to the name of the mutation operator and this instance of its mutation. If you have run test results, the name will show "killed" or "alive" depending on its status.
    • To see how a particular mutant compares to its original, double click one of the nodes as shown below.


    Figure 7: Viewing Results

    Top | Contents

    6.0 Exercise
    Your exercise is to kill all of the living mutants (there should be 10) remaining by running the JTI and COI operators on the Recipe class as described above.

    NOTE: There is a bug in the mutants and results view--it will show one living mutant even after you have killed them all. However, it shows that your mutants are killed if you look through the various operators and classes. Your console output after running test cases will show the true mutation results. Also, ignore the error message "No mutant is exist for Recipe"

    Some helpful shortcuts:

    • You should not have to modify your source code.
    • You do not have to regenerate mutants every time you make a change to your unit tests, just rerun your test cases. Just click the recently used runtime configuration to do this.
    • You can press the yellow arrow at the top of the "Mutants and Results" view to refresh your mutation results when you have executed your test cases.
    • With each change to your unit test, run it with a jUnit runtime configuration to confirm that it is working and that all of its testcases pass. If you do not check this, you are likely to produce even more living mutants due to a method that fails on the original code.
    Top | Contents

    7.0 Resources
    Top | Contents

    Back to Software Engineering Tutorials
    ©2006 North Carolina State University, Laurie Williams and Ben Smith
    Email the authors with any questions or comments about this tutorial.
    Last Updated: Monday, March 30, 2009 3:03 PM