Mutation Testing with Jumble

Ben Smith, Andy Meneely and Laurie Williams [Contact Authors]
CSC 712 - Software Engineering
Department of Computer Science
North Carolina State University

Back to Software Engineering Tutorials
0.0 Contents
1.0 What are Mutants?
2.0 Example
3.0 What is Jumble?
4.0 Installing Jumble
5.0 Jumble with iTrust
6.0 Exercise: Interpreting Results
7.0 Exercise: Running Jumble with a JUnit suite
8.0 Resources
1.0 What Are Mutants?

The goal of mutation testing is to evaluate the coverage of your tests. The general idea is to automatically inject faults into the system, re-run the tests, and expect the tests to fail. A version of fault-injected code is called a mutant. A mutant is considered to be live if the tests still pass (usually implying that something was not fully tested). Examining live mutants can shed light on what kinds of test cases your suite is missing.

Mutants are modifications of source code that have been modified by mutation operators. These mutation operators are usually simple: for example, changing a "<" to ">".

Top | Contents
2.0 Example

Example 2.1: Method and Corresponding Unit Test 


//original code under test
public class Example
{
	public boolean isLessThanThree(int number)
	{
		return (number < 3);
	}
}


//unit test
public class ExampleTest extends junit.framework.TestCase
{

	public void testLessThanThree()
	{
		assertTrue(isLessThanThree(2));
	}

	public void testLessThanThreeFail()
	{
		assertFalse(isLessThanThree(3));
	}

}

Example 2.2: Mutated Method 

public class Example
{
	public boolean isLessThanThree(int number)
	{
		return (number > 3); //the "<" operator mutated
	}
}

Notice that with this mutation, the method testLessThanThree would fail because the mutated method now returns the boolean number > 3, which in this case would be false and the assert statement is checking to see that the desired result is true. The method testLessThanThreeFail would not fail, however, because 3 is not greater than 3 and the returned boolean would be false, which is what the assert statement is checking for.

Because testLessThanThree failed, this mutant is said to be killed; the mutation caused the test to fail, which means the test is adequately catching this kind of mistake.
Top | Contents
3.0 What is Jumble?

Jumble is a Java application that mutates a Java class at the bytecode level and runs its corresponding unit test to determine the number of killed mutants. If no tests failed, the mutant lives.

After all the results have been tabulated, Jumble returns a mutation score which is a percentage of all the generated mutants that have been killed. Additionally, your console output will tell you the details regarding each live mutant and this tutorial will tell you how to interpret those results.

Jumble will need to be loaded into your copy of Eclipse and must be pointed to the project (or projects) on which it will perform mutation testing. The following instructions contain download links and a short example for setup and use of the Jumble tool.

Top | Contents
4.0 Installing Jumble

  1. From an open Eclipse workbench, go to Help -> Software Updates -> Find and Install...
  2. Select the bullet "Search for more updates to install". Click Next.
  3. On the dialog box that appears next, click "New Remote Site..." and enter the data into the fields as you see below.
    4. 	Name: Jumble
    
	URL: http://agile.csc.ncsu.edu/SEMaterials/tutorials/jumble/site
  4. After clicking OK, be sure that "Jumble" is checked. Click Next.
  5. On the next frame, select "I accept the terms of the license agreement" and click Next.
  6. A new frame will appear which has the Jumble feature and the install location.

    7. NOTE: Be sure the install location is the default Eclipse directory!
  7. Click Finish.
  8. The following dialog box will appear as in Figure 4.1:


    Figure 4.1: Verifying the Update Site

    Click Install.
  9. After the installation process is complete, Eclipse will ask you if you would like to restart the workbench. Click Yes.
Top | Contents
5.0 Jumble with iTrust

For the exercise, you will run mutation testing on the iTrust source code. If you do not have it, you must download it and import it into your workspace before you continue. We will be mutating EditHealthHistoryAction. Perform the following steps:

  1. Open or expand iTrust -> src -> edu.ncsu.csc.itrust -> action.
  2. Right click on EditHealthHistoryAction and choose Jumble -> Jumble Class as shown in Figure 5.3.


    3. Figure 5.3: Running Jumble

NOTE:: Jumble does not preconfigure your project choice. You will probably get a stacktrace error on first execution. To fix this, execute the following steps:

  1. Click on the arrow to the right of the run menu and select "Open Run Dialog..." as shown in Figure 5.1.


    2. Figure 5.1: Opening the Run Dialog

  2. In the dialog that appears, find and click the "Run Jumble" runtime configuration.
  3. In the "Project" field, type iTrust (or the name of your iTrust project) as shown in Figure 5.2.


    4. Figure 5.2: Setting the Project

  4. Click Apply and then click Close.
Now that you have selected iTrust as your project under test, your initial test results will be gathered, and then Jumble will proceed to execute the corresponding test case on the resultant mutants.

NOTE:: Unless otherwise specified, if the name of the class is ThisIsTheClassName its corresponding test is ThisIsTheClassNameTest. Sometimes, however, we will have multiple unit tests for a single class, so we will need to define a suite of test cases. Instructions for using a test suite are in section 7.0

You can now interpret your Jumble results

Top | Contents
6.0 Exercise: Interpreting Results
You should now have Jumble output that looks like the following (not including your test case data generation): 
Mutating edu.ncsu.csc.itrust.action.EditHealthHistoryAction
Tests: edu.ncsu.csc.itrust.action.EditHealthHistoryActionTest
Mutation points = 13, unit test time limit 31.42s
..M FAIL: edu.ncsu.csc.itrust.action.EditHealthHistoryAction:29: changed return value (areturn)
.M FAIL: edu.ncsu.csc.itrust.action.EditHealthHistoryAction:40: changed return value (areturn)
M FAIL: edu.ncsu.csc.itrust.action.EditHealthHistoryAction:53: 100 (d) -> 101 (e)
.......
Score: 76%
Here is a list of items and what they mean:
  • Score - is a percentage of mutants killed out of the total mutants. It is used to assess the validity of our test suite.
  • Mutation Points - is a number corresponding to the number of locations in the code where a mutation occured.
  • [periods] - periods indicate that a mutant was created and died.
  • M FAIL: - indicates a mutation that your unit tests did not detect. The information following the : tells you what was changed and on what line of Java code the change occured.

    • Sometimes the output provided does not give us enough information about what the mutation actually was. This information can be found Jumble website.

    The first mutation indicates that the code (found in EditHealthHistoryAction):

    	public String getPatientName() throws DBException, iTrustException{
		return authDAO.getUserName(pid);
	}
    ... was changed to .... 
    	public String getPatientName() throws DBException, iTrustException{
		return null;
	}
    ..and that no unit test in EditHealthHistoryActionTest detected this change. For this example, we will provide you the unit test case for detecting this mutation: 
    	public void testPatientNameNull() throws Exception {
		assertNotNull(action.getPatientName());
	}
    Paste this code into EditHealthHistoryTest, save it and run jumble on EditHealthHistory again.

    NOTE:: You may have to enter the project name again. Your Jumble output should be: 

    Mutating edu.ncsu.csc.itrust.action.EditHealthHistoryAction
Tests: edu.ncsu.csc.itrust.action.EditHealthHistoryActionTest
Mutation points = 13, unit test time limit 45.23s
....M FAIL: edu.ncsu.csc.itrust.action.EditHealthHistoryAction:40: changed return value (areturn)
M FAIL: edu.ncsu.csc.itrust.action.EditHealthHistoryAction:53: 100 (d) -> 101 (e)
.......
Score: 84%

    For the rest of this exercise, you must:

    1. Kill the second living mutant (Hint: Use EvilTestConnectionDriver)
    2. Interpret the third mutant and decide whether you should fix the code or just kill the mutant
Top | Contents
7.0 Exercise: Running Jumble with a suite of tests
The previous exercise showed how to run mutation tests using a single unit test in the same package as the source code. While this is the case for many of iTrust's classes, it is not the case for all of them (DAO's, example, are one of these exceptions). Here, we will run Jumble on AccessDAO.java with a suite of tests.

The Jumble Eclipse plugin doesn't support this feature however, it does create a runtime configuration we can modify. First, try to run Jumble on edu.ncsu.csc.itrust.dao.mysql.AccessDAO.java. You should get a message in the console like this:

Mutating edu.ncsu.csc.itrust.dao.mysql.AccessDAO
Tests: edu.ncsu.csc.itrust.dao.mysql.AccessDAOTest
Score: 0% (NO TEST CLASS)
Mutation points = 41

This error occurred because edu.ncsu.csc.itrust.dao.mysql.AccessDAOTest doesn't exist. We need to create a suite of tests that test AccessDAO.java and point Jumble to it.

To create a test suite, first create a new source folder called "mutation" in your iTrust folder. Right-click on iTrust in the package explorer and select "New > Source Folder..."


Figure 7.1: Creating a new source folder



Now create an appropriate package in the source folder. Right-click on the "mutations" source folder and select "New > Package". Call it "edu.ncsu.csc.itrust.dao.access".

Now navigate to the edu.ncsu.csc.itrust.dao.access package in the "unittests" folder. Right click on the "access" package, and go to "New > Other...". Select New JUnit Test Suite.


Figure 7.2: Creating a new JUnit test suite



Hit Next. You should see the following screen. Change the name of the suite from AllTests to AccessDAOSuite. Hit Finish


Figure 7.3: Naming your new JUnit test suite



The new AccessDAOSuite.java has been placed in the access package. Drag it to your new package in the "mutations" folder.


Figure 7.4: Moving your new JUnit test suite



Lastly, we need to make our test suite an actual TestCase. In your new suite, add an extends TestCase to the class declaration.

Now we need to point Jumble to our new test suite. Go to Run > Open Run Dialog.... Find Run Jumble(1), and rename it to Run Jumble on AccessDAO

Still in the Run dialog, switch over to the Arguments tab. Under Program Arguments, scroll all the way to the bottom. Add the following argument to Jumble: edu.ncsu.csc.itrust.dao.access.AccessDAOSuite. Your arugments should look something like this:


Figure 7.5: Jumble runtime configuration for a JUnit suite



Hit Run. Among many other things, your console should include the following output: 
Mutating edu.ncsu.csc.itrust.dao.mysql.AccessDAO
Tests: edu.ncsu.csc.itrust.dao.access.AccessDAOSuite
Exercise: One of the live mutants from this run is the following, 
M FAIL: edu.ncsu.csc.itrust.dao.mysql.AccessDAO:111: 20 -> 21
What does this live mutant mean? Think of two ways you can get rid of this mutant.
Top | Contents
8.0 Resources
Top | Contents
Back to Software Engineering Tutorials
©2006 North Carolina State University, Andy Meneely, Laurie Williams and Ben Smith
Email the authors with any questions or comments about this tutorial.
Last Updated: Tuesday, October 17, 2006 8:45 PM