Andy Meneely, Ben Smith, and Laurie Williams
CSC 326 - Software Engineering
Department of Computer Science
North Carolina State University

Back to Software Engineering Tutorials

JTest is a comprehensive automated testing tool that integrates static analysis, metrics, code coverage, unit test generation, and coding policy enforcement into a single platform. The idea is to combine all of the latter technologies into a single tool so that they can all be run at once. JTest employs a large database of known vulnerabilities that can be found in Java code, working with a variety of technologies (JSP, JDBC, etc). JTest is a customized version of the Eclipse IDE, so most of the project management work is performed with Eclipse features.

In this tutorial, we will be employing three main features of JTest: static analysis, metrics, and unit test generation.

In this tutorial, we assume that you already have JTest 8.4 set up and have the proper licensing. If you are using this tutorial for a class (e.g. CSC 591-003 for NCSU Fall 2009) consult your lab instructions on how to access and start JTest.

Also, this tutorial assumes that your JTest has Subclipse installed. If you need to install Subclipse, use the Update Site found on their website and our tutorial on plugin installation.

For this tutorial, we will be using the intentionally-vulnerable web application WebGoat to use JTest's features. We will first be installing WebGoat's source code by checking out the project from the Subversion repository.

1. Start JTest. Go to the JTest perspective (Window > Open Perspective > Other... > JTest). Check that the workspace has a "Servers" folder with Apache Tomcat 5.5 configuration (this may be installed by default by your instructor). If it's not there, follow the instructions in the Troubleshooting section.

Figure 2.1: Checking to see if servers exist
2. Go to File > New > Other... > SVN > Checkout Projects from SVN. Click Next.
3. Select "Create new repository location". Click Next.
4. For URL, put the following URL
   http://webgoat.googlecode.com/svn/trunk/webgoat/main
   Click Next.
5. Select the project folder and click Finish.

Figure 2.2: Checking out the WebGoat source
6. In the course of checking out the project, you will be asked to create a new project, choose Web > Dynamic Web Project. Click Next.

Figure 2.3: Choosing a Dynamic Web Application
7. For the project name, put WebGoat. Also, make sure that Tomcat 5.5 is the Target Runtime. Click Finish.

Figure 2.4: Asserting that Tomcat is Target Runtime
8. As the project is checking out, you may be asked to approve license agreements, get a warning about replacing local resources, or about changing perspectives - these are normal. Checkout can take a few minutes.
9. Once the project has checked, go to the JTest perspective (Window > Open Perspective > Other... > JTest). In the WebGoat project on the left, right-click on the JavaSource folder and go to Build Path > Use as Source Folder

Figure 2.5: Setting JavaSource as an Eclipse source folder

The goal of security-focused static analysis is to find specific vulnerabilities in your code. Thus, let's start with a location in the code where we know a vulnerability exists and see if JTest finds it. Let's start with an SQL Injection vulnerability.

1. Still in the JTest perspective, open the file JavaSource > org.owasp.webgoat.lessons.SQLInjection.Login (called Login.java). Scroll down to the login method on line 118
   • Tip: line numbers are on the bottom of the screen
   • Another tip: click on the down arrow in the Package Explorer and go to Package Presentation > Hierarchical. See screenshot:
     
     Figure 3.1: Setting the package presentation as hierarchical
2. Looking at the method Login.login(...), locate the SQL injection vulnerability. For a moment, consider what characteristics of this code a program could automatically detect to find a possible SQL injection vulnerability.

Figure 3.2: Locating the SQL Injection Vulnerability
3. Now let's run static analysis on this Login.java file. Go to JTest > Test Configurations

Figure 3.3: JTest Test Configurations
4. Under Built-in expand Security. Right-click the CWE Top 25 Most Dangerous Programming Errors and select New Child.

Figure 3.4: Creating a new child
5. This creates a new "child" JTest configuration that we can customize. Since static analysis can take a long time, it's more practical to filter out alert types that we are not currently considering. In our new "User-Defined" configuration, go to the "Static" tab.

Figure 3.5: Customizing the CWE configuration
6. Uncheck the following static analysis warning types:
   • Hibernate. This application is not using Hibernate, so we don't need those warnings
   • Optimization. These specific warnings are not pertinent for this example
7. Press Apply and then Close - NOT Run Test!
8. Now make sure that the Login.java file is selected in the package explorer (if you have the project selected, the test will run over the whole project!). Go to JTest > Test Using... > User Defined > CWE Top 25 Most Dangerous Programming Errors

Figure 3.6: Running the customized user test
9. When the static analyzer is done running, you can close the status window when it says "Finished". A JTest view should appear at the bottom of the screen (if not do Window > Show View > Tasks).

Figure 3.7: The JTest execution window (after completion)
10. This is a "task list" of warnings that the analyzer generated for possible security vulnerabilities and coding standards violations. Expand the pluses to WebGoat > yourID > Fix Static Analysis Violations > Security Language Rules > Severity 3 > Consider using 'prepareStatement()'...

Figure 3.8: Viewing the SQL injection vulnerabilities
11. Double-click the first warning to show the location in the code. Note the following:
    • The static analyzer looks for calls to createStatement, not concatenation to a query string.
    • The static analyzer catches three instances of this, the other two are found in other, similar login methods.
    • Little warning symbols () are in the code to denote that a warning is on that line
12. Take about ten minutes to look through the other warnings caught by this static analysis run. If you have a partner, discuss with them the following questions:
    • Does this specific warning catch every kind of SQL injection vulnerability?
    • How many of these warnings represent real vulnerabilities?
    • Are there any violations that you disagree with as a whole?
    • Would fixing these warnings result in small code changes, or large code changes?
    • In this case, the static analyzer only looked at this one file. Is that enough information for determining how secure the method is? (e.g. look at "Encapsulate all dangerous data returning methods with a validation function [SPR.VPPD-2]").

Next, we are going to switch to the Login module for a different vulnerability type.

1. Go to the package DBSQLInjection and open that copy of Login.java. Navigate to the getAllEmployees method.

Figure 3.9: Inspecting the getAllEmployees method
2. Answer the following questions with your partner:
   • Is there an SQL injection vulnerability here? Why or why not?
   • What would the static analyzer report? Run the analyzer on this file to confirm your answer.
3. Last of all, go back to the login method of the SQLInjection package (JavaSource/org.owasp.webgoat.DBSQLInjection.Login, line 130), shown below

Figure 3.9: The code for the Login method in DBSQLInjection.java
4. Enter a few blank lines between the line that starts with ResultSet and the if statement. Put the following code in:

   				answer_results = WebSession.getConnection(s).prepareStatement(query).executeQuery();
   				

   So it looks like this (new line highlighted in blue):
   
   Figure 3.10: Modifying the code for DBSQLInjection.java
5. Run the static analyzer again on this file (make sure the correct file is selected in the Package Explorer before running!)
6. With your partner, answer the following questions:
   • Is this line a SQL Injection vulnerability? Why or why not?
   • Did the static analyzer flag this line with an SQL Injection vulnerability? If so, what type?
   • Can you come up with a code snippet (here or elsewhere )that is NOT vulnerable to SQL injection, but would be flagged? What would that look like?
   • How could the static analyzer be improved to find more SQL Injection vulnerabilities?
7. When you are done, be sure to delete your changes so that the code is restored to its original state (from when we checked it out). The easiest way to do this is to use SVN Revert. Right-click on the file you changed and go to Team > Revert.... Click Ok to approve reverting the code.

Figure 3.11: Reverting the code for DBSQLInjection.java

JTest also gathers metrics about the code as it is going through static analysis. For performance reasons, let's create a JTest configuration that only collects metrics.

1. Go to JTest > Test Configurations...
2. In the Built-in configurations, right-click the Metrics configuration and create a new child (similar to before).
3. In our new configuration, to the Static tab, then on the lower set of tabs go to the Metrics tab. Below is a list of all the metrics we can gather. The defaults are fine for this exercise, but feel free to check off any metrics you are interested in.

Figure 4.1: The list of metrics JTest can gather
4. As before, click Apply and Close - don't run it as it is (we don't know what we've selected to run this on!)
5. In the package explorer, select the package org.owasp.webgoat.CrossSideScripting. Run our metrics config on this: JTest > Test Using > User-Defined > Metrics
   
   Figure 4.2: Running the user-defined metrics gathering procedure
6. The test will execute as before.

Figure 4.3: Metrics gathering (after execution)
7. When the test is done running, close the status window as before. Open the Metrics view by going to Window > Show View > Metrics. Take a look at the metrics generated for this package of classes.

Figure 4.4: Viewing the resultant metrics
8. Note that this view is linked to the Package Explorer. Select the class EditProfile.java within the CrossSideScripting package. Note that the metrics change.
9. With your partner, discuss the following questions:
   • Which files are the most complex?
   • Are there any metrics that stick out as being indicative of Cross Site Scripting vulnerabilities?

JTest not only runs static analysis and gathers metrics, it also generates basic unit tests that can also reveal vulnerabilities. In this part of the tutorial, we will be generating unit tests on parts of the code that do not have full exception handling. Note that improper exception handling can lead to unknowingly leaking potentially valuable information.

1. Go to JTest > Test Configurations...
2. Right-click on the Built-in test Unit testing > Generate Unit Tests and choose "New Child".

Figure 5.1: Generating a new child for generation of unit tests
3. On our new run configuration, go to the Generation tab. Make sure the Enable Unit Test Generation box is checked. Keep the other options by default.

Figure 5.2: Selecting the default test generation options
4. Click Apply and Close - do not run yet!
5. In the package explorer, select the Login.java in org.owasp.webgoat.lessons.SQLInjection. Go to JTest > Test Using > User-Defined > Generate Unit Tests

Figure 5.3: Running the user-defined unit test generation procedure
6. This can take a little bit longer than the other tests. When it finishes, there should be a new project and the bottom of the Package Explorer called WebGoat.jtest

Figure 5.4: The WebGoat.jtest project
7. Run the unit test by going to the jtest.WebGoat > src > LoginTest. Right-click on the java file and go to Run as > JUnit test

Figure 5.5: Running the generated test
8. You can view the results in the JUnit view at the bottom of the screen. Note that some of the tests pass and some of them fail in the JUnit view at the bottom. Since these tests are intended to find a fault (maybe a vulnerability), a failing test ought to reveal a fault (maybe a vulnerability).

Figure 5.6: The generated test results
9. Double-click on the testGetAllEmployees1 failing test to go to the test code itself. With your partner, decide how the code in Login.java might be modified so that the test passes (without modifying the test!!).
10. With your partner, answer the following questions:
    • Are there tests that are wrong, that is, they reveal bugs or vulnerabilities that don't exist?
    • In general, does fixing a failing unit test ensure that a vulnerability is gone? Why or why not?

JTest does a whole lot more than we discuss here. JTest can perform more thorough analyses beyond security. Below are a few resources related to JTest.

• JTest website, Parasoft
• DevX: JUnit + JTest: Automated Test Case Design and Static Analyis
• OpenSeminar: Security
• CWE Top 25

To install the Tomcat runtime, follow these steps:

1. Go to Window > Show View > Other... > Servers

Figure 7.1: Viewing the installed servers
2. In the Servers view, right click and select "New"

Figure 7.2: Creating a new server
3. Select Apache > Tomcat 5.5. Click next.
4. For the installation, point the folder to a Tomcat installation. For NCSU CSC 591-003, use C:\tomcat\apache-tomcat-5.5.28 (otherwise you can download it if you need to). Click Finish
   
   Figure 7.3: Configuring the Tomcat parameters