Final PhD Examination — Dissertation Defense for  Sarah Heckman

Title: ” A Systematic Model Building Process for Predicting Actionable Static Analysis Alerts”

Date: May 11, 2009
Time: 9:00 a.m.
Place: EBII, Room 3211

Examination Committee:
Laurie Williams (chair & advisor)
Stephen Heber
Robert St. Amant
Tao Xie

All faculty and graduate students are invited.
Title: “A Systematic Model Building Process for Predicting Actionable Static Analysis Alerts”
Abstract:

Automated static analysis tools can identify potential source code anomalies, like null pointers, buffer overflows, and unclosed streams that could lead to field failures. These anomalies, which we call /alerts/, require inspection by a developer to determine if the alert is important enough to fix. Actionable alert identification techniques can supplement automated static analysis tools by classifying or prioritizing the alerts generated by automated static analysis such that the likelihood of a developer inspecting actionable alerts first is increased. By classifying and prioritizing actionable static analysis alerts, the developer will focus his or her time on inspecting and fixing actionable alerts rather than inspecting and suppressing unactionable alerts.

The goal of my research is to /reduce inspection time by accurately predicting actionable and unactionable alerts when using static analysis by creating and validating a systematic actionable alert identification model/. The Systematic Actionable Alert Identification (SAAI) process uses machine learning to identify actionable alerts. Investigation of the following three hypotheses will inform the goal of my research:

* Hypothesis 1: The artifact characteristics of an alert and the surrounding source code are predictive of the actionability of an alert.

* Hypothesis 2: A systematic actionable alert identification technique using machine learning can accurately identify actionable alerts.

*Hypothesis 3: A systematic actionable alert identification technique using machine learning is project specific.

A benchmark, FAULTBENCH provides the evaluation framework for the proposed SAAI model building process and comparison with other actionable alert identification techniques. The dissertation presents a feasibility study and three empirical studies evaluating the hypotheses above. The feasibility study evaluates an adaptive actionable alert identification technique that utilizes the alert’s type and code location in addition to developer feedback to prioritize actionable alerts. The first empirical study investigates hypotheses 1-4 using FAULTBENCH on 15 SAAI models generated on five treatments for each of three subject programs. The treatments considered different grouping of alerts within revisions to train and test SAAI. The second empirical study is a comparative evaluation of the generated SAAI models with other actionable alert identification technique in further evaluation of Hypothesis 2. Additionally, an empirical user study was conducted where students in the senior capstone project course used a custom SAAI model during development of their software project.

The evidence from the three empirical studies support Hypotheses 1 and 2. All but one of the 57 artifact characteristics used to build systematic actionable alert identifier models were in one or more of the artifact characteristics subsets. Additionally, eight of the 15 FAULTBENCH subject treatments reported accuracy greater than 90% when using a SAAI model. When comparing SAAI models with other actionable alert identification techniques from literature found that SAAI models had the highest accuracy for 11 of the 15 treatments. Both of the above results support hypothesis 2. Hypothesis 3 is not supported because the accuracies are greater than 90% when an attribute subset and machine learning algorithm selected for one subject program is used on another subject program.

The contributions of this work are as follows:

* A systematic actionable alert identifier model building process to predict actionable and unactionable automated static analysis alerts;

* A benchmark, FAULTBENCH , for evaluating and comparing actionable alert identification techniques; and

* A comparative evaluation of systematic actionable alert identifier models with other actionable alert identification techniques from literature.