Realsearch

Secure Open Source Collaboration: An Empirical Study of Linus’ Law

Andy Meneely — Fri, 28 Aug 2009 12:43:30 +0000

Andrew Meneely and Laurie Williams. Computer and Communications Security (CCS) 2009.

Open source software is often considered to be secure. One factor in this confidence in the security of open source software lies in leveraging large developer communities to find vulnerabilities in the code. Eric Raymond declares Linus’ Law “Given enough eyeballs, all bugs are shallow.” Does Linus’ Law hold up ad infinitum? Or, can the multitude of developers become “too many cooks in the kitchen”, causing the system’s security to suffer as a result? In this study, we examine the security of an open source project in the context of developer collaboration. By analyzing version control logs, we quantified notions of Linus” Law as well as the “too many cooks in the kitchen” viewpoint into developer activity metrics. We performed an empirical case study by examining correlations between the known security vulnerabilities in the open source Red Hat Enterprise Linux 4 kernel and developer activity metrics. Files developed by otherwise-independent developer groups were more likely to have a vulnerability, supporting Linus’ Law. However, files with changes from nine or more developers were 16 times more likely to have a vulnerability than files changed by fewer than nine developers, indicating that many developers changing code may have a detrimental effect on the system’s security.

Should software testers use mutation analysis to augment a test set?

Ben Smith — Tue, 11 Aug 2009 06:44:47 +0000

BB.H. Smith and L. Williams, Should software testers use mutation analysis to augment a test set? Journal of Systems Software, vol. 82, no. 11, pp. 1819-1832, 2009.

Mutation testing has historically been used to assess the fault-finding effectiveness of a test suite or other verification technique. Mutation analysis, rather, entails augmenting a test suite to detect all killable mutants. Concerns about the time efficiency of mutation analysis may prohibit its widespread, practical use. The goal of our research is to assess the effectiveness of the mutation analysis process when used by software testers to augment a test suite to obtain higher statement coverage scores. We conducted two empirical studies and have shown that mutation analysis can be used by software testers to effectively produce new test cases and to improve statement coverage scores in a feasible amount of time. Additionally, we find that our user study participants view mutation analysis as an effective but relatively expensive technique for writing new test cases. Finally, we have shown that the choice of mutation tool and operator set can play an important role in determining how efficient mutation analysis is for producing new test cases.

Williams Receives Inaugural ACM SIGSOFT Influential Educator Award

Sarah Heckman — Wed, 24 Jun 2009 19:12:51 +0000

Congratulations to Laurie on receiving the first ACM SIGSOFT Influential Educator Award at ICSE 2009! Check out the CSC news story.

Final PhD Examination — Dissertation Defense for Sarah Heckman

Ben Smith — Mon, 04 May 2009 21:06:07 +0000

Final PhD Examination — Dissertation Defense for Sarah Heckman

Title: ” A Systematic Model Building Process for Predicting Actionable Static Analysis Alerts”

Date: May 11, 2009
Time: 9:00 a.m.
Place: EBII, Room 3211

Examination Committee:
Laurie Williams (chair & advisor)
Stephen Heber
Robert St. Amant
Tao Xie

All faculty and graduate students are invited.
Title: “A Systematic Model Building Process for Predicting Actionable Static Analysis Alerts”
Abstract:

Automated static analysis tools can identify potential source code anomalies, like null pointers, buffer overflows, and unclosed streams that could lead to field failures. These anomalies, which we call /alerts/, require inspection by a developer to determine if the alert is important enough to fix. Actionable alert identification techniques can supplement automated static analysis tools by classifying or prioritizing the alerts generated by automated static analysis such that the likelihood of a developer inspecting actionable alerts first is increased. By classifying and prioritizing actionable static analysis alerts, the developer will focus his or her time on inspecting and fixing actionable alerts rather than inspecting and suppressing unactionable alerts.

The goal of my research is to /reduce inspection time by accurately predicting actionable and unactionable alerts when using static analysis by creating and validating a systematic actionable alert identification model/. The Systematic Actionable Alert Identification (SAAI) process uses machine learning to identify actionable alerts. Investigation of the following three hypotheses will inform the goal of my research:

* Hypothesis 1: The artifact characteristics of an alert and the surrounding source code are predictive of the actionability of an alert.

* Hypothesis 2: A systematic actionable alert identification technique using machine learning can accurately identify actionable alerts.

*Hypothesis 3: A systematic actionable alert identification technique using machine learning is project specific.

A benchmark, FAULTBENCH provides the evaluation framework for the proposed SAAI model building process and comparison with other actionable alert identification techniques. The dissertation presents a feasibility study and three empirical studies evaluating the hypotheses above. The feasibility study evaluates an adaptive actionable alert identification technique that utilizes the alert’s type and code location in addition to developer feedback to prioritize actionable alerts. The first empirical study investigates hypotheses 1-4 using FAULTBENCH on 15 SAAI models generated on five treatments for each of three subject programs. The treatments considered different grouping of alerts within revisions to train and test SAAI. The second empirical study is a comparative evaluation of the generated SAAI models with other actionable alert identification technique in further evaluation of Hypothesis 2. Additionally, an empirical user study was conducted where students in the senior capstone project course used a custom SAAI model during development of their software project.

The evidence from the three empirical studies support Hypotheses 1 and 2. All but one of the 57 artifact characteristics used to build systematic actionable alert identifier models were in one or more of the artifact characteristics subsets. Additionally, eight of the 15 FAULTBENCH subject treatments reported accuracy greater than 90% when using a SAAI model. When comparing SAAI models with other actionable alert identification techniques from literature found that SAAI models had the highest accuracy for 11 of the 15 treatments. Both of the above results support hypothesis 2. Hypothesis 3 is not supported because the accuracies are greater than 90% when an attribute subset and machine learning algorithm selected for one subject program is used on another subject program.

The contributions of this work are as follows:

* A systematic actionable alert identifier model building process to predict actionable and unactionable automated static analysis alerts;

* A benchmark, FAULTBENCH , for evaluating and comparing actionable alert identification techniques; and

* A comparative evaluation of systematic actionable alert identifier models with other actionable alert identification techniques from literature.

Does Calling Structure Information Improve the Accuracy of Fault Prediction?

yshin2 — Tue, 28 Apr 2009 21:59:19 +0000

Yonghee Shin, Robert Bell, Thomas Ostrand, and Elaine Weyuker,
“Does Calling Structure Information Improve the Accuracy of Fault Prediction?“,
The 6th IEEE Working Conference on Mining Software Repositories (MSR 2009), co-located with ICSE 2009, May 16-17, 2009, Vancouver, Canada (To appear)

Sarah Heckman to teach at NCSU

Ben Smith — Wed, 15 Apr 2009 00:26:16 +0000

We at RealSearch are proud to announce that Ms. Sarah Heckman will be starting August 16th, 2009 in her new Teaching Assistant Professor position! Congratulations, Sarah!

Congratulations, Ben Smith!

ljhaywar — Sun, 29 Mar 2009 18:49:15 +0000

We congratulate Ben Smith for passing his written qualifier examination! Ben will be receiving his Master’s Degree in May. Way to go, Ben!

Congratulations, Lauren Hayward

admin — Sun, 29 Mar 2009 18:29:32 +0000

We would like to congratulate Lauren Hayward on passing her examination and graduating this May with her Master’s of Science in Computer Science.

Jazz Sangam: A Real-time Tool for Distributed Pair Programming of a Team Development Platform

Ben Smith — Tue, 20 Jan 2009 12:55:00 +0000

Devide, J., Meneely, A., Ho, C-w, Williams, L., and Devetisikiotis, M., Jazz Sangam: A Real-time Tool for Distributed Pair Programming of a Team Development Platform, Infrastructure for Research on Collaborative Software Engineering (IReCoSE) workshop at ACM SIGSOFT Foundations of Software Engineering (FSE), Atlanta, GA, to appear.

On Preparing Students for Distributed Software Development with a Synchronous, Collaborative Development Platform

Ben Smith — Thu, 01 Jan 2009 20:55:36 +0000

Meneely, A. and Williams, L., On Preparing Students for Distributed Software Development with a Synchronous, Collaborative Development Platform, ACM Technical Symposium on Computer Science Education (SIGCSE) 2009, Chatanooga, TN, to appear.