The Shape of Code

Fault reports are generated by a program’s users, and the extreme difficulty of obtaining any information about how users use a program and number of users makes it almost impossible to do reliable fault prediction.

Source code is often available. Are there any source code characteristics that could be used to make fault predictions?

A significant number of published software fault prediction papers are based on the idea that the code contained in the functions modified to fix reported faults have ‘woo‘ characteristics that is not present in the code of other functions. Discovering these woo characteristics would make it possible to reliably predict faults. Countless hours of machine learning have been invested on the search for woo.

Faults are reported in the code that users execute, and the more often the code is executed, the more opportunities there are for fault triggering input values to occur.

Functions that have been modified to fix faults tell us something about the code that is executed by users. If a function in file A calls a function in file B which in turn calls a function in file C, and file B has be modified to fix a fault, then we know that a function in A has been called, and perhaps also in C (yes, call-chains are usually links between functions, not links between the files that contain them; most files contain only a few functions). Are files in the call-chain neighbourhood of a fault-fixed file more likely, in the future, to be modified because of a reported fault, than files that are not in such a call-chain neighbourhood?

The paper Do Bugs Propagate? An Empirical Analysis of Temporal Correlations Among Software Bugs by Gu, Han, Kim and Zhang extracted data on files/functions from multiple releases of the four systems: HTTPClient, Jackrabbit, Lucene, and Rhino, including LOC, call-graph, number of changes/authors, and number of fixed faults.

For a particular system, data on each file/function and its neighbourhood in release was then used to build a regression model that predicts the likelihood of each file needing to be changed to fix a reported fault in the next release, .

The technical details of the regression fitting process for this data are more complicated than usual, and are discussed at the end of the post. The important question is whether information on call-chain neighbourhood fault reports have a worthwhile impact on the performance of a fault prediction model.

Yes, call-chain neighbourhood information does make a worthwhile improvement to fault prediction in the next release (as expected, LOC has a big impact).

The authors also built a co-change graph (i.e., files modified in the same commit), and a type hierarchy graph (i.e., is a class in file X extends a class in file B, an edge from A to B is created). Information on these neighbourhoods made a worthwhile improvement to fault prediction in the next release.

The authors of the paper give a purely code focused explanation of the behavior, i.e., the coding mistake propagate within a neighbourhood. Perhaps code does move between functions in the same file. To distinguish between mistake propagation and user usage the call-graph analysis needs to be function-based, rather than file-based.

This data longitudinal because it follows the same subject (i.e., each file) through time (i.e., each release), and the response variable is to be fitted to a logistic equation. The statistical technique used to fit this kind of data the generalized estimating equation, a form of generalized linear model (the regression technique used in many of these blog posts) that handles correlation between observations, i.e., the same file is measured multiple times.

The analysis code that comes with the data is written in Matlab (Octave is an Open source mostly compatible program). To understand the analysis (in correlation_analysis.m), I implemented it in R (code and data). The coefficients of the fitted models are different from those given in the paper, but directionally the same. Octave had issues with the statistical library used for the analysis, so it was not possible to replicate the regression model coefficients.