The Shape of Code

Version control systems usually include support for classifying each check-in as being a particular instance of some category, e.g., Adaptive, Corrective, Perfective, or Other (this software maintenance category has a long history).

Categories have been studied for over 2,000 years. Categories is the first subject of Aristotle’s six works on logical analysis and dialectic.

Categorization is used to perform inductive reasoning (i.e., the derivation of generalized knowledge from specific instances), and also acts as a memory aid (about the members of a category). Categories provide a framework from which small amounts of information can be used to infer, seemingly unconnected (to an outsider), useful conclusions.

Experimental work on human classification has produced a variety of theories. See “Classification and Cognition” by W. K. Estes for an evidence-based analysis.

The evidence clearly shows that the boundaries between different members of a category are fuzzy. How consistent are developers in assigning software activities to the same instance of a category?

The study Determining the Distribution of Maintenance Categories: Survey versus Measurement by Schach, Jin, Yu, Heller, and Offutt, asked two people to categorise 215 maintenance changes involving the first 20 versions of Linux. The rows/columns in the table below show the number of changes assigned to each category by each person:

          Adaptive Corrective Perfective  Other     Total
Adaptive     2         0          0        0           2
Corrective   0        82         16        0          98
Perfective   0         5         99        2         106
Other        0         0          0        9           9
 
Total        2        87        115       11         215

The terminology used in the statistics analysis of agreement uses the term “raters” to refer to the people who classify items into categories. Cohen’s Kappa, , is a measure of agreement between two raters (not more), which varies from zero (i.e., raters chose at random) to one (perfect agreement). For this data (code+data): , which is very good agreement between the two raters.

Another classification dataset is described in the paper Two datasets of defect reports labelled by a crowd of annotators of unknown reliability by Hernández-González, Rodriguez, Inza, Harrison, and Lozano, where five unspecified people classified two defect datasets (the paper had five authors, hmm; the associated paper does not use any established statistical technique to measure rater agreement {the paper’s really about something else}), one containing 962 defects from the Compendium project, and the other 675 defects from the Mozilla project.

The study used the Orthogonal defect classification (ODC) defect impact classification (ODC involves eight different classifications, some easier to assign than others), which contains 13 impacts, i.e., Bug, Capability, Documentation, Feature, Installability, Integrity/Security, Migration, Performance, Reliability, Requirements, Standards, Support, Usability. The Mozilla data does not contain instances of three defect impacts: Serviceability, Standards, and Accessibility.

Conger’s kappa is an extension of Cohen’s kappa to handle more than two raters, and is applicable when all raters classify all items (which they do here). Fleiss’s kappa is the brand-name technique for this kind of data, and does not require that all raters classified all items. This experimental design matches the assumptions made in the derivation of Conger’s kappa, and do not match those made for Fleiss’s kappa.

The 95% confidence intervals for the two sets of agreements between five-raters are (code+data): 0.26 to 0.29 (962 items) and 0.28 to 0.33 (675 items). Both have fair to poor agreement.

I was not surprised by the poor inter-rater agreement for these two datasets. There is a lot of domain specific knowledge is needed to assign some of the members of the ODC defect impact classification, e.g., Installability and Migration. As outsiders (I assume), the raters in this study only had the text associated with the defect report to make their decision.

Does ODC only achieve the claimed benefits if the people using it have been trained so that their agreement rate is higher? I am not aware of any other studies comparing ODC defect classification consistency. But then, studies like this rarely get done anyway.