Archive
Classification of code updates
Version control systems usually include support for classifying each check-in as being a particular instance of some category, e.g., Adaptive, Corrective, Perfective, or Other (this software maintenance category has a long history).
Categories have been studied for over 2,000 years. Categories is the first subject of Aristotle’s six works on logical analysis and dialectic.
Categorization is used to perform inductive reasoning (i.e., the derivation of generalized knowledge from specific instances), and also acts as a memory aid (about the members of a category). Categories provide a framework from which small amounts of information can be used to infer, seemingly unconnected (to an outsider), useful conclusions.
Experimental work on human classification has produced a variety of theories. See “Classification and Cognition” by W. K. Estes for an evidence-based analysis.
The evidence clearly shows that the boundaries between different members of a category are fuzzy. How consistent are developers in assigning software activities to the same instance of a category?
The study Determining the Distribution of Maintenance Categories: Survey versus Measurement by Schach, Jin, Yu, Heller, and Offutt, asked two people to categorise 215 maintenance changes involving the first 20 versions of Linux. The rows/columns in the table below show the number of changes assigned to each category by each person:
Adaptive Corrective Perfective Other Total Adaptive 2 0 0 0 2 Corrective 0 82 16 0 98 Perfective 0 5 99 2 106 Other 0 0 0 9 9 Total 2 87 115 11 215 |
The terminology used in the statistics analysis of agreement uses the term “raters” to refer to the people who classify items into categories. Cohen’s Kappa, 
, is a measure of agreement between two raters (not more), which varies from zero (i.e., raters chose at random) to one (perfect agreement). For this data (code+data): 
, which is very good agreement between the two raters.
Another classification dataset is described in the paper Two datasets of defect reports labelled by a crowd of annotators of unknown reliability by Hernández-González, Rodriguez, Inza, Harrison, and Lozano, where five unspecified people classified two defect datasets (the paper had five authors, hmm; the associated paper does not use any established statistical technique to measure rater agreement {the paper’s really about something else}), one containing 962 defects from the Compendium project, and the other 675 defects from the Mozilla project.
The study used the Orthogonal defect classification (ODC) defect impact classification (ODC involves eight different classifications, some easier to assign than others), which contains 13 impacts, i.e., Bug, Capability, Documentation, Feature, Installability, Integrity/Security, Migration, Performance, Reliability, Requirements, Standards, Support, Usability. The Mozilla data does not contain instances of three defect impacts: Serviceability, Standards, and Accessibility.
Conger’s kappa is an extension of Cohen’s kappa to handle more than two raters, and is applicable when all raters classify all items (which they do here). Fleiss’s kappa is the brand-name technique for this kind of data, and does not require that all raters classified all items. This experimental design matches the assumptions made in the derivation of Conger’s kappa, and do not match those made for Fleiss’s kappa.
The 95% confidence intervals for the two sets of agreements between five-raters are (code+data): 0.26 to 0.29 (962 items) and 0.28 to 0.33 (675 items). Both have fair to poor agreement.
I was not surprised by the poor inter-rater agreement for these two datasets. There is a lot of domain specific knowledge is needed to assign some of the members of the ODC defect impact classification, e.g., Installability and Migration. As outsiders (I assume), the raters in this study only had the text associated with the defect report to make their decision.
Does ODC only achieve the claimed benefits if the people using it have been trained so that their agreement rate is higher? I am not aware of any other studies comparing ODC defect classification consistency. But then, studies like this rarely get done anyway.
Predictive Modeling: 15th COW workshop
I was at a very interesting workshop on Predictive Modelling and Search Based Software Engineering on Monday/Tuesday this week, and I am going to say something about the talks that interested me. The talks were recorded, and the videos will appear on the website in a few weeks. The CREST Open Workshop (COW) runs roughly once a month and the group leader, Mark Harman, is always on the lookout for speakers, do let him know if you are in the area.
- Tim Menzies talked about how models built from one data set did well on that dataset but often not nearly as well on another (i.e., local vs global applicability of models). Academics papers usually fail to point out that any results might not be applicable outside the limited domain examined, in fact they often give the impression of being generally applicable.
Me: Industry likes global solutions because it makes life simpler and because local data is often not available. It is a serious problem if, for existing methods, data on one part of a companies’ software development activity is of limited use in predicting something about a different development activity in the same company and completely useless at predicting things at a different company.
- Yuriy Brun talked about something that is so obviously a good idea, it is hard to believe that it had not been done years ago. The idea is to have your development environment be aware of what changes other software developers have made to their local copies of source files you also have checked out from version control. You are warned as soon your local copy conflicts with somebody else’s local copy, i.e., a conflict would occur if you both check in your local copy to the central repository. This warning has the potential to save lots of time by having developers talk to each about resolving the conflict before doing any more work that depends on the conflicting change.
Crystal is a plug-in for Eclipse that implements this functionality, and Visual Studio support is expected in a couple of releases time.
I have previously written about how multi-core processors will change software development tools, and I think this idea falls into that category.
- Martin Shepperd presented a very worrying finding. An analysis of the results published in 18 papers dealing with fault prediction found that the best predictor (over 60%) of agreement between results in different papers was co-authorship. That is, when somebody co-authored a paper with another person, any other papers they published were more likely to agree with other results published by that person than with results published by somebody they had not co-authored a paper with. This suggests that each separate group of authors is doing something different that significantly affects their results; this might be differences in software packages being used, differences in configuration options or tuning parameters, so something else.
It might be expected that agreement between results would depend on the techniques used, but Shepperd et al’s analysis found this kind of dependency to be very small.
An effect is occurring that is not documented in the published papers; this is not how things are supposed to be. There was lots of interest in obtaining the raw data to replicate the analysis.
- Camilo Fitzgerald talked about predicting various kinds of feature request ‘failures’ and presented initial results based on data mined from various open source projects; possible ‘failures’ included a new feature being added and later removed and significant delay (e.g., 1 year) in implementing a requested feature. I have previously written about empirical software engineering only being a few years old and this research is a great example of how whole new areas of research are being opened up by the availability of huge amounts of data on open source projects.
One hint for PhD students: It is no good doing very interesting work if you don’t keep your web page up to date, so people can find out more about it
I talked to people who found other presentations very interesting. They might have failed to catch my eye because my interest or knowledge of the subject is low or I did not understand their presentation (a few gave no background or rationale and almost instantly lost me); sometimes the talks during coffee were much more informative.
Recent Comments