The wind is not yet blowing in software engineering research
An article by Andrew Gelman is getting a lot of well deserved publicity at the moment. The topic of discussion is sloppy research practices in psychology and how researchers are responding to criticism (head in the sand and blame the messenger).
I imagine that most software developers think this is an intrinsic problem in the ‘soft’ sciences that does not apply to the ‘hard’ sciences, such as software; I certainly thought this until around 2000 or so. Writing a book containing a detailed analysis of C convinced me that software engineering was mostly opinion, with a tiny smattering of knowledge in places.
The C book tried to apply results from cognitive psychology to what software developers do. After reading lots of books and papers on cognitive psychology I was impressed with how much more advanced, and rigorous, their experimental methods were, compared to software engineering.
Writing a book on empirical software engineering has moved my views on to the point where I think software engineering is the ideal topic for the academic fraudster.
The process of cleaning up their act that researchers in psychology are going through now is something that software engineering will have to go through. Researchers have not yet reached the stage of directly pointing out that software engineering research is a train wreck. Instead, they write parody papers and polite article showing how dirty popular datasets are.
Of course, industry is happy to keep quiet. The development of software systems is still a sellers market (the packaged market is dog eat dog) or at least that is still the prevalent mentality. I doubt anything much will change until the production of software systems becomes a buyers market, which will create a real incentive for finding out what really works.
I’ve retired from a professorship in Computing and am now doing a PhD in Philosophy of Science related to Computing, in respect of which I am very interested in Derek’s comments about Empirical Software Engineering, research standards and the gap between academic and industrial experience of software development. My own take on this is that Empirical Software Engineering has developed over the last 20 years to a state where we can see that Software is a phenomenon about which we do have – and can extend – experimentally warranted knowledge, and as a result of this it can be seen that not all knowledge claims based on professional practice are actually valid. I worry that Philosophers of Science (and Technology) are too ready to assume that actual practice must be correct – in this they have swung to the opposite extreme from the philosophers of my youth, who thought they could tell scientists what they must do, on the basis of examples that had nothing to do with what scientists actually did do. Because of Empirical Software Engineering I do think Software Engineering has a better scientific basis than “Computer Science” – unfortunately Computer Science has more prestige in top universities. I argue that Software Engineering is not a science but a practice, whereas Empirical Software Engineering *is* a science that supports Software Engineering Practice.
@Julian Newman
A practical view of the philosophy of computing would be make money before what you have is superseded; in academia it would be publish and quickly move onto something else. The underlying problem, with respect to understanding what is going on, is that things change so quickly. What is needed is a study of the anthropology of computing; no need to visit darkest Africa to find strange practices, they can be found in any software development group.
Yes, software engineering is a craft activity, there are groups, book and conferences that promote this very idea.
“… not all knowledge claims based on professional practice are actually valid.” I don’t think anybody in industry (apart from clueless managers) think professional practitioners know a lot about what they are doing. However, I think professional practice is often closer to what might be considered the optimal approach than the academic practices. This is because professionals should have a reasonable idea of what is supposed to happen, so at least they know where they would like to end up.
I think the only difference between academic computing departments with Science or Engineering in their name is marketing; how they would like to be perceived, or at least how the Dean wants to be perceived. If data on departments with Science/Engineering in their name was available, do you think it would be possible to predict which was which (just from the non-name data)?
Perhaps things are changing, thiry years after the fall of Rome data is starting to become available again.