Archive
Software effort estimation in 2026
It’s 2.5 years since my 2024 post summarizing what I knew about software effort estimation. For ongoing projects the role of estimation is as a planning tool, breaking down a large chunk of work into what are estimated to be small, well-defined, chunks. With two-thirds of estimates being within a factor of four (over or under) of the actuals, accuracy was not a primary goal.
The impact of coding agents capable of implementing these small tasks for peanuts is on developer staffing, not on the need for planning and breaking down work into connected small chunks.
Coding agents follow the specification they are given. Large, complicated programs will have large complicated specifications, which are likely to require a lot of human effort to create. For instance, the specification for the recent LLM generated C compiler
How long will it take to create a specification and how many iterations will be needed to create a good enough program? Estimation has not gone away, it has just moved upstream of the implementation. While specifications have existed millennia longer than software development, we are at the start of learning to use them to direct coding agents (the term vibe coding was created in February 2025). While logs of developer coding agent sessions are starting to be collected, it will take some years for patterns of practice to stabilize.
The process followed to estimate Function Points appears to be ideally placed to be extended to cover specification creation for coding agents.
Peanuts in bulk can be expensive. The novelty of much lower costs, when using coding agents, will soon wear off. Agent costs will become the new normal, which means there will be pressure to reduce them, and management will want estimates because the requirement to fill in boxes has not gone away.
Is agent token consumption purely a driven by the LLM(s) used and the selected configuration values? A recent study investigated the impact of the wording of the specification on the tokens spent by a selection of agents. The results showed that a specification containing bare user stories increased token spending by 30%. We will have to wait to find out whether the implications of this result (i.e., investing in more detailed user stories reduces token spending) replicates.
It would be very useful to have an estimate of likely token usage for a given specification, before any code is generated, or perhaps a real-time cost estimate during generation. One study found that current frontier models failed to accurately predict their own token usage (correlation of 0.39), humans predictions were equally poor.
Many projects don’t have a separate line item for testing costs, which often get bundled with implementation costs (the SEA Data Warehouse dataset separates out design/implementation/testing effort). Large projects may have people who work on integration and/or release testing.
When coding agents are used, testing is clearly a distinct piece of work separate from implementation. How effective is the specification used to generate the code at generating good enough tests? Current research is an extension of existing testing practices, rather than anything coding agent specific. A lot of existing research has focused on unit testing, and of course models have been fine-tuned for unit testing.
Implementation cost reductions have brought N-version programming into the realm of consideration for some projects.
Will the way developers work change to be like tradesmen who are hired to install a new fitted kitchen or bathroom? With developers installing new application functionality specified by the client. Perhaps some will be more like builders hired to make structural changes to a house, while others will do small jobs, like plumbers, electricians and carpet fitters.
As always, some researchers fail to move on when the world changes and continue to publish papers that use LLMs to estimate human task implementation time.
Specification based programming
The use of LLM to write software has focused on integrating them within existing practices, i.e., using LLMs as very fancy auto-completers for chunks of code or functionality. This use is programming by conversation, or less politely, programming by stream of thought. The term vibe-coding creates an illusion of trendiness; after all, software engineering is a hedonistic activity.
With vibe-code on top of vibe-code on top of vibe-code, refactoring becomes a complete rewrite, at least in theory. A rewrite assumes that it’s possible to extract a specification that is complete and accurate enough to recreate the software. A lot of software has a short lifetime, so a major rewrite may never be needed. However, for software that is expected to have a long life, management are going to want a more controlled/structured/repeatable approach.
LLMs’ ability to write software is now good enough to support a more controlled/structured/repeatable approach: Programming by specification. That is a specification of the desired behavior is given to one or more LLMs, which use it to generate the appropriate software.
The human input to the program creation process is via the specification.
Features are changed/added/removed by updating the specification. Bugs are fixed by updating the specification. If there are mistakes in the generated code, the specification has to work around them, in the same way that compiler bugs have to be worked around.
Business logic can be expressed as a specification, which is how application domain experts, who are not programmers, are able to create minimal viable products using LLMs.
How might a specification be created?
Agile has taught the lesson that software creation is an iterative process. Requiring a complete specification before coding starts is the stuff of armchair project managers.
One possible specification iteration process starts with a basic outline specification of what is required, and is followed by the following cycle:
- Using the current specification, developer+LLM produces code. Perhaps particular functionality is implemented, or the work continues for some amount of time, or etc,
- the transcript of the LLM conversation is used to create an updated specification of the code that exists when work stopped. Conversations involving code that came and went is not part of the updated specification, although logging it for future reference costs little,
- a new version of all the software covered by the updated specification is generated. This can be tested using existing tests and also by differential testing using multiple implementations created from the same specification (a recent paper generated five implementations in different languages),
- if more functionality is needed, go to step 1.
Specifications share many characteristics with source code. They can be split up and organized into modules/packages/components/phases, as was done for this LLM generated C compiler.
LLM generated code is more verbose than human generated code, just like the machine code generated by early compilers.
Open source projects could soon just be making the specification available. Why ship the source code generated from a specification, projects don’t ship the assembler code generated by compilers, they ship the original source code. However, given the current reliability of LLM source code generation, they are benefits to making the generated source of at least one implementation available (as a kind of checksum).
Reduced implementation costs, using LLMs, make it possible to create programs containing more functionality (Jevrons paradox in action). This in turned leads to specifications becoming larger, complicated and poorly organized, just like source code.
English usage is full of ambiguities. This ambiguity can be reduced by using a controlled language. If specification programming becomes popular, it’s easy to imagine the invention of controlled languages becoming as popular as the invention of programming languages. In 1957, there were compilers for at least 28 programming languages.
Specification based programming is a continuation of the trend of computers handling more of the details involved in program creation, with the program creation process requiring less and less knowledge about computers. Increasing amounts of computer time are spent to reduce or eliminate developer time.
Programming has evolved from physically connecting subsystems by cables to specify the flow of bits in a punch card computer, to a sequence of machine code instructions executed by a stored-program computer, then high-level programming languages reducing the need to know lots of details about the underlying cpu (details that remain include: number of bits in the integer types and type compatibility rules).
Specification based programming requires discipline, and I don’t expect it to be popular. I expect multiple LLM-derived project disasters need to occur before there are any significant changes to the current LLM approaches to software development.
Payback time-frame for research in software engineering
What are the major questions in software engineering that researchers should be trying to answer?
A high level question whose answer is likely to involve life, the universe, and everything is: What is the most cost-effective way to build software systems?
Viewing software engineering research as an attempt to find the answer to a big question mirrors physicists quest for a Grand Unified Theory of how the Universe works.
Physicists have the luxury of studying the Universe at their own convenience, the Universe does not need their input to do a better job.
Software engineering is not like physics. Once a software system has been built, the resources have been invested, and there is no reason to recreate it using a more cost-effective approach (the zero cost of software duplication means that manufacturing cost is the cost of the first version).
Designing and researching new ways of building software systems may be great fun, but the time and money needed to run the realistic experiments needed to evaluate their effectiveness is such that they are unlikely to be run. Searching for more cost-effective software development techniques by paying to run the realistic experiments needed to evaluate them, and waiting for the results to become available, is going to be expensive and time-consuming. A theory is proposed, experiments are run, results are analysed; rinse and repeat until a good-enough cost-effective technique is found. One iteration will take many years, and this iterative process is likely to take many decades.
Very many software systems are being built and maintained, and each of these is an experiment. Data from these ‘experiments’ provides a cost-effective approach to improving existing software engineering practices by studying the existing practices to figure out how they work (or don’t work).
Given the volume of ongoing software development, most of the payback from any research investment is likely to occur in the near future, not decades from now; the evidence shows that source code has a short and lonely existence. Investing for a payback that might occur 30-years from now makes no sense; researchers I talk to often use this time-frame when I ask them about the benefits of their research, i.e., just before they are about to retire. Investing in software engineering research only makes economic sense when it is focused on questions that are expected to start providing payback in, say, 3-5 years.
Who is going to base their research on existing industry practices?
Researching existing practices often involves dealing with people issues, and many researchers in computing departments are not that interested in the people side of software engineering, or rather they are more interested in the computer side.
Algorithm oriented is how I would describe researchers who claim to be studying software engineering. I am frequently told about the potential for huge benefits from the discovery of more efficient algorithms. For many applications, algorithms are now commodities, i.e., they are good enough. Those with a career commitment to studying algorithms have a blinkered view of the likely benefits of their work (most of those I have seen are doing studying incremental improvements, and are very unlikely to make a major break through).
The number of researchers studying what professional developers do, with an aim to improving it, is very small (I am excluding the growing number of fake researchers doing surveys). While I hope there will be a significant growth in numbers, I’m not holding my breadth (at least in the short term; as for the long term, Planck’s experience with quantum mechanics was: “Science advances one funeral at a time”).
Recent Comments