I don't have any research papers or statistics to give you, but I'll relate my experience from working in a team/organization that historically had low-to-average unit test coverage and no end-to-end tests, and gradually moving the bar to where we are now, with more of an ATDD (but, ironically, not traditional TDD) approach.
Specifically, this is how project timelines used to play out (and still play out on other teams/products in the same organization):
- Up to 4 weeks of analysis and implementation
- 2 weeks of regression testing, bug fixing, stabilization, and release prep
- 1-2 weeks of fixing known defects
- 2-3 weeks of code cleanup and post-production issues/support (unknown defects/unplanned outages)
This seems like ridiculous overhead but it's actually very common, it's just often masked in many organizations by missing or ineffectual QA. We have good testers and a culture of intensive testing, so these issues are caught early, and fixed up front (most of the time), rather than being allowed to play out slowly over the course of many months/years. 55-65% maintenance overhead is lower than the commonly-accepted norm of 80% of the time being spent on debugging - which seems reasonable, because we did have some unit tests and cross-functional teams (including QA).
During our team's first release of our latest product, we had started retrofitting acceptance tests but they weren't quite up to snuff and we still had to rely on a lot of manual testing. The release was somewhat less painful than others, IMO partly because of our haphazard acceptance tests and also partly because of our very high unit test coverage relative to other projects. Still, we spent nearly 2 weeks on regression/stabilization and 2 weeks on post-production issues.
By contrast, every release since that initial release has had early acceptance criteria and acceptance tests, and our current iterations look like this:
- 8 days of analysis and implementation
- 2 days of stabilization
- 0-2 combined days of post-production support and cleanup
In other words, we progressed from 55-65% maintenance overhead to 20-30% maintenance overhead. Same team, same product, main difference being the progressive improvement and streamlining of our acceptance tests.
The cost of maintaining them is, per sprint, 3-5 days for a QA analyst and 1-2 days for a developer. Our team has 4 developers and 2 QA analysts, so (not counting UX, project management, etc.) that's a maximum of 7 man-days out of 60, which I'll round up to a 15% implementation overhead just to be on the safe side.
We spend 15% of each release period developing automated acceptance tests, and in the process are able to cut 70% of each release doing regression tests and fixing pre-production and post-production bugs.
You might have noticed that the second timeline is much more precise and also much shorter than the first. That's something that was made possible by the up-front acceptance criteria and acceptance tests, because it vastly simplifies the "definition of done" and allows us to be much more confident in the stability of a release. No other teams have (so far) succeeded with a bi-weekly release schedule, except perhaps when doing fairly trivial maintenance releases (bugfix-only, etc.).
Another interesting side-effect is that we've been able to adapt our release schedule to business needs. One time, we had to lengthen it to about 3 weeks to coincide with another release, and were able to do so while delivering more functionality but without spending any extra time on testing or stabilization. Another time, we had to shorten it to about 1½ weeks, due to holidays and resource conflicts; we had to take on less dev work, but, as expected, were able to spend correspondingly less time on testing and stabilization without introducing any new defects.
So in my experience, acceptance tests, especially when done very early in a project or sprint, and when well-maintained with acceptance criteria written by the Product Owner, are one of the best investments you can make. Unlike traditional TDD, which other people correctly point out is focused more on creating testable code than defect-free code - ATDD really does help catch defects a lot faster; it's the organizational equivalent of having an army of testers doing a complete regression test every day, but way cheaper.
Will ATDD help you in longer-term projects done in RUP or (ugh) Waterfall style, projects lasting 3 months or more? I think the jury's still out on that one. In my experience, the biggest and ugliest risks in long-running projects are unrealistic deadlines and changing requirements. Unrealistic deadlines will cause people to take shortcuts, including testing shortcuts, and significant changes to requirements will likely invalidate a large number of tests, requiring them to be rewritten and potentially inflating the implementation overhead.
I'm pretty sure that ATDD has a fantastic payoff for Agile models, or for teams that aren't officially Agile but have very frequent release schedules. I've never tried it on a long-term project, mainly because I've never been in or even heard of an organization willing to try it on that kind of a project, so insert the standard disclaimer here. YMMV and all that.
P.S. In our case, there is no extra effort required from the "customer", but we have a dedicated, full-time Product Owner who actually writes the acceptance criteria. If you're in the "consultingware" business, I suspect it could be a lot more difficult to get the end users to write useful acceptance criteria. A Product Owner/Product Manager seems like a pretty essential element in order to do ATDD and although I can once again only speak from my own experience, I've never heard of ATDD being successfully practiced without someone to fulfill that role.
