TDD negative experience

Question

What is a negative side of your TDD experience? Do you find baby steps (the simplest fix to make test green) annoying and useless? Do you find no-value tests (when test has sense initially but in final implementation checks the same logic as other test) maintanence critical? etc.

The questions above are about things which I am uncomfortable with during my TDD experience. So I am interested whether other developers have similar feelings and what do they think about them.

Would be thankful for the links to articles describing negative sides of TDD (Google is fullfilled by positive and often fanatic articles).

Answer 1

Like everything that comes under the "Agile" banner, TDD is something that sounds good in theory, but in practice it's not so clear how good it is (and also like most "Agile" things, you are told that if you don't like it, you are doing it wrong).

The definition of TDD is not etched in stone: guys like Kent Beck demand that a non-compiling test must be written before a single line of code and every single line of code should be written to pass a failing test. Up front design is minimal and everything is driven by the tests. It just doesn't work. I've seen a big enterprise app developed using that methodology and I hope that it is the worse code I see in my career (it won't be far off; and that was despite having some talented developers working on it). From what I've seen it results in a huge number of poorly thought out tests that mainly validate that function calls occur, that exceptions are thrown when variables are null and the mocking framework gets a thorough workout (whoop-de-whoop); your production code gets heavily coupled to these tests and the dream of constant and easy refactoring does not appear - in fact people are even less likely to fix bad code because of all the test it will break. In this kind of environment software managers would rather have bad software with passing tests and high code coverage than good software with less tests.

Conversely I've heard people argue that TDD means designing the tests up front on a high level as part of the planning phase - alongside the architectural design. These tests may change during development as more information becomes available, but they have been carefully considered and offer a good guide as to what the code should actually do. To me that makes perfect sense.

Answer 2

This (Clojure author) Rich Hickey interview contains the following. I feel 100 % sympathetic:

Life is short and there are only a finite number of hours in a day. So, we have to make choices about how we spend our time. If we spend it writing tests, that is time we are not spending doing something else. Each of us needs to assess how best to spend our time in order to maximize our results, both in quantity and quality. If people think that spending fifty percent of their time writing tests maximizes their results—okay for them. I’m sure that’s not true for me—I’d rather spend that time thinking about my problem. I’m certain that, for me, this produces better solutions, with fewer defects, than any other use of my time. A bad design with a complete test suite is still a bad design.

Another similar statement from Donald Knuth in Coders at Work book interview, copy-pasted from here:

Seibel: Speaking of practical work, in the middle of working on The Art of Computer Programming you took what turned into a ten-year break to write your typesetting system TeX. I understand you wrote the first version of TeX completely away from the computer.

Knuth: When I wrote TeX originally in 1977 and ’78, of course I didn’t have literate programming but I did have structured programming. I wrote it in a big notebook in longhand, in pencil. Six months later, after I had gone through the whole project, I started typing into the computer. And did the debugging in March of ’78 while I had started writing the program in October of ’77. The code for that is in the Stanford archives—it’s all in pencil—and of course I would come back and change a subroutine as I learned what it should be. This was a first-generation system, so lots of different architectures were possible and had to be discarded until I’d lived with it for a while and knew what was there. And it was a chicken-and-egg problem—you couldn’t typeset until you had fonts but then you couldn’t have fonts until you could typeset. But structured programming gave me the idea of invariants and knowing how to make black boxes that I could understand. So I had the confidence that the code would work when I finally would debug it. I felt that I would be saving a lot of time if I waited six months before testing anything. I had enough confidence that the code was approximately right.

Seibel: And the time savings would be because you wouldn’t spend time building scaffolding and stubs to test incomplete code?

Knuth: Right.

Answer 3

My negative experience with TDD was my first one. TDD sounded great to me, I'd done QA for years and still had the horrors fresh in my mind. I wanted to squash every bug before it made it into a build. Unfortunately, using TDD doesn't guarantee that you'll write good tests. In fact, my initial predisposition was to write simple tests that generated simple code. Really simple code that contained few abstractions. Really simple tests that were intertwined with the class internals. And once you have a few thousand itty bitty tests in place you sure don't feel like you're moving any faster when you have to change a hundred of them to refactor your code to use very important domain concept X.

The light came on for me - TDD is not a testing skill. It's a design skill. It can only lead you to good, simple, workable code with practice and a constant awareness of the design directions it leads you in. If you're writing tests for the sake of code coverage you're going to create brittle tests. If you're writing tests to help you design your abstractions then it's just a more rigorous way of writing top-down code. You get to see the code from the caller's perspective first, which encourages you to make his life easier, rather than mirroring a class's internals to its outside edge.

I think TDD is useful, but I'm not dogmatic about it. If those "no-value tests" are making maintenance difficult - Delete them! I treat tests the same way as the rest of the code. If it can be refactored away and make things simpler, then do it!

I haven't seen it personally, but I've heard that some places track code coverage and test counts. So if metrics gathering is a side effect of TDD, then I could see that as a negative as well. I will enthusiastically delete 1000 lines of code, and if that obsoletes 20 tests and drops my code coverage %, oh well.

Answer 4

I'm going to go out on a limb here and declare with brutal honesty that it's literally a ritualistic waste of time. (In the majority of situations.)

I bought a book on Unit Testing which also discussed TDD, and while I agree with the benefits of UT, after about a hundred hours of trying out TDD, I gave up on it for a myriad of reasons. I'm kind of cross-posting here, but TDD:

1. Isn't better documentation than real documentation.
2. Doesn't catch bugs or regressions.
3. Doesn't really make my designs better than they end up being if I apply some functional programming and composability concepts.
4. Is time that could be better spent doing code reviews or polishing documentation and specifications.
5. Gives managers a false sense of security when they see a list of hundreds of green icons.
6. Improves productivity in the implementation of algorithms with limited input-output mappings.
7. Is clumsy in that you may know what you're doing as a result of TDD, but you aren't earning any understanding of why it works so well, why your designs come out the way they do.

Another concern is the debated degree of perfection to which one must do TDD to do it successfully. Some insist that if TDD isn't done persistently by everyone on the team from the beginning of the project, you'll only suffer. Others insist that no one ever does TDD by the book. If these are both true, then it follows that TDD practitioners are suffering, whether they realize it or not.

Of course, if it's being argued that by doing things in a TDD-like manner you'll come to designs that can work easily with TDD, well, there are much quicker ways to achieve that -- namely, by actually studying the concepts of composability. There's plenty of resources out there, a lot of rigorous mathematical theory even (largely in functional programming but also in other fields). Why not spend all your TDD time learning?

Culturally, TDD shows symptoms of being a ritualistic practice. It rides on guilt; it encourages procedure over understanding; it has tons of doctrines and slogans ("fake it 'til you make it" is really quite alarming if you look at it objectively). Wikipedia's definition of the word "ritual" is in fact quite apposite:

In psychology, the term ritual is sometimes used in a technical sense for a repetitive behavior systematically used by a person to neutralize or prevent anxiety; it is a symptom of obsessive–compulsive disorder.

Answer 5

To add, another concern with TDD I noticed is:

TDD causes an inadvertent shift in development team's focus from Quality Code to Testcases and Code Coverage! I personally did not like TDD as it makes me less creative and makes software development a dull mechanical process! Unit testcases are useful when judiciously used but becomes a burden when treated the goal of software development.

I know a guy who is a manager and is technically dull once got obsessed with TDD. It was such a magical thing for him that he believed would bring in magical solutions to all the issues in his poorly architectured software with least maintainable code. Not to say what happened to that project-it failed miserably in his hands, while all his testcases were green. I guess TDD helped him get some kind of statistical information like "99/ 100 of my cases are green" etc and that was the reason for his obsession as he would never be able to evaluate the quality or suggest enhancements in design.

Answer 6

My primary negative experience is trying to use TDD to edit another programmer's code who has no tests, or very, very basic integration tests. When I go to add a feature, or fix a problem with said code; I would prefer to write a test first (the TDD way). Unfortunately, the code is tightly coupled, and I cannot test anything without a lot of refactoring.

The refactoring is a great exercise anyway, but it is required to get the code into a testable state. And after this step, I have no checks and balances to see if my changes broke anything; short of running the application and examining every use case.

On the other hand, adding features/fixing bugs to a TDD project becomes very straightforward. By nature, code written with TDD is usually quite decoupled with small pieces to work with.

In any case, TDD is a guideline. Follow it to the point where you find you get maximum effectiveness. Decent test coverage and decoupled code, well written code.

Answer 7

I made the experience that I sometimes rely on my tests too much when it comes to the design of the system. I am basically too low in the nitty-gritty implementation details to take the step back to look at the bigger picture. This often results in an unnecessarily complex design. I know, I am supposed to refactor the code but sometimes I have the impression that I could save a lot of time by taking the step back more often.

That being said, if you have a framework like rails where your architectural decisions are very limited these problems are basically non existent.

Another problem is when you trust your tests blindly. Truth is - like any other code - your tests can have bugs too. So be as critical towards your tests as you are towards your implementation.

Answer 8

As a big fan of TDD I sometimes see these drawbacks

• Temptation to write too many test for the sake of nearly 100% code coverage. In my opinion it is not necessary to write tests
  • for simple property getters/setters
  • for every case when an exception is thrown
  • that checks the same functionality through different layers. (Example: if you have a unittest to check input validation for every parameter then it is not necessary to repeat all these tests through an integration test, too)

• Maintenance costs of test code for similar tests, that vary only slightly (created through code-duplication (aka copy-paste-inheritance)). If you already have one it is easy to create a similar one. But if you do not refactor the test code, by eliminating code duplication into helper-methods you might need some time to fix the tests if implementation details of you business-code changes.

• If you are under time pressure you may be tempted to eliminate broken tests (or comment them out) instead of fixing them. This way you lose the investment in the tests

Answer 9

TDD zealots.

For me, they are just one of a long line of religious loonies knocking on my door, trying to prove that my way of doing things is irreparably broken and the only path to salvation is Jesus, Kent Back, or Unit Testing.

IMO, their biggest lie is that TDD will lead you to salvation a better algorithm design. See the famous Soduku solver written in TDD: here, here, here, here and here

And compare it Peter Norvig sudoku solver done not by using TDD, but using old-fashioned engineering: http://norvig.com/sudoku.html

Answer 10

I have yet to come across more than one scenario as a game developer where TDD was worthwhile. And the instance in which it was, was a piece of code which was purely mathematical in nature and needed a solid approach to test a huge number of edge cases simultaneously - a rare need.

Maybe something, someday will change my mind, but amongst XP practices, the idea of refactoring mercilessly, and of code evolving its own form are far more important and lead to the greatest productivity for me, cf. a quote from a paper by James Newkirk:

Simplicity - "What is the simplest thing that could possibly work?"
The message is very clear. Given the requirements for today, design and write your software. Do not try to anticipate the future, let the future unfold. It will often do this in very unpredictable ways making anticipation a cost that is often too expensive to afford."

The concepts of courage and of tightening feedback loops which he mentions are also, to my mind, key to productivity.

Answer 11

My negative TDD experience, limited as it may be, is simply knowing where to start! For instance, I'll try to do something TDD and either have no idea where to begin barring testing trivial things (can I new up a Foo object, can I pass in a Quux to the Baz, and the like. Tests that don't test anything), or if I am trying to implement it in an existing project then I find that I would have to rewrite various classes to be able to use them in TDD. The end result is that I quickly abandon the notion entirely.

It probably doesn't help that often I'm the only person in the entire company who knows what unit testing (TDD or otherwise) is and why it's a good thing.

Answer 12

If you use TDD from these "fanatic" articles you will get wrong safety feeling that your software have no errors.

Answer 13

TDD have some benefits:

• You focus on how to call your code and what to expect first (as you write the test first) instead of focusing on solving the problem and then you glue in a call from the application. The modularity makes it easier to mock and wrap.
• The tests ensure that your program works the same before and after a refactoring. This does not mean your program is error free, but that it keeps working the same way.

TDD is about long term investment. The effort pays off when you reach maintenance mode of your application, and if the application is not planned to reach that point you may never recover the investment.

I consider the TDD red-green-cycle with the baby steps similar to a checklist for a plane. It is annoying and tedious to check each and every thing in the plane before take-off especially if it is trivially simple (the TDD baby steps) but it has been found that it increases safety. Besides verifying everything works, it essentially resets the plane. In other words, a plane is rebooted before every take-off.

Answer 14

My negative experience about TDD is something I feel with a lot of new and hyped stuff. In fact I enjoy TDD because it ensures the validity of my code, and even more important: I can recognize failing tests, after adding new code or any kind of refactoring.

What annoys me about TDD is the fact that there are a lot of rules or guidelines about it. As it is still quite new, we most of us experience to be beginners at TDD. So what works well for some of us, might not work for others. What I want to say is, that there is no real "wrong or right" way to perform TDD: There is the way that works for me - and my team if I have one.

So as long as you write tests - before or after the production code doesn't really matter IMHO - I am not sure if test driven really means you have to follow all guidelines that are stated right now, since they are not yet proven to be the ideal solution for every day work. If you find a better way in writing tests, you should post it in a blog, discuss it here, or write an article about it. So in ten years or so we might have shared enough experience to be able to tell which rule of TDD can be assumed to be good or not in a certain situation.

Answer 15

I've found TDD performs poorly when it comes to emergent systems. I'm a video games developer, and recently used TDD to create an system that uses multiple simple behaviours to create realistic-looking movement for an entity.

For instance, there are behaviours responsible for moving you away from dangerous areas of different types, and ones responsible for moving you towards interesting areas of different types. Amalgamating the output of each behaviour creates a final movement.

The guts of the system were implemented easily, and TDD was useful here for specifying what each subsystem should be responsible for.

However I ran into problems when it came to specifying how the behaviours interact, and more importantly how they interact over time. Often there was no right answer, and although my initial tests were passing, QA could keep finding edge cases where the system didn't work. To find the correct solution I had to iterate thorugh several different behaviours, and if I updated the tests each time to reflect the new behaviours before I checked they worked in-game, I may have ended up throwing out the tests time and time again. So I deleted those tests.

I should have possibly had stronger tests that captured the edge cases QA discovered, but when you have a system like this that sits on top of many physics and gameplay systems, and you're dealing with behaviours over time, it becomes a bit of a nightmare to specify exactly what's happening.

I almost certainly made mistakes in my approach, and like I said for the guts of the system TDD worked brilliantly, and even supported a few optimising refactors.

Answer 16

I have, on more than one occasion, written code that I discarded the next day since it was clumsy. I restarted with TDD and the solution was better. So I have not had too much in the line of negative TDD experience. However, that being said, I have spent time thinking about a problem and coming up with a better solution outside of the TDD space.