TDD vs. Productivity

Question

In my current project (a game, in C++), I decided that I would use Test Driven Development 100% during development.

In terms of code quality, this has been great. My code has never been so well designed or so bug-free. I don't cringe when viewing code I wrote a year ago at the start of the project, and I have gained a much better sense for how to structure things, not only to be more easily testable, but to be simpler to implement and use.

However... it has been a year since I started the project. Granted, I can only work on it in my spare time, but TDD is still slowing me down considerably compared to what I'm used to. I read that the slower development speed gets better over time, and I definitely do think up tests a lot more easily than I used to, but I've been at it for a year now and I'm still working at a snail's pace.

Each time I think about the next step that needs work, I have to stop every time and think about how I would write a test for it, to allow me to write the actual code. I'll sometimes get stuck for hours, knowing exactly what code I want to write, but not knowing how to break it down finely enough to fully cover it with tests. Other times, I'll quickly think up a dozen tests, and spend an hour writing tests to cover a tiny piece of real code that would have otherwise taken a few minutes to write.

Or, after finishing the 50th test to cover a particular entity in the game and all aspects of it's creation and usage, I look at my to-do list and see the next entity to be coded, and cringe in horror at the thought of writing another 50 similar tests to get it implemented.

It's gotten to the point that, looking over the progress of the last year, I'm considering abandoning TDD for the sake of "getting the damn project finished". However, giving up the code quality that came with it is not something I'm looking forward to. I'm afraid that if I stop writing tests, then I'll slip out of the habit of making the code so modular and testable.

Am I perhaps doing something wrong to still be so slow at this? Are there alternatives that speed up productivity without completely losing the benefits? TAD? Less test coverage? How do other people survive TDD without killing all productivity and motivation?

Answer 1

Let me begin by thanking you to share your experience and voicing out your concerns... which I have to say are not uncommon.

• Time/Productivity: Writing tests is slower than not writing tests. If you scope it to that, I'd agree. However if you run a parallel effort where you apply a non-TDD approach, chances are that the time you spend break-detect-debug-and-fix existing code will put you in the net negative. For me, TDD is the fastest I can go without compromising on my code-confidence. If you find things in your method, that are not adding value, eliminate them.
• Number of tests: If you code up N things, you need to test N things. to paraphrase one of Kent Beck's lines "Test only if you would want it to work."
• Getting stuck for hours: I do too (sometimes and not > 20 mins before I stop the line).. It's just your code telling you that the design needs some work. A test is just another client for your SUT class. If a test is finding it difficult to use your type, chances are so will your production clients.
• Similar tests tedium : This needs some more context for me to write up a counterargument. That said, Stop and think about the similarity. Can you data-drive those tests somehow? Is it possible to write tests against a base-type? Then you just need to run the same set of tests against each derivation. Listen to your tests. Be the right kind of lazy and see if you can figure out a way to avoid tedium.
• Stopping to think about what you need to do next (the test/spec) isn't a bad thing. On the contrary, it's recommended so that you build "the right thing". Usually if I can't think of how to test it, I usually can't think of the implementation either. It's a good idea to blank out implementation ideas till you get there.. maybe a simpler solution is overshadowed by a YAGNI-ish pre-emptive design.

And that brings me to the final query : How do I get better ? My (or An) answer is Read, Reflect and Practice.

e.g. Of late, I keep tabs on

• whether my rhythm reflects RG[Ref]RG[Ref]RG[Ref] or is it RRRRGRRef.
• % time spent in the Red / Compile Error state.
• Am I stuck in a Red/Broken builds state?

Answer 2

You don't need 100% percent test coverage. Be pragmatic.

Answer 3

TDD is still slowing me down considerably

That's actually false.

Without TDD, you spend a few weeks writing code which mostly works and spend the next year "testing" and fixing many (but not all) of the bugs.

With TDD, you spend a year writing code which actually works. Then you do final integration testing for a few weeks.

The elapsed time is probably going to be the same. The TDD software will be substantially better quality.

Answer 4

Or, after finishing the 50th test to cover a particular entity in the game and all aspects of it's creation and usage, I look at my to-do list and see the next entity to be coded, and cringe in horror at the thought of writing another 50 similar tests to get it implemented.

This makes me wonder how much you're following the "Refactor" step of TDD.

When all your tests are passing, this is time for you to refactor the code and remove duplication. While people usually remember this, sometimes they forget that this is also time to refactor their tests as well, to remove duplication and simplify things.

If you have two entities that merge into one to enable code re-use, consider merging their tests as well. You really only need to test incremental differences in your code. If you're not regularly performing maintenance on your tests, they can quickly become unwieldy.

A couple of philosophical points about TDD that might be helpful:

• When you can't figure out how to write a test, despite extensive experience writing tests, then that's definitely a code smell. Your code is somehow lacking in modularity, which makes writing small, simple tests difficult.
• Spiking out a bit of code is perfectly acceptable when using TDD. Write the code you want to, to get an idea of what it looks like, then delete the code and start with tests.
• I see practicing extremely strict TDD as a form of exercise. When you first start out, definitely write a test first every single time, and write the simplest code to make the test pass before moving on. However, this is not necessary once you become more comfortable with the practice. I don't have a unit test for every single possible code path I write, but through experience I'm able to choose what needs to be tested with a unit test, and what can be covered by functional or integration testing instead. If you've been practicing TDD in a strict fashion for a year, I would imagine you're close to this point as well.

EDIT: On the topic of philosophy of unit testing, I think this might be interesting for you to read: The Way of Testivus

And a more practical, if not necessarily very helpful, point:

• You mention C++ as your development language. I've practiced TDD extensively in Java, using excellent libraries like JUnit and Mockito. However, I've found TDD in C++ to be very frustrating due to the lack of libraries (in particular, mocking frameworks) available. While this point doesn't help you in your current situation much, I hope you take it into consideration before ditching TDD altogether.

Answer 5

Very interesting question.

What's important to note, is that C++ is not very easily testable, and gaming, in general, is also a very bad candidate for TDD. You can't test if OpenGL/DirectX draws triangle red with driver X, and yellow with driver Y easily. If the bump map normal vector is not flipped after shader transforms. Nor you can test clipping issues over driver versions with different precisions, and so on. Undefined drawing behavior because of incorrect calls can also be tested only with accurate code review and SDK at hand. Sound is also a bad candidate. Multithreading, which again is quite important for games, is pretty much useless to unit-test. So it's tough.

Basically gaming is a lot of GUI, sound and threads. GUI, even with standard components you can send WM_ to, is tough to unit-test.

So what you can test, is model loading classes, texture loading classes, matrix libraries and some-such, which is not a lot of code, and, quite often, not very reusable, if it's only your first project. Also, they are packed into proprietary formats, so it's not quite likely that 3rd party input can differ much, unless you release modding tools etc.

Then again, I'm not TDD guru or evangelist, so take all that with a grain of salt.

I would probably write some tests for main core components (for example matrix library, image library). Add a bunch of abort() on unexpected inputs in every function. And most importantly, concentrate on resistant/resilient code that doesn't break easily.

Concerning off by one errors, clever use of C++, RAII and a good design goes a long way to prevent them.

Basically you have a lot to do just to cover the basics if you want to release the game. I'm not sure if TDD will help.

Answer 6

I agree with the other answers, but I also want to add a very important point: Refactoring costs!!

With well written unit tests, you can safely do re-writes of your code. First, well written unit tests provide excellent documentation of the intent of your code. Second, any unfortunate side effects of your refactoring will be caught in the existing test suite. Thus, you have guaranteed that the assumptions of your old code are true for your new code as well.

Answer 7

How do other people survive TDD without killing all productivity and motivation?

This is completely different from my experiences. You are either amazingly intelligent and write code without bugs, (eg. off by one errors) or you don't realize your code has bugs which prevent your program working, and so aren't actually finished.

TDD is about having the humility to know that you (and me!) make mistakes.

For me the time writing unittests is more than saved in reduced debugging time for projects that are done using TDD from the beginning.

If you don't make mistakes then maybe TDD isn't as important for you as it is for me!

Answer 8

I've only a few remarks:

1. It seems you're trying to test everything. You probably shouldn't, just the high-risk and edge cases of a particular piece of code / method. I'm pretty sure the 80 / 20 rule applies here: You spend 80% writing tests for the last 20% of your code or cases that aren't covered.

2. Prioritize. Get into agile software development, and make a list of what you really really really need to do to release in one month time. Then release, just like that. This'll make you think about the priority of features. Yeah, it'd be cool if your character could do a backflip, but does it have business value?

TDD is good, but only if you don't aim for 100% test coverage, and it's not if it keeps you from producing actual business value (i.e. features, things that add something to your game).

Answer 9

If good practice would take less upfront effort than bad practice, everyone would already be doing it without needing to enforce it.

It is the very nature of good practice that it includes additional upfront effort. It pays back dividends in the long run, but without clear enforcement developers are liable to use shortcuts that net them quick wins but long term drains on maintainability.

That being said, I don't like dogma either. Pragmatism is very much relevant here, but it is sometimes difficult to find that pragmatic balance. There are two dogmas here that I would advise not following:

• 100% code coverage
• Tests musts be written before the implementation

Instead, I would pragmatically suggest that:

• Ensure code coverage for the non-trivial important bits
• Don't implement something without also backing it by tests - but the order of these two steps is up to you.

This will ease the tension that purism tends to create. Note that both of these are phrased with some subjectivity baked in. What is considered important? Should you write tests first or second?

Both are reasonable topics that experts can have differing opinions on, and I'm not going to tell you I know your codebase better than you. I would argue that if you stay within the lines of the advice that I set out here, that you're not going to run foul of TDD's spirit, if not its dogma.

Answer 10

Yes, writing tests and code might take longer than just writing code - but writing code and associated unit tests (using TDD) is much more predictable than writing code and then debugging it.

Debugging is almost eliminated when using TDD - which makes all development process much more predictable and in the end - arguably faster.

Constant refactoring - it is impossible to do any serious refactoring without comprehensive unit test suite. The most efficient way to build that unit testing based safety net is during TDD. Well refactored code significantly improves overall productivity of the designer/team who maintains the code.

Answer 11

Here is a PluralSight article on evidence-base conclusions around TDD.

Some results looked at by the article are summarised below. The columns give the research source, and the rows show the outcome being measured.

• Tick = improves the outcome
• Circle with minus sign = 'inconclusive'
• Cross = worsens the outcome

---

• The controlled experiment shows increased productivity when using TDD
• The pilot study is inconclusive on productivity when using TDD
• Semi-industry research shows a reduction in productivity when using TDD
• Industry research shows a short term reduction in productivity when using TDD, but a gain in the long term

Also see - https://theqalead.com/general/statistics-studies-benefits-test-driven-development/

Answer 12

Consider narrowing the scope of your game and get it where someone can play it or you release it. Maintain your testing standards without having to wait too long to release your game might be a middle-ground to keep you motivated. The feedback from your users may provide benefits in the long-term and your testing allows you to feel comfortable with the additions and changes.