What should you test with unit tests?

Question

I'm freshly out of college, and starting university somewhere next week. We've seen unit tests, but we kinda not used them much; and everyone talks about them, so I figured maybe I should do some.

The problem is, I don't know what to test. Should I test the common case? The edge case? How do I know that a function is adequately covered?

I always have the terrible feeling that while a test will prove that a function works for a certain case, it's utterly useless to prove that the function works, period.

Answer 1

My personal philosophy has thusfar been:

1. Test the common case of everything you can. This will tell you when that code breaks after you make some change (which is, in my opinion, the single greatest benefit of automated unit testing).
2. Test the edge cases of a few unusually complex code that you think will probably have errors.
3. Whenever you find a bug, write a test case to cover it before fixing it
4. Add edge-case tests to less critical code whenever someone has time to kill.

Answer 2

Among the plethora of answers thusfar no one has touched upon equivalence partitioning and boundary value analysis, vital considerations in the answer to the question at hand. All of the other answers, while useful, are qualitative but it is possible--and preferable--to be quantitative here. @fishtoaster provides some concrete guidelines, just peeking under the covers of test quantification, but equivalence partitioning and boundary value analysis allow us to do better.

In equivalence partitioning, you divide the set of all possible inputs into groups based on expected outcomes. Any input from one group will yield equivalent results, thus such groups are called equivalence classes. (Note that equivalent results does not mean identical results.)

As a simple example consider a program that should transform lowercase ASCII characters to uppercase characters. Other characters should undergo an identity transformation, i.e. remain unchanged. Here is one possible breakdown into equivalence classes:

| # |  Equivalence class    | Input        | Output       | # test cases |
+------------------------------------------------------------------------+
| 1 | Lowercase letter      | a - z        | A - Z        | 26           |
| 2 | Uppercase letter      | A - Z        | A - Z        | 26           |
| 3 | Non-alphabetic chars  | 0-9!@#,/"... | 0-9!@#,/"... | 42           |
| 4 | Non-printable chars   | ^C,^S,TAB... | ^C,^S,TAB... | 34           |

The last column reports the number of test cases if you enumerate all of them. Technically, by @fishtoaster's rule 1 you would include 52 test cases--all of those for the first two rows given above fall under the "common case". @fishtoaster's rule 2 would add some or all from rows 3 and 4 above as well. But with equivalence partitioning testing any one test case in each equivalence class is sufficient. If you pick "a" or "g" or "w" you are testing the same code path. Thus, you have a total of 4 test cases instead of 52+.

Boundary value analysis recommends a slight refinement: essentially it suggests that not every member of an equivalence class is, well, equivalent. That is, values at boundaries should also be considered worthy of a test case in their own right. (One easy justification for this is the infamous off-by-one error!) Thus, for each equivalence class you could have 3 test inputs. Looking at the input domain above--and with some knowledge of ASCII values--I might come up with these test case inputs:

| # | Input                | # test cases |
| 1 | a, w, z              | 3            |
| 2 | A, E, Z              | 3            |
| 3 | 0, 5, 9, !, @, *, ~  | 7            |
| 4 | nul, esc, space, del | 4            |

(As soon as you get more than 3 boundary values that suggests you might want to rethink your original equivalence class delineations, but this was simple enough that I did not go back to revise them.) Thus, boundary value analysis brings us up to just 17 test cases -- with a high confidence of complete coverage -- compared to 128 test cases to do exhaustive testing. (Not to mention that combinatorics dictate that exhaustive testing is simply infeasible for any real-world application!)

Answer 3

Probably my opinion is not too popular. But I suggest that you to be economical with unit tests. If you have too many unit tests you can easily end up spending half of your time or more with maintaining tests rather than actual coding.

I suggest you to write tests for things you have a bad feeling in your gut or things that are very crucial and/or elementary. IMHO unit tests are not a replacement for good engineering and defensive coding. Currently I work on a project that is more or less unusuable. It is really stable but a pain to refactor. In fact nobody has touched this code in one year and the software stack it's based on is 4 years old. Why? Because it's cluttered with unit tests, to be precise: Unit tests and automatized integration tests. (Ever heard of cucumber and the like?) And here is the best part: This (yet) unusuable piece of software has been developed by a company whose employees are pioneers in the test-driven development scene. :D

So my suggestion is:

• Start writing tests after you developed the basic skeleton, otherwise refactoring can be painful. As a developer who develops for others you never get the requirements right at the start.

• Make sure your unit tests can be performed quickly. If you have integration tests (like cucumber) it's ok if they take a bit longer. But long running tests are no fun, believe me. (People forget all reasons why C++ has become less popular...)

• Leave this TDD stuff to the TDD-experts.

• And yes, sometimes you concentrate on the edge cases, sometimes on the common cases, depending where you expect the unexpected. Though if you always expect the unexpected, you should really rethink you workflow and discipline. ;-)

Answer 4

If you start following Test Driven Development practices, they will sort of guide you through the process, and knowing what to test will come naturally. Some places to start:

Tests come first

Never, ever write code before writing the tests. See Red-Green-Refactor-Repeat for an explanation.

Write regression tests

Whenever you encounter a bug, write a testcase, and make sure it fails. Unless you can reproduce a bug through a failing testcase, you haven't really found it.

Red-Green-Refactor-Repeat

Red: Start by writing a most basic test for the behavior that you are trying to implement. Think of this step as writing some example code that uses the class or function that you are working on. Make sure it compiles/has no syntax errors and that it fails. This should be obvious: you haven't written any code, so it must fail, right? The important thing to learn here is that unless you see the test fail at least once, you can never be sure that if it passes, it does it because of something that you've done, not because of some bogus reason.

Green: Write the most simple and stupid code that actually makes the test pass. Don't try to be smart. Even if you see that there's an obvious edge case but the test take in account, don't write code to handle it (but don't forget about the edge case: you'll need it later). The idea is that every piece of code you write, every if, every try: ... except: ... should be justified by a test case. The code doesn't have to be elegant, fast or optimized. You just want the test to pass.

Refactor: Clean up your code, get the method names right. See if the test is still passing. Optimize. Run the test again.

Repeat: You remember the edge case that the test didn't cover, right? So, now it's its big moment. Write a testcase that covers that situation, watch it fail, write some code, see it pass, refactor.

Test your code

You are working on some specific piece of code, and this is exactly what you want to test. This means that you should not be testing library functions, the standard library or your compiler. Also, try to avoid testing the "world". This includes: calling external web APIs, some database intensive stuff, etc. Whenever you can, try to mock it up (make an object that follows the same interface, but returns static, predefined data).

Answer 5

If you are testing first with Test Driven Development, then your coverage is going to be up in the 90% range or higher, because you won't be adding functionality without first writing a failing unit test for it.

If you are adding tests after the fact, then I cannot recommend enough that you get a copy of Working Effectively With Legacy Code by Michael Feathers and take a look at some of the techniques for both adding tests to your code and ways of refactoring your code to make it more testable.

Answer 6

For unit tests, start with testing that it does what it is designed to do. That should be the very first case you write. If part of the design is "it should throw an exception if you pass in junk", test that too since that is part of the design.

Start with that. As you get experience with doing that most basic of testing you'll begin to learn whether or not that is sufficient, and start to see other aspects of your code that need testing.

Answer 7

The stock answer is to "test everything that could possibly break".

What's too simple to break? Data fields, brain-dead property accessors, and similar boilerplate overhead. Anything else probably implements some identifiable portion of a requirement, and may benefit from being tested.

Of course, your mileage -- and your working environment's practices -- may vary.