Is the execution time of a unit test useful for anything? Is a unit test an appropriate place for any sort of code profiling? Why or why not?
Sample Use Case:
How could I use this info?
One of the most important rules about unit tests is they should run fast.
Developers should be able to run the whole suite of unit tests in seconds, and definitely not in minutes and minutes. Developers should be able to quickly run them after changing the code in anyway. If it takes too long, they won't bother running them and you lose one of the main benefits of the tests. We currently have about 800 tests that run in around 30 seconds.
So if any tests do start taking too long, you can see which test is taking too long and do something about it.
Our unit test framework tells us how long it took for each test to run. Most of them take 0- 5ms (if I remember correctly), we have one or two that take about 3 seconds.
Is the execution time of a unit test useful for anything?
Yes.
Is a unit test an appropriate place for any sort of code profiling?
Yes.
Why?
Because you need facts and measurements. If a test suite is slow, it's important to know where so you can focus -- separately -- on why.
Code profiling is only done when code is proven to be slow.
Unit tests are an easy, cheap, obvious, painless way to locate (1) slow code and (2) sudden changes in performance.
Consider the development of your non-functional requirements:
Functional Requirement: Once the filters have been selected, the report should be generated. Non-Functional: The report should generate in 2 seconds.
If your test's execution time is > 2 seconds, you could even fail the test as a "usability" test.
It could be useful if you are testing a procedure that is known to be slow. You could write unit tests that test the procedure with different sized inputs and you could monitor the performance of just that piece of code. Of course, whether or not this is applicable in your project really depends on your project. I use when it is easy to determine which function has a bottle-neck, and we want to isolate and test just that function.
I can only think of timing out tests if something like an endless loop could occur in the tested code.
If you run unit tests as a part of post-commit checks or a build, the machine handling it may have all kinds of loads that will make precisely timed tests useless. Run such performance tests (not unit tests) in a controlled environment.
It can be useful to collect stats of test execution times to predict running times of large test suites (integration servers, like TeamCity, do this).
It could be valuable if you were to keep track of average performance for a given test across all test runs. A sudden increase in execution time might signal the introduction of a new bottleneck into the code.
Also in general it is desirable to keep the execution for an entire set of unit tests as fast as possible in order to encourage running the tests frequently. Knowing the execution of individual tests will help identify which tests are taking the most time across all tests.
