Wow, so many questions!
Let's take a step back for a moment.
Why do we write tests? Any tests, of any kind?
The team spends time writing tests in order to save time.
Suppose a bug report is filed, and we need to diagnose
whether the defect is in module A, B, or C? (I'm being simplistic,
sometimes there's greater subtlety to it than that.)
If A & B have such good unit test coverage that we
feel confident in their correctness, we can triage
the issue by focusing our efforts on how C might
interact with the buggy behavior.
Suppose a change to existing feature or addition of new feature
motivates refactoring our code to accommodate recently discovered needs.
How do I know my
refactor
didn't break existing functionality?
Test coverage gives us the courage to wade in, make changes,
and still be confident the old stuff will keep on ticking.
There's a thread that runs through these:
coverage.
You should measure your current test coverage,
and let that play a factor in deciding which tests to write next.
Ok, having reflected for a moment on what we hope
to get out of them, let's look at the process of writing tests.
If you're thinking about it before writing a new feature,
I claim you can "get your tests for free!".
That is, you know you're going to have to execute
that line of target code you wrote for the new feature.
You could do it interactively.
Or you could write an automated test which exercises
the target code and verifies some sensible result.
This initially looks a lot like traditional "printf debugging",
but once the target code is doing the Right Thing
the test can lock in the observed result and you
get your test "for free".
Months down the road future maintenance engineers
will thank you, when they have to refactor that code.
Back to measuring coverage.
Not all target code in your repo was written in that way.
So now you want to retrofit tests on top of it. Start with a
measurement,
e.g. $ pytest --cov *.py.
Or use poor man's profiling: set a debugger breakpoint
for the method of interest, run tests, and either
we hit the breakpoint or we don't.
If not, write a covering test,
and verify the breakpoint triggers.
But wait, you say.
How do I know whether to write a unit test or
an integration test?
It doesn't matter.
Just arrange for an automated test to exercise
the source line of interest.
If the API offers convenient access to
the guts of the code, letting you call
a tiny helper directly, then great!
You have a tiny unit test.
If the API makes it "hard" to do that,
you might wind up with a big integration
test which takes seconds to run and
which pushes several frames onto the stack
by the time it hits that line of code.
Sigh!
You have an integration test.
Score a win anyway.
Then write a ticket to change the target code's
Public API so it is easier to directly test,
and prioritize it in a subsequent sprint.
If you can cover the Happy Path for much of the
code you care about, you're probably ahead of the game.
Folks can spend a lot of time trying to move
from 90% coverage to 95% coverage,
exercising all those error handlers.
Usually you will see diminishing returns from such effort.
The important lesson to learn is that the Public API
should be designed with all consumers in mind,
both target code functions and test code functions.
You might be able to refactor, to improve testability.
More likely you will learn what traps to avoid
next time when you're implementing some new feature
that we might possibly wish to test.
- Is it a good strategy ...
I claim the most important thing to do,
for a codebase with near zero coverage,
is to write a couple of integration tests
that exercise a fair bit of target code,
even if stack depth gets a bit deep.
Any exercised line at least has an opportunity
to throw fatal error in CI/CD, so if someone
broke it by always dividing by zero we will
know about it quickly.
Once your code is largely covered,
you can come back and write more narrowly focused unit tests.
Often you will need
DI
and similar refactors, changes to your Public API,
to make the target code unit testable.
Pay attention to where most bug reports are coming from,
and let that prioritize your refactors.
- How do you approach writing integration tests?
Write a Happy Path test for some input data
that a stakeholder cares about.
If you have more time, maybe try to exercise error handlers.
But often that effort will be better spent writing
narrowly focused unit tests that exercise error handlers.
Integration tests can be written to verify documented
failure modes, e.g. deliberately supplying wrong password
and verifying the expected "login denied, retry"
user flow occurs.
But typically the effort spent writing and monitoring
an integration test is better spent if it tests a single
item in a requirements document.
Think of it as a small slice of an E2E system test.
(And in the end you always a manual tester to occasionally
follow a written sequence of test steps, to verify
that Selenium or other automated test technology
isn't hiding something that prevents users from
accomplishing their goals. For example, a CSS "whoops!"
that gives tiny font or low contrast text could be
enough render a component unusable, despite the DOM
having the "same" rendered objects it had last month
when users were happy.)
If you want to exhaustively verify that error handlers
function properly, and that all edge case inputs are
dealt with, an integration test is not the way to go.
Prefer narrowly focused unit tests for that.
If your app wasn't written with that in mind, you might need
DI
refactors or other (breaking!) changes to the Public API
you designed, so it can accommodate the "one more" requirement
of being readily testable.
