3

In an environment where you can't merge code that fails tests you obviously can't merge new, failing tests.

But how to deal with a newly found bug exploit, meaning you wrote a new test exposing an already existing bug? Test-driven-development would say: Hooray, you have the test, now fix the bug and then commit both.

But what if you a) are a Test engineer who found the exploit but are not able to fix it or b) you would be able to fix it but not right now?

Option One: Commit the test to an extra branch, mention it in the bug issue and essentially leave the code until somebody has mercy and bases their fix upon this branch and commits both.

Option Two: Mark the test somehow as "not nice that it fails but expected", merge it and the Continuous Integration lets it trough. As soon as the bug is fixed, you end the test's special treatment and mark it mandatory. Maybe, the CI would even display the list of expected failures so you have an additional kind of bug tracker.

Are there any common practices / standards out there? Especially for Option Two, do standard test frameworks like JUnit, Google Test,... have any way of modeling it?

Robert Harvey
  • 200,592
PhilLab
  • 159

2 Answers2

5

Option one. Just because you can't merge code doesn't mean you can't check it in. You simply need to create a branch for your failing test, and then it's ready when someone starts to work on it.

That being said, this isn't for us to decide. This is something your team needs to decide for itself. There are no magic bullets that you can adopt that will make this situation easier. Do what works best for your team and your way of working.

Bryan Oakley
  • 25,479
4

Never break the build. But tests that should pass in the future are completely valid. Many test runners understand an “xfail”, “pending”, or “todo” category for this kind of case. If one of those tests (unexpectedly) starts passing you can remove this category. By merging these todo tests, you can make sure that they continue to compile and run, which may be preferable over storing these tests separately.

Examples:

  • In JUnit you can use annotations to mark pending tests, though you might need a separate library. Simply ignoring the test might be insufficient because the test won't be attempted.
  • Google Test can mark tests as disabled and can execute those tests, but interprets a failure as a failure rather than pending. With enough macros, you could roll your own solution.
  • A pending state is completely normal in BDD tools like Cucumber.
  • Pytest has excellent support for an xfail category.
  • Perl's Test::More has TODO blocks.

Whether to use this approach depends a lot on your team's workflow. Todo tests are great when you are implementing a spec or are tackling a backlog of user stories – especially in a BDD context where the test is the specification. And todo tests can be good for highly important problems where the test is valuable documentation.

But for normal tests, it might be better to not bog down the test suite with lots of failures. When writing an issue or bug report, consider including the test code in that issue rather than merging it. Writing a good test is already a large part of debugging and solving the problem, so ideally the problem can be fixed quickly.

amon
  • 135,795