Should i write tests if it takes more time than testing manually?

Question

Im making a website and trying to learn Test Driven Development (TDD)

I'm doing one of CS50 projects, were we need to make a website to trade stocks. I decided to use this oportunity to learn TDD. I learned how to use the unittest module from python. But now, im taking way more time to write the tests than actually writting the code.

Since Flask has the debug mode, i dont need to close and open the application to test changes manually, so testing with it is a very fast process. But when im writting tests, i dont think on just what i want, i need to think how to make the test actually test what i want.

For example, i took 30 minutes to write the code to test if an component of the HTML was showing up in the website, because i had to write a Regex that matched it. Something that, by the eye, would have taken less than 30 seconds.

Perhaps in a more complex project it would pay off, but in this case i take less time checking every page of the website (30 sec to 1 minute) than to write a single test.

Im not doing unit tests tho (even if im using the unittest module). Im actually doing integration tests, because i dont know how to do unit testing with websites.

So it is worth it to do testing in this kind of code? Where the code is so simple that the whole development process would take less time than writting the testing code

Answer 1

Lets take this XKCD literally:

You wrote it took you 30 minutes to automate a task which manually takes 30 seconds. Now lets make the conservative assumption that you will run your automated test suite (which includes the test) at least once a day - then you find the break-even point for a five-years return-of-investment at 2 hours of development time. According to this simple model, it seems to be clearly worth it, at least when you expect a life-time of your program of five years or more.

Of course, in reality, things are not that simple. The tests themselves are code which requires maintenance and debugging. Specificially for user interface examples like the ones mentioned in the question such tests can become sometimes brittle and require extra effort whenever one wants to change the slightest things afterwards.

Hence, this is usually a trade-off, and there is no easy answer to this. The "correct" answer will depend on a hell lot of factors like the whole project situation, the quality expectations of your audiences, your own expertise, the expected stability of requirements, the expected life time of the program and more.

You may have a look into Should I bother to write unit test for UI/UX Components?, which may contain at least some of the answers you might be looking for. Another good Q&A for balancing time for unit tests vs. manual testing is this one: how much time do you spend on Unit testing?

Answer 2

Testing once by hand is always more affordable than building a unit test. However, once the unit test is written, it has the potential to be a useful product. The testing done by hand is always nothing more than a completed document.

Testing is something that benefits from thinking in a holistic view. In particular, what is your process for releasing a website? What requirements are leveraged on it, and how do you prove things are correct?

If a system is particularly discrete, with very few interconnects, the V&V process may be nothing more than "I did the test by hand once, here's the document, and nothing has changed in that component since then." And that can indeed be far cheaper than unit testing. However, as you get more and more interconnects, it becomes harder to confirm that everything is working. I can't be the first person to accidentally copy/paste an if over an else if in the middle of a logic block and change the behavior of something that I wasn't intentionally messing with.

The unit tests are a tool to guard against that kind of failure. How much you should develop those tools is up to you, and Doc Brown answers that one quite nicely

Answer 3

If you think its quicker to do a manual test than an automated test you aren't testing enough.

Lets take your 30min to write test vs 30 sec to manually test example.

EVERY time you make any change to the code you potentially break the feature you are testing and MUST re run this test.

• Did you make more than 60 code changes?
• Do you have more than one test to perform?

Also, did it really take 30 sec to manually check?

• Did you record the test result?
• Are you sure you performed the test correctly on the correct code base?
• Did you get two or more people to run the test to check its being done right?
• Have you documented the manual test well enough that another person can run it manually if they make a change 3 years from now?
• How many browsers did you test with?
• etc

If you are serious about making your code bug free you have to be serious about testing, and if you are serious about testing then automation gives you massive massive time savings over manual.

Answer 4

Write tests for those that will read them. In the language they know. Testers who think like users need tests written in English (or whatever their native language is). Testers who code need tests written in code (in whatever their native language is).

If you want automated tests that user focused testers understand you need to spend some time ensuring the test code is high level, well abstracted, and well named. Doing this will take far more time then running the manual test. So is it worth it? It's worth it when it ensures everyone really does understand what the application is supposed to do. Knowing what it actually does is secondary.

A good test suite will help you understand the app. Don't lose that by optimizing time spent testing.