What are best practices for testing the absence of functionality

Question

Context: we operate in a highly regulated industry (medical), and aim to have automated test cases to cover all of our requirements - allowing us to still release quickly, but safely.

We have a requirement or acceptance criteria that reads something like:

x object should be read-only for users

Editing this object not a piece of functionality that is available in our web application (or via an API) - this state can only be created by the backend (kotlin) application itself, but it is important to do what we can to verify this, (and ideally in an automated way).

The problem: how do you test for the absence of some functionality?

Our current thinking is similar to this answer, specifically:

tests are just examples and not a proof

Therefore all our tests are examples of a sort, and it's acceptable to have a slightly wooly test, for example the absence of an edit button. It's likely that if we weren't in a regulated industry we'd not put as much thought into it, and accept that you have to trust the design to some extent for this type of requirement.

Some good thoughts from below (thanks for all the decent discussions):

• Code reviews/verification: yep, we do this
• API testing: testing the resource returned is read-only is something we do
• Security testing: absolutely will do this

Answer 1

For a web application, this is actually easy.

Presumably there is functionality to edit these objects (e.g. for maintainers or admins), and this functionality must somehow map to a system state change when certain requests are received from the web app. All you have to do is verify that when the server side receives such a request in the context of a user session, the state change doesn't occur.

(If you also want to verify that the object doesn't look editable in the front-end, that is a pure front-end test like other GUI tests. But the more important thing, as always in web applications, is to ensure that users cannot effect a particular change, no matter whether they issue the request through the official web client or by circumventing it.)

Answer 2

A common approach for requirements of the form "this must never happen" is fuzz testing.

The test case can be formulated like this:

1. Set up initial state.
2. Have a subroutine that verifies that the state of objects is as it should be.
3. Run a web application fuzzer for given time. Set it up so that it is logged in with user permissions and can click everything and send any forms.
4. Run the subroutine from step 2 again to verify that things that should be protected didn't change.

The browser-level fuzz testers for web applications are often called monkey testers, due to the way how they randomly click all buttons available on the user interface.

If your application has an API between frontend and backend, it is useful to also run an API-level fuzzer on it. This verifies that any access checks are properly implemented on the backend, to avoid bypassing by modifying the frontend javascript using browser tools.

Answer 3

Sometimes you can't actually test things. But you can verify them by inspection of the source code, build files, etc.

If you can show that the software provides no method for the user to edit something, then it must be read only.

Answer 4

Negative requirements are always tricky. I know many places where they are forbidden because of the difficulty in qualification.

Interestingly, the example you gave has already taken the step to turn it into a positive requirement. "x object should be read-only for users" is testable. You get object x, and you check that it is read-only.

You are reading into this, and turning it into "x object should not be writable for users (in any way)" which is truly a negative requirement. But the requirement as written has already been converted into a positive requirement by making a property "read-only" and using positive phrasings.

A common way to get close to the testing you seek is to break the software into layers. The general purpose layer is obliged to open files through an API provided by a core layer. In this API, you have a "is read-only" flag that can be tested for in a positive way. Then, in the core, you define what "read-only" actually means.

Testing the core may still require testing a negative, but now changes in the general layer can be tested with a simple positive test. Most changes are in this general layer, so most changes support fast releases.

And, once you get to the core, you might be able to test that negative by testing the complement. While there may be a myriad of ways to interact with a file in the general layer, there is likely to be only a handful of ways to interact with a file at the core. It then becomes reasonable to test the core behavior positively, and then argue that that correctly proves the negative phrased question.

All of this reminds me when I stood up on my soap box one day and asked for an explanation why "there will be no data races in this multi-threaded C++ application" wasn't a requirement...

Answer 5

One option may be to test for the inverse, with functionality being considered a failure state. Let's break the requirement up, and look at its success and failure states:

Requirement: x is read-only for users.

Success: x is read-only.
Failure: x is not read-only.

Of these, it tends to be much easier to determine whether an object is writeable, so you may want to craft a test for failure. Determine how the user is able to interact with x, and test all possible interactions. If even one interaction successfully modifies x, then you know that x is not read-only, and thus that the overarching requirement has not been met.

Answer 6

I am assuming the functionality does not exist and you therefore can't test it. That means for example data stored as a plain old object with no public member or setter. However you need to prove it does not exists.

From my experience with medical software you need to produce a document, probably a design review where you will describe several things :

• How is the data stored (database, file, in memory object, etc.)
• The means to edit this data is they were to exist (permissions, public setters, etc.)
• A description of the absence of such mean, by code review for example.

Ideally this document should be revised every time there are modifications on the scope of your data to ensure there is still no mean to edit it (at least once for each release where such modifications occur).

Answer 7

It’s tricky if your code is designed so that the compiler doesn’t even compile a statement that would modify x. Say const int x = 3; x = 5; You can’t compile and run a unit test that tests this.

I have at times collected such statements, commented them out with instructions that each of these statements should fail to compile, which you would have to check manually. So if someone changed x to be non-const, the assignment would compile which would be wrong.

For a developer it would be much preferable to have a const property than a read/write property where writing throws an unconditional exception.

Answer 8

A lot of languages support reflection.

You can often use this to check for readonly or final modifiers.