I want to know the most robust way of checking for PHP error messages when running automated tests that interact with a PHP application via a browser emulator.
The reason is: I use Behat, Mink and Selenium/Webkit/etc to test my PHP applications in the following ways:
- as acceptance tests, to make sure expected functionality exists
- as end-to-end tests, to test success of complete integration
And so it seems obvious to me that most tests run by Behat/Mink should additionally parse the pages they do tests on for PHP errors, warnings, notices etc, as a matter of course.
But, it's not easy, for a number of reasons:
- The server that the PHP application is running on may output errors differently, depending on factors such as whether XDebug is used.
- On a production server, errors, warnings and so on would not be output at all.
AFAIK, nothing can be done about point 2? So if I wanted my behat test results regarding parsing for PHP errors to be meaningful, it would have to be done on a server where the configuration is to display errors.
But regarding point 1, I have a simple idea for how to achieve it: explicitly use error and exception handlers within the PHP application so if/when they are output, they will be in a format that can be easily parsed and tested for.
It seems like a reasonable idea, yet has a bad smell about it. So I would like to know if:
- There is another way of more robustly looking for errors in end-to-end tests
- I am going about this whole thing in the wrong way.
