README

Pragmatic testing. Runs all PHP files in the path ./themis-tests/ in the project root. If any assertions fail, any exceptions are not handled or anything is written to the error log - it is considered an error.

Themis can also compare the output of the PHP script with a file and raise record any discrepencies as errors.

Themis can run the PHP language test suite files with extension .phpt.

Error Conditions

In general; if the script completes with only writing to standard output or standard error using echo, print(), fwrite(STDERR, ...) or fwrite(STDOUT, ...) the test is considered successful.

The following events are considered a failure:

• If the PHP test function assert() is used, and it detects an error.
• If an expected exception is not thrown.
• If any exception is thrown and not caught by a global error handler.
• If something is logged using the error_log() function.
• If something is written to the STDERR stream.
• If the command exits and the exit code is >0, using exit(int $code).

Tests can also be conducted by comparing the STDOUT and STDERR logs against a validation file. This is useful if you are testing semantics like the order in which events occur.

Writing a test

Tests are written in the project root under the folder ./charm-tests/. The test scripts are just normal PHP files like this one:

Normal runtime errors

<?php
// triggers a division by zero
echo 1/0;

Assertions

The assert() function is a beautiful PHP function which is designed for testing. The beauty lies in the fact that this function has no performance overhead in production beacuse the PHP interpreter will automatically remove it. You should therefore use the assert() command often; particularly when the input to the function you are testing comes directly from your internal logic. Do NOT use it for validating user input (since the function is ignored in production).

<?php
// Triggers warning "Undefined variable $test"
assert($test === NULL, 'Variable $test is not NULL`);

Running tests

We recommend adding the themis script to your composer.json file:

    "scripts": {
        "test": "@php vendor/bin/charm-testing"
    },
    "script-descriptions": {
        "test": "Run testing"
    }

Next you simply run the command composer test from the command line:

> composer test

NOTE If you want to pass arguments to Charm/Testing, you have to separate arguments intended for composer from arguments intended for charm-testing:

composer test -- --help will give you argument help for charm-testing composer test --help will give you argument help for composer

Alternatively you can run the command ./vendor/bin/charm-testing directly.

Running a specific test

If you want to run only a specific test, you can provide part of the test filename as a parameter:

composer test KEYWORD

This will scan the ./charm-tests/ folder recursively, and only run the tests that contain the string KEYWORD in the path.

Automating tests

The tool provides error exit codes if any of the tests fail to run.

Writing a minimal test

Tests are written as ordinary PHP files in the ./charm-tests/ folder.

<?php
assert(false, "Error");

This file does not use any dependencies and relies on the PHP built in function assert(). The assert() function essentially just checks if the first argument is true, or it will trigger the error string given in argument 2.

You can use assert() throughout your code without worrying about performance implications because the command is a PHP language construct especially designed for testing. With the default configuration of PHP, the assert() function is completely ignored by PHP.

Validating a sequence of output

Sometimes you want to test some semantics or the order in which things are output. Charm/Testing can be configured to check the output from STDOUT or STDERR coming by simply creating a file with the same name as your script:

some-test.php should have a some-test.php.STDOUT file to validate STDOUT and a some-test.php.STDERR file to validate STDERR.

This example creates a "test" which will fail 50% of the times:

<?php
if (mt_rand(0,1) === 0) {
    echo "SUCCESS\n";
} else {
    echo "FAILURE\n";
}

To ensure that only SUCCESS\n is recordes as a positive test result, you can capture the output using the following command on your command line:

> php my-test-file.php > my-test-file.php.STDOUT

This will create a file which contains the expected output from a successfully run test.

Writing a test

Create a folder charm-tests/ in your project root. Tests are contained in simple PHP files which return an array like this:

<?php return [
    /**
     * First array item is a string that describes the test.
     */
    "Test description",

    /**
     * Second item is a function which performs the testing. The function
     * must return data which will be compared to the expected result.
     */
    function($data) {
        // perform your test and return a value
        return strtoupper($args);
    },

    /**
     * The next items are arrays of testing criteria and their expected result
     */
    [ "hello world",        "HELLO WORLD" ],        // succeeds
    [ "blåbærsyltetøy",     "BLåBæRSYLTETøY" ],     // succeeds
    [ "blåbærsyltetøy",     "BLÅBÆRSYLTEDØY" ],     // fails, but is the correct result

];


The test report
---------------

The test report contains the date and the result of all the tests with a nice
side-by-side diff highlighting the difference between the expected result and
the actual result.

Running the tests
-----------------

Run the function `vendor/bin/charm-testing` and you will see the tests being performed.
The output is a markdown file that you can save for future reference.


Simple Automation Example
-------------------------

The `charm-testing` command can easily be automated in various contexts, because it
provides an exit code which indicates the percentage of failed tests rounded up.

 * If no tests failed, the exit code is `0`.
 * If all tests failed, the exit code is `100`.
 * If 1 test failed, the exit code is `1` (or `100` if you only have
   one test. ;-)

Invoking the tests from PHP:

exec('exec /usr/bin/env vendor/bin/charm-testing', $output, $exitCode); if ($exitCode > 0) {

// some tests failed!

}

Handling failed tests from the command line:

./vendor/bin/charm-testing || echo "Some tests failed!"

or

if [[ ! $(./vendor/bin/charm-testing) ]]; then

echo "Some tests failed"

fi