Consider fuzzy approval for non-deterministic programs #4
Description
In my use case of a concurrent program it proved to be useful to be somewhat lenient when approving (large) outputs against the master files.
The output of two threads was in a race condition, leading to switched lines in the output files. The harness contains a MAX_DEVIATION constant (currently set to 0.0) that can be used to allow minor variations in the output. Non-perfect matches with the master files are then reported as "aborted" tests (equivalent to failed assumptions up to JUnit 4), which are then neither "green" nor "red."
Of course, this can only work for minor differences. If the program output is very loosely related to the input or contains many timestamps (e.g. through logging), this approach is too simple.