Schedule model testing every N training epochs
🚀 Feature
A check_test_every_n_epoch trainer option to schedule model testing every n epochs, just like check_val_every_n_epoch for validation.
Motivation
Sometimes validation and test tasks are very different. For instance, in unsupervised anomaly detection or segmentation, the training and validation set cannot contain anomalous samples, and therefore the set of metrics that can be computed on such set is limited.
Test metrics cannot also be computed on a second validation set that contains a portion of test data, because all parameters and hyperparameter optimization should be performed on clean (anomaly-free) samples.
The only way the user has to check test metrics is to run a test epoch, but currently pytorch-lightning allows to run a test epoch only after the trainer.fit() call has ended, by explicitly calling trainer.test().
Pitch
When initializing a Trainer with check_test_every_n_epoch: x every x epochs the fitting automatically stops, a test epoch is executed, logging and computing test metrics, and then the fitting resumes.
Alternatives
I tried to:

Calling self.trainer.test() from on_training_epoch_start module callback when the value of trainer.current_epochs becomes multiple of a desired value. Apparently, doing this works fine, but after calling the test method, the number of epochs continues to increase from the last value, but the trainer global_step is reset to the value it had when test was last called, creating the beautiful effect shown in figure and making logs unreadable.
Raising KeyboardInterrupt from on_training_epoch_start module callback to stop the fitting, testing, manually saving a checkpoint, and resuming training from the saved checkpoint in a while loop. Cumbersome, poor disk usage, cannot gracefully stop training with Ctrl+C anymore.
Computing test metrics in validation_epoch_end by manually looping on test set using pytorch only. Cumbersome, code duplication, makes test hooks useless, poor code readability.

Additional context
Tensorboard logging current epoch each step when trying solution 1: