Model- and data-specific unit tests
🚀 Feature
Trainer option to run callbacks for model- and data-specific unit testing
Motivation
The docs mention a fast_dev_run option for initializing a trainer. Correct me if I'm wrong, but my understanding is this simply hits every line of code in the train/val loops to ensure there won't be any crashes on a full-scale training. This seems to fall a bit short of proper unit testing that would validate several assertions in addition to confirming there are no crashes.
Unit testing for machine learning models appears to be an ongoing and relatively immature development in the community. However, there are a few useful articles discussing its use and importance

PyTorch Lightning maintainer Jeremy Jordan wrote a helpful article that suggests several tests
This Medium article which was originally written for TensorFlow, but already has a PyTorch port here
Additional unit tests ideas are brainstormed in this article

Pitch


Define a set of model- and data-specific unit tests using the above articles as a starting point


Separate the information needed to run the tests into two categories

Generic: consistent across all models/dataset, can be deduced within the Lightning framework automatically
Specific: depends on the model and/or dataset - practitioner is expected to provide this



Determine an appropriate set of pl.LightningModule callbacks to allow users to provide the specific inputs discussed above


Use these callbacks in pl.Trainer(do_unit_tests=True) to execute the tests

I would suggest tests are on by default
If the callbacks have not been implemented, skip the tests and provide a logging message to the user to encourage them to define the callbacks



Alternatives

Use the existing fast_dev_run - I explain above why this is different from proper unit tests
Allow users to integrate with a 3rd party test library (like the link above for torchtest) - I think it would be preferable to have this functionality built-into the Lightning API rather than relying on other imports. Doing so would make it easier to automate the generic inputs discussed in the pitch
Expect users to write their own test suites - PyTorch Lightning strives to minimize boilerplate code and this seems like an excellent candidate for that!