Allow a flag into trainer to save checkpoints at partial epochs
🚀 Feature
Allow a flag into trainer to save checkpoints at partial epochs
Motivation
When you have a large dataset that takes tens of hours per epoch, it's important to have checkpoints along the way. Right now we only get a checkpoint on_epoch_end.
Workaround
Also interested to see if there is a good workaround. I guess I can set a reference to trainer inside my model and manually call on_epoch_end, but that feels like a hack and won't work without changing lightning code because of
if self.epochs_since_last_check >= self.period:
inside on_epoch_end.
Other ideas?
Also interested if there are better ways to solve this. Also thought about taking samples out of the dataset to make 'mini epochs,' but this breaks epochs naming convention. (eg an epoch implies all data has been run through once)
Thank you!