Slurm resubmit at the end of epoch.
From my understanding, the current resubmit will stop the model at the middle of epoch, which may have problem with dataloader resuming.
Is it possible that lightning automatically estimates that if a new epoch can be finished within the time limit, and decide if to halt or continue at the end of each epoch.