Not-yet-existing resume_from_checkpoint for auto-resubmit
🚀 Feature
Accept Not-yet-existing resume_from_checkpoint in Trainer for automatic training resume / auto-resubmit.
Motivation
In cloud ML training services (e.g. Google AI platform training, AWS SageMaker, AWS Batch), there are Job auto-retry feature.
If we can specify checkpoint path, Job auto-retry can be used for training resume / resubmit.
Unfortunately, PyTorch-Lightning cannot specify Non-(yet-)existing file as resume_from_checkpoint argument of Trainer, it simply raise an error.
The motivation of this feature request is enabling training resume through Not-yet-existing resume_from_checkpoint.
(This feature looks similar to auto-resubmit of pl's SLURM. but I am totally newbie about it, it could be nonsense.)
Pitch
current checkpoint restore process:

  
    
      pytorch-lightning/pytorch_lightning/trainer/connectors/checkpoint_connector.py
    
    
        Lines 57 to 60
      in
      3abfec8
    
  
  
    

        
          
                   We attempt to restore weights in this order: 
        

        
          
                   1. HPC weights. 
        

        
          
                   2. if no HPC weights restore checkpoint_path weights 
        

        
          
                   3. otherwise don't restore weights 
        
    
  


It use (existing) resume_from_checkpoint.
If specified path's file is empty, it raise an error.
What I hope is
         1. HPC weights. 
         2. if no HPC weights ""try to"" restore checkpoint_path weights  
         3. otherwise don't restore weights 

It means that if checkpoint_path (resume_from_checkpoint) file do not exist, simply ignore it and start training from scratch.
In this case, training start normally from scratch, then pl save checkpoints.
If we set save-path == resume_from_checkpoint, latest checkpoint file exist in resume_from_checkpoint path.
When job auto-retry is triggered, because now checkpoint file exists in resume_from_checkpoint, in retried job pl load checkpoint from resume_from_checkpoint, so training properly resume.
Alternatives
Use hpc_save & hpc_load's resume system for normal training.
As far as I read the codes, "HPC weights load" (for slurm...?) enable auto-resubmit based on directory (not file) + file name rule (hpc_ckpt_{ckpt_number}.ckpt).
If we accept checkpoint directory (e.g. resume_from_checkpoint_dir), same mechanism can be used for resume/resubmit.