Early Stopping + result dictionary + no validation not working.
🐛 Bug
The case where the user does not use validation and returns a dictionary (instead of a TrainResult) during training does not work in combination with early stopping.
The test case which should check this is here:

  
    
      pytorch-lightning/tests/callbacks/test_early_stopping.py
    
    
        Lines 136 to 159
      in
      bd35c86
    
  
  
    

        
          
           def test_early_stopping_no_val_step(tmpdir): 
        

        
          
               """Test that early stopping callback falls back to training metrics when no validation defined.""" 
        

        
          
            
        

        
          
               class CurrentModel(EvalModelTemplate): 
        

        
          
                   def training_step(self, *args, **kwargs): 
        

        
          
                       output = super().training_step(*args, **kwargs) 
        

        
          
                       output.update({'my_train_metric': output['loss']})  # could be anything else 
        

        
          
                       return output 
        

        
          
            
        

        
          
               model = CurrentModel() 
        

        
          
               model.validation_step = None 
        

        
          
               model.val_dataloader = None 
        

        
          
            
        

        
          
               stopping = EarlyStopping(monitor='my_train_metric', min_delta=0.1) 
        

        
          
               trainer = Trainer( 
        

        
          
                   default_root_dir=tmpdir, 
        

        
          
                   early_stop_callback=stopping, 
        

        
          
                   overfit_batches=0.20, 
        

        
          
                   max_epochs=2, 
        

        
          
               ) 
        

        
          
               result = trainer.fit(model) 
        

        
          
            
        

        
          
               assert result == 1, 'training failed to complete' 
        

        
          
               assert trainer.current_epoch < trainer.max_epochs 
        
    
  


The check in the last line is wrong. It should actually compare:
    assert trainer.current_epoch < trainer.max_epochs - 1
To Reproduce
Steps to reproduce the behavior:

Fix the test case
Run tests.

Code sample
I guess using the test case is simpler and easier.
Expected behavior
That is an interesting question indeed. Possibilities are:

Test case should pass with correct comparison
The docs and  @williamFalcon in #3193 (comment) suggest that only 'loss' should work.

So before fixing this issue, it should be settled what the expected behavior is.
If you tell me, I'm happy to help.
I could also include it in the pull request where I already tried to bring the docs in line with the test cases.
Environment

CUDA:

GPU:

GeForce GTX 1080 Ti


available:         True
version:           10.1


Packages:

numpy:             1.19.1
pyTorch_debug:     False
pyTorch_version:   1.6.0
pytorch-lightning: 0.9.1dev
tensorboard:       2.2.0
tqdm:              4.48.2


System:

OS:                Linux
architecture:

64bit
ELF


processor:         x86_64
python:            3.8.3
version:           #113~16.04.1-Ubuntu SMP Fri Jul 10 04:37:08 UTC 2020



Additional context