EarlyStopping when using an IterableDataset
I am trying to use an IterableDataset while also including early stopping. Using version 0.6.1, it looks like the early stopping callback is only checked after an epoch. When using an IterableDataset, I don't see how this is ever called.
I have implemented a quick solution on my local machine after noticing a metric check after the validation loop is run:

  
    
      pytorch-lightning/pytorch_lightning/trainer/training_loop.py
    
    
         Line 415
      in
      4c6c3d0
    
  
  
    

        
          
           self.early_stop_callback.check_metrics(self.callback_metrics) 
        
    
  


My quick fix was to add a call to EarlyStopping.on_epoch_end like so:
# ---------------
# RUN VAL STEP
# ---------------
is_val_check_batch = (batch_idx + 1) % self.val_check_batch == 0
can_check_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
should_check_val = (not self.disable_validation and can_check_epoch and
                    (is_val_check_batch or early_stop_epoch))

# fast_dev_run always forces val checking after train batch
if self.fast_dev_run or should_check_val:
    self.run_evaluation(test=self.testing)

    if self.enable_early_stop:
        self.early_stop_callback.check_metrics(self.callback_metrics)
        # CHECK EARLY STOP CALLBACK HERE
        early_stop_epoch = self.early_stop_callback.on_epoch_end(self.current_epoch, self.callback_metrics)
Now that early_stop_epoch is set after each validation, it is possible to stop using an IterableDataset.
# end epoch early
# stop when the flag is changed or we've gone past the amount
# requested in the batches
if early_stop_epoch or self.fast_dev_run:
    break
If there is a native way to trigger the early stopping callback already, then my question is how? If not, then I could submit a PR.
What's your environment?

OS: Ubuntu 18
Packaging: pip
Version: 0.6.1