Early stopping callback 
🐛 Bug

Early stopping does not have the desired effect when creating a custom callback. Even when creating a custom callback with the default values, the training will stop before the early stopping before the conditions are met.
To Reproduce

Create callback

early_stop_callback = EarlyStopping(
        monitor='val_loss',
        min_delta=0.00,
        patience=3,
        verbose=False,
        mode='min',
        strict=True
    )

Create trainer

trainer = Trainer.from_argparse_args(Namespace(**dict(train_config)), early_stop_callback=early_stop_callback)

Train

trainer.fit(model)
Here are the validation steps in the model:
    def validation_step(self, batch, batch_idx):
        batch, y = batch
        y_hat = self(batch)

        loss = F.cross_entropy(y_hat, y.long())
        labels_hat = torch.argmax(y_hat, dim=1)
        n_correct_pred = torch.sum(y == labels_hat).item()

        return {'val_loss': loss, "n_correct_pred": n_correct_pred, "n_pred": len(y)}

    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        val_acc = sum([x['n_correct_pred'] for x in outputs]) / sum(x['n_pred'] for x in outputs)
        tensorboard_logs = {'val_loss': avg_loss, 'val_acc': val_acc}

        return {'val_loss': avg_loss, 'log': tensorboard_logs}
Expected behavior
In my case, training stops after 2 epochs, whether the validation loss increases or not. The callback behavior should be the same as the default. When I don't pass a custom callback, it works fine. I'm probably doing something wrong.
Environment

PyTorch Version : 1.4.0+cu100
OS: Ubuntu 18.04
How you installed PyTorch (conda, pip, source): pip
Python version: 3.6.9
CUDA/cuDNN version: 10.0.130/7.6.4
GPU models and configuration: GeForce GTX 860M

Thanks'!