Looking for an example with custom metric for early_stop
❓ Questions and Help
What is your question?
Hi!
I was checking the docs and early_stop_callback and the only example I could find was https://pytorch-lightning.readthedocs.io/en/0.4.9/examples/Examples/#trainer-example. Is there any sample code where this is used? Im looking for a Lightning model that returns a custom metric and the early stop is based on that metric.
For my case, im working on NLP and the metric WER(Word error rate) is my main focus during training. I would like to configure the training so it stops when WER in the test set doesnt decrease for 2 epochs. Is this possible?
Code
This is my test step if it helps, right now I'm logging WER but my training loop doesnt use it for early stop
def test_step(self, batch, batch_idx):
        spectrograms, labels, input_lengths, label_lengths = batch
        y_hat = self(spectrograms)  # (batch, time, n_class)
        output = F.log_softmax(y_hat, dim=2)
        output = output.transpose(0, 1)  # (time, batch, n_class)

        loss = self.criterion(output, labels, input_lengths, label_lengths)

        decoded_preds, decoded_targets = GreedyDecoder(output.transpose(0, 1), labels, label_lengths)
        n_correct_pred = sum([int(a == b) for a, b in zip(decoded_preds, decoded_targets)])

        test_cer, test_wer = [], []
        for j in range(len(decoded_preds)):
            test_cer.append(cer(decoded_targets[j], decoded_preds[j]))
            test_wer.append(wer(decoded_targets[j], decoded_preds[j]))
        avg_cer = sum(test_cer) / len(test_cer)
        avg_wer = sum(test_wer) / len(test_wer)
        logs = {
            "cer": avg_cer,
            "wer": avg_wer,
        }
        return {
            "val_loss": loss,
            "n_correct_pred": n_correct_pred,
            "n_pred": len(spectrograms),
            "log": logs,
        }
What's your environment?

OS: [Linux, ]
Packaging [pip, ]
Version [0.9.0]