Inconsistent accuracy with pl.metrics.Accuracy() across PL 1.1.8 and PL 1.2.x
🐛 Bug
I have a simple binary segmentation model and train it to segment objects in an image. I measure the accuracy with pl.metrics.Accuracy(). After I switched from PL 1.1.8 to PL 1.2.x without any code-changes the accuracy-values where different (see also my discussion-topic).
I tried to reproduce the problem and even If I seed everything and use synthetic data the problem persist. I have a BoringModel-Colab which uses the same input-image over and over again to overfit the network on purpose.
If I run this with PL 1.1.8 for 3 epochs, I get
loss=0.738, val_binary_segmentation_accuracy=0.375
DATALOADER:0 TEST RESULTS
{'fake_test_acc': 0.7458580136299133}

and with PL 1.2.5 I get
loss=0.738, val_binary_segmentation_accuracy=0.444
DATALOADER:0 TEST RESULTS
{'fake_test_acc': 0.7458580136299133}

As the loss and the test-accuracy (which is also just the loss) is the same, I supect the inconsistency to be in the metric
Please reproduce using the BoringModel
https://colab.research.google.com/drive/1eRgcdQvNWzcEed2eTj8paDnnQ0qplXAh?usp=sharing
To Reproduce
Run the Colab once with:
! pip install torchtext==0.8.0 torchvision==0.8.0 torch==1.7.1
! pip install pytorch-lightning==1.1.8

and one with
 ! pip install pytorch-lightning==1.2.5

Expected behavior
In this case, where everything is seeded and I show the network just the same image, I would expect the accuracy to be the same.