performance drop from v1.1.8 to >= v1.2.0 when using metrics
🐛 Bug
Hi, I'm observing longer epoch times for pytorch lightning versions >= 1.2.0 when using metrics in the training or validation steps. Depending on the model, dataset etc I observed an increase of 10 seconds per epoch.
I tested it using Accuracy() and ConfusionMatrix() and the impact is larger for the former.
Removing the metrics results in the same epoch times regardless of the lightning version.
My colleagues and I verified this on different machines using a simple mnist model.
This behavior also occurs with the soon-to-be deprecated internal metrics and the torchmetrics variant of Accuracy()
Currently, the mnist host website is down, so I replicated this using cifar10 in the boring model notebook, to which a link can be found below.
The impact in this example is not as severe as observed with our mnist example, but still in the range of 3-5 seconds per epoch on my machine, which some might find neglectable, but this adds up.
The problem using colab is, that I was not able to install pytorch lightning 1.1.8, since it is incompatible with the current torchtext library, which is imported by lightning if available, and is installed in the colab environment by default. Uninstalling it using pip did not work. To verify this one unfortunately has to run this somewhere else
The printed epoch/fit times in the notebook where only measured using 1.2.4.
The following times for 3 epochs and the complete fit were measured on my machine using a V100 for 1.1.8 and 1.2.4:
pl 1.2.4:
epoch: 64 s
epoch: 64 s
epoch: 64 s
fit: 198 s

pl 1.1.8
epoch: 59 s
epoch: 58 s
epoch: 59 s
fit: 182 s

I verified this behavior for all versions since 1.2.0 including the 1.2.0rcs.
Please reproduce using the BoringModel
https://colab.research.google.com/drive/1ILDOsZqlOKplrdKBkuGUiNiWnJwYYYoW?usp=sharing
To Reproduce
Use following https://colab.research.google.com/drive/1ILDOsZqlOKplrdKBkuGUiNiWnJwYYYoW?usp=sharing
Expected behavior
Same epoch times regardless of lightning version
Environment
this is the machine used for the times above

CUDA:

GPU:

Tesla V100-SXM2-32GB
Tesla V100-SXM2-32GB


available:         True
version:           10.2


Packages:

numpy:             1.20.1
pyTorch_debug:     False
pyTorch_version:   1.8.0
pytorch-lightning: 1.2.4
tqdm:              4.59.0


System:

OS:                Linux
architecture:

64bit
ELF


processor:         x86_64
python:            3.8.8
version:           #141-Ubuntu SMP Fri Feb 19 13:46:27 UTC 2021



Additional context