Different training loss behavior between pytorch-lightning and pure pytorch
🐛 Bug


You can see three train loss curves here, actually there are four curves (in the legend below):

orange: pl on k80+cu110
red: pure pytorch on k80+cu110，which is fully overlaped under the orange one, so you can't see it.
grey: pl on p100+cu101
blue: pure pytorch on p100+cu101

I seed everything in these two models and set deterministic=True. I check the random init weights and training data order are all the same. The curves of k80 and p100 can't overlap is not surprising, but what confusing me is that:

two curves on k80 (orange and red) can overlap, yet two curves on p100 (grey and blue) can't overlap
on p100 (grey and blue), duing the starting several training steps(about 10+ steps), pl and pure pytorch loss value are the same, and two curves can overlap, , but then loss value become a little different (about 0.00x), and then more and more different, and two curves can't overlap any more.
curves on k80  (orange and red) finally convergence, yet on p100 (grey and blue) they stop at 0.5 and can't reduce any more.

Since the strange behavior on p100 can't be reproduced on k80, and my full project is huge, I don't know where the bug could be happen and how to create a smallest demo to report the bug. Any ideas?

Environment
K80:

CUDA:
- GPU:
- Tesla K80
- Tesla K80
- Tesla K80
- Tesla K80
- available:         True
- version:           11.0
Packages:
- numpy:             1.19.2
- pyTorch_debug:     False
- pyTorch_version:   1.7.1
- pytorch-lightning: 1.2.4
- tqdm:              4.58.0
System:
- OS:                Linux
- architecture:
- 64bit
-
- processor:         x86_64
- python:            3.7.2
- version:           #1 SMP Wed Feb 3 15:06:38 UTC 2021

P100:

CUDA:
- GPU:
- Tesla P40
- Tesla P40
- Tesla P40
- Tesla P40
- available:         True
- version:           10.1
Packages:
- numpy:             1.19.5
- pyTorch_debug:     False
- pyTorch_version:   1.4.0
- pytorch-lightning: 1.2.4
- tqdm:              4.57.0
System:
- OS:                Linux
- architecture:
- 64bit
- ELF
- processor:         x86_64
- python:            3.6.5
- version:           #1 SMP Wed Apr 1 17:49:22 CST 2020

Additional context