Problem with syncing logged values with multi-gpu and ddp
🐛 Bug
When logging values with sync_dist and ddp (on two GPUs), the logged value is changed and the wrong value of averaged metric is logged.
It can be reproduced with dummy training_step() and batch_size=1
    def training_step(self, batch, batch_idx):
        loss = torch.tensor(1.0, device=self.device, requires_grad=True)
        self.log('loss', loss, on_epoch=True, sync_dist=True)
        return loss

The logged and returned value is 2.0
To Reproduce
This was discovered running ddp with two GPUs
Expected behavior
Logged value 1.0, value of loss not changed
Environment

CUDA:

GPU:

GeForce GTX TITAN X
GeForce GTX TITAN X
GeForce GTX TITAN X
GeForce GTX TITAN X


available:         True
version:           11.0


Packages:

numpy:             1.19.4
pyTorch_debug:     True
pyTorch_version:   1.7.0
pytorch-lightning: 1.1.6
tqdm:              4.54.0


System:

OS:                Linux
architecture:

64bit
ELF


processor:
python:            3.7.8
version:           1 SMP PREEMPT Sun, 27 Dec 2020 10:50:46 +0000



Additional context
Logging was done with Comet.ml