Tensorboard loss graph differs from command-line output when using accumulated gradients
🐛 Bug
Tensorboard loss graph differs from command-line output when using accumulated gradients
To Reproduce
Steps to reproduce the behavior:

Run a model with accumulated gradients
Compare printed loss to tensorboard loss
See error

Expected behavior
The loss displayed via tensorboard should agree with the command-line output.
Environment

CUDA:
- GPU:
- GeForce RTX 2070 with Max-Q Design
- available:         True
- version:           10.2
Packages:
- numpy:             1.18.1
- pyTorch_debug:     False
- pyTorch_version:   1.5.0
- pytorch-lightning: 0.7.3
- tensorboard:       2.2.1
- tqdm:              4.45.0
System:
- OS:                Linux
- architecture:
- 64bit
- ELF
- processor:         x86_64
- python:            3.8.2
- version:           #41158678979119.10~9593806-Ubuntu SMP Mon Apr 13 17:50:40 UTC