Gradient accumulation fails with fp16 precision
🐛 Bug
Setting accumulate_grad_batches > 1 and precision = 16 causes the following error:
RuntimeError: unscale_() has already been called on this optimizer since the last update().
Please reproduce using the BoringModel and post here
https://colab.research.google.com/drive/1_7pxqPlpc79k0VYlRdtRXE0JQbhSBWHy?usp=sharing
Environment

CUDA:

GPU:

Tesla T4


available:         True
version:           10.1


Packages:

numpy:             1.18.5
pyTorch_debug:     False
pyTorch_version:   1.6.0+cu101
pytorch-lightning: 1.0.4
tqdm:              4.41.1


System:

OS:                Linux
architecture:

64bit



processor:         x86_64
python:            3.6.9
version:           #1 SMP Thu Jul 23 08:00:38 PDT 2020