Logging loss is extremely large when training with fp16
🐛 Bug

Logging loss is extremely large(e.g. 10000) when training with fp16, but I find that the loss in tfboard is quite normal(e.g. 1.0). I guess the loss in logging does not discard unstable training steps?
Expected behavior

Loss in logging should be approximate between turning on/off fp16. Also,  logging loss should be consistent with tfboard loss.
Environment
Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).
You can get the script and run it with:
wget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py
# For security purposes, please check the contents of collect_env_details.py before running it.
python collect_env_details.py


PyTorch Version (e.g., 1.0): nightly
OS (e.g., Linux): Linux
How you installed PyTorch (conda, pip, source): pip
Build command you used (if compiling from source):
Python version: 3.6
CUDA/cuDNN version: 10.2
GPU models and configuration:
Any other relevant information:

Additional context