Logging of GPU memory utilization can significantly slow down training
When training using GPUs pytorch_lightning automatically logs the GPU memory utilization during training. This is a useful feature, but can severely impact performance dependent on the speed of the nvidia-smi call.
On our particular cluster (University-scale hpc cluster based on IBM LSF), this leads to a performance decrease of almost 10 fold when training on GPU vs. CPU.
Describe the bug
Logging of GPU memory can have a severe impact on training performance.
To Reproduce
Remove gpu memory logging by commenting out the lines 937 to 939 in pytorch_lightning/models/trainer.py
see here
Expected behavior
Logging of GPU memory utilization should not impede performance (by running the nvidia-smi call in the background) or it should at least be possible to deactivate it in case performance issues arise.
Desktop (please complete the following information):

OS: Ubuntu Linux 16.04, NVIDIA Geforce GTX 1080