multi-gpu training is slow in lightning
🐛 Bug
I have migrated my code from pytorch to lightning. However, I noticed the iteration time is almost double in lightning. I have tried with one server and two servers (each have 4 V100 GPUs). It is always slower in lightning. I am using ddp distributed backend. However, I have noticed the same with horovod as well. I passed number --gpus  for my script in the argument so I can set num_workers > 0.
Expected behavior
I  expected that lightning is just a wrapper over pytorch. I dont get why I am having reduction in training speed.
Environment

PyTorch: 1.6.0
PyTorch-Lightning: 0.9.0
How you installed PyTorch: pip
GPU models and configuration: (2 nodes with 4 V100 GPUs each).