batch size behaves strange for ddp
🐛 Bug
I am running SimCLR code. I notice the batch size plays an important role for contrastive learning, so I want to increase batch size.
I can run batch_size=64, on a single 2080Ti. When I use 4 GPUs by setting distributed_backend='ddp'. I still have to set batch_size=64, since each GPU is fully occupied by this setting when I checked by nvidia-smi. To be more clear, if I increase batch_size, there will be an error for CUDA out of memory.
However, from what I understand I should be able to set batch_size = 4*64 or so.
Please reproduce using [the BoringModel and post here]
I use the SimCLR example provided by pl.
To Reproduce

Expected behavior
I should be able to set batch_size = 4*64 or so.
Environment
I am using 1.0.2 version from conda-forge.  I understand the lastest version is .5 or .6, but I am pretty sure there are other bugs, so using the latest version I even cannot start train SimCLR.
Note: Bugs with code are solved faster ! Colab Notebook should be made public !

IDE: PyCharm on Ubuntu

Additional context