the speed of each iteration halves with two GPUs
🐛 Bug
I am running this github repository https://github.com/mindslab-ai/faceshifter. In the first experiment, I used one GPU and the speed was about 1.34it/s (there was 994470 iterations, using DDP).  In the next experiment, I used two GPUs and this time the speed almost halved to 1.38s/it (this time there was 497235 iterations). Consequently, the speed of the whole algorithm does not change considerably.
Expected behavior
I expect that the speed of each iteration does not change with 2 GPUs, as I am working with DDP and the whole batch is given to one GPU and since I have not changed batch size in both experiments (with one and two GPUs).
Environment

PyTorch Version: 1.8
OS: ubuntu 18
Python version: Python 3.6.10
CUDA/cuDNN version: V11.1.74
GPUs: GeForce RTX 3090, GeForce RTX 3090