Why training with lightning is 1.5x slower than vanilla pytorch
I am assuming this is due to user error but cannot figure it out
Running pytorch-lightning==0.9.0 on Linux
Converted pytorch code for training bert transformer model to pytorch lightning.
Vanilla code trains ~1.5x faster than lightning code. (4.8 it/s vs 2.7 it/s)
Running both code bases on 1x GPU, both with 32 bit precision.
Tried various configs on lightning code
Tried distributed_backend: dp/None both same it/sec; ddp ~10% slower than dp/None
lightning code
pytorch code
Unclear as to what to try next to resolve this