Optimizer initialization with DDP
❓ Questions and Help
What is your question?
I would have expected optimizers to always be initialized after parameters have been moved to their destination device.
However, some ddp backends such as
ddp_backend,             ddp_spawn_backend,            ddp2_backend
initialize the optimizer with the CPU parameters before moving the model to the GPU while others such as gpu_backend pass the GPU parameters.
I'm currently trying to understand two things:
(1) Where does the linking from the CPU to GPU parameters happen?
(2) Is it actually necessary to initialize the optimizer before moving to the specific device or could it be done afterwards? (Most tutorials initialize the optimizer after placing the parameters on the corresponding device, e.g.  this one)
The reason I'm asking is that I did some parameter/gradient bending to be views into other tensors. This does not work with the current implementation as the optimizer keeps the reference to the CPU parameters with these tweaks but works fine when adapting the pytorch_lightning code by moving the optimizer creation after the model has been moved to the correct device.