Getting an error when trying to use torch.load on model trained using DDP
❓ Questions and Help
What is your question?
I'm trying to save my model with torch.save() with the trainer and logger detached and then load it with torch.load(). If I train the model using DDP on 2 P4 gpus, I get the following error when I try to load it:
RuntimeError: [enforce fail at inline_container.cc:222] . file not found: archive/data/94881266361728

Reverting back to pytorch-lightning version 0.7.5 (big version gap.. I know) seems to fix the issue as does training on a single GPU. I'm wondering if there's something that got attached to the model somewhere in the version gap that I need to detach before pickling.
Code
I can't share code unfotunately.
   
What have you tried?
I've tried running on a single GPU or reverting lightning versions.
What's your environment?

OS: [e.g. iOS, Linux, Win] Ubuntu 20.04
Packaging [e.g. pip, conda] conda
Version [e.g. 0.5.2.1] 1.0.7 (w/ pytorch 1.7 and cuda 11)