0.5.3 broke DDP vs 0.5.2.1
Using ddp on 0.5.3.2 would cause a gpu worker to crash at an epoch transition (start of epoch 7 for me) and hang the whole training process. I rolled back to 0.5.2.1 (used for my last project) and the issue was gone. Single gpu training works fine on both versions and toggling amp makes no difference. I'm using pytorch 1.3 in NGC 19.10-py3.
Any ideas on how to debug this? since it just hangs without an error code...
Edit:
Should add that I don't have any special hooks/functions running at the start of epochs