Leaked semaphores with DDP training
I constantly get this warning when training on an AWS instance (8 GPUs, using DDP). It does not crash, but the training hangs for a few seconds before continuing.
/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 3 leaked semaphores to clean up at shutdown

I can share my docker container if necessary, as it might be an issue with library versions.