Data loading hangs before first validation step
🐛 Bug
After training epoch, before the first validation step, training gets stuck somewhere in the data loaders (I think).
I can't provide a reproduction script unfortunately: Getting the training into the specific situation takes a long time (must train for long enough for the situation to arise).
I train on 4x 1080 Ti using DDP and num_workers=20. After the first training epoch, before the first validation, training gets stuck. All GPUs are reported to have 100% compute and memory utilization, but only 50/250 W power consumption. Only the 4 main Python threads seem to be doing any working (busy looping?). The 20 worker processes seem to have been stopped already.
To me it looks like the main threads are still busy waiting for new samples, while the dataloaders have already gone.
Note that I use limit_train_batches=0.1, maybe this is the cause?
Unfortunately I don't have ptrace capability on the machine, so can't use GDB etc. I printed the stack traces of all Python threads every 10s using a debugging thread. Logs of the hang situation are here: https://gist.github.com/jonashaag/b74ae9fc9267bde2cecd35ae316232c0
I am currently training without limit_train_batches to see if it's due to that setting. EDIT: No, I can also reproduce without limit_train_batches set.
Environment
* CUDA:
        - GPU:
                - GeForce GTX 1080 Ti
                - GeForce GTX 1080 Ti
                - GeForce GTX 1080 Ti
                - GeForce GTX 1080 Ti
        - available:         True
        - version:           11.0
* Packages:
        - numpy:             1.19.2
        - pyTorch_debug:     True
        - pyTorch_version:   1.8.0.dev20201028
        - pytorch-lightning: 0.10.0
        - tqdm:              4.51.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                -
        - processor:         x86_64
        - python:            3.7.8
        - version:           #88~16.04.1-Ubuntu SMP Wed Feb 12 04:19:15 UTC 2020