ValueError: bad value(s) in fds_to_keep when using DDP
🐛 Bug
I see the following error when I try to use ddp for distributed training. I see #538 is the similar issue, but I couldn't figure out how to apply the solution to my problem.
The following is the error log.
2019-12-27 22:17:31,780 - __main__ - INFO - Loaded dictionary
2019-12-27 22:17:31,816 - model.dictionary - INFO - Error parsing line 703
2019-12-27 22:17:31,832 - gensim.models.utils_any2vec - INFO - loading projection weights from data_in/GoogleNews-vectors-negative300.bin.gz
2019-12-27 22:19:41,750 - gensim.models.utils_any2vec - INFO - loaded (3000000, 300) matrix from data_in/GoogleNews-vectors-negative300.bin.gz
2019-12-27 22:19:45,292 - __main__ - INFO - Loaded and imported w2v weights (50161) words
2019-12-27 22:19:45,696 - root - INFO - gpu available: True, used: True
2019-12-27 22:19:45,696 - root - INFO - VISIBLE GPUS: 0,1
Traceback (most recent call last):
  File "snrm_trainer.py", line 128, in <module>
    main(hparams)
  File "snrm_trainer.py", line 94, in main
    trainer.fit(model)
  File "/home/kyoungrok/anaconda3/envs/sigir/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 343, in fit
    mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))
  File "/home/kyoungrok/anaconda3/envs/sigir/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 162, in spawn
    process.start()
  File "/home/kyoungrok/anaconda3/envs/sigir/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/kyoungrok/anaconda3/envs/sigir/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/kyoungrok/anaconda3/envs/sigir/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/kyoungrok/anaconda3/envs/sigir/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/kyoungrok/anaconda3/envs/sigir/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 59, in _launch
    cmd, self._fds)
  File "/home/kyoungrok/anaconda3/envs/sigir/lib/python3.7/multiprocessing/util.py", line 432, in spawnv_passfds
    False, False, None)
ValueError: bad value(s) in fds_to_keep
To Reproduce
Code sample
I attach two source files:

snrm.py (model)
snrm_trainer.py  (trainer)

Expected behavior
The model runs on multi-gpus
Environment
PyTorch version: 1.3.1
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.13.4

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 430.64
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.4

Versions of relevant libraries:
[pip] numpy==1.16.4
[pip] pytorch-lightning==0.5.3.2
[pip] torch==1.3.1
[pip] torchvision==0.4.2
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.4                      243
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.15           py37ha843d7b_0
[conda] mkl_random                1.1.0            py37hd6b4f25_0
[conda] pytorch                   1.3.1           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] pytorch-lightning         0.5.3.2                  pypi_0    pypi
[conda] torchvision               0.4.2                py37_cu101    pytorch

Additional context
None