Support HTCondor in addition to SLURM
🐛 Bug
New DDP implementation is still not working for me. I am in the situation where I need to send a single job i.e. run a single executable due to the batch system we have on our university GPU cluster. Therefore, rather than calling the training script multiple times with different env variables as suggested in the documentation, my solution is to create a super script (see below) which will then call the main training script multiple times each time forking a new subprocess. However, the training never seem to commence in the end.
Code sample

My super script looks as follows

import sys
import subprocess
import time
import os
import random

def main():

    # fetch arguments
    args = sys.argv[1:]

    # Extract the number of gpus from argument list
    gpus = 0
    for idx in range(len(args)):
        if args[idx] == "--gpus":
            gpus = int(args[idx+1])

    # random master port 0
    master_port_0 = random.randint(10000, 19000)

    # create a subprocess for each gpu
    for gpu in range(gpus):
        my_env = os.environ.copy()
        my_env["MASTER_ADDR"] = "localhost"
        my_env["MASTER_PORT"] = str(master_port_0+gpu)
        my_env["WORLD_SIZE"] = str(gpus)
        my_env["NODE_RANK"] = str(gpu)
        my_env["LOCAL_RANK"] = "0"
        subprocess.Popen(["/usr/bin/python3", "train.py"] + args, env=my_env)
        print("Sleep for 10 seconds")
        time.sleep(10)

    print("Sleep for another 60 seconds")
    time.sleep(60)


The output that I get when I run for 2 GPUs is:

Allocating cuda
Model signature
Creating trainer
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Allocating cuda
Model signature
Creating trainer
initializing ddp: GLOBAL_RANK: 2, MEMBER: 3/2
Expected behavior
I would expect some output: "All DDP processes registered.." but I get nothing, it seems that my subprocesses get stuck somewhere.
Environment

PyTorch Version (e.g., 1.0): 1.5.1+cu101
OS (e.g., Linux): Linux
How you installed PyTorch (conda, pip, source): pip3
Python version: 3.7
CUDA/cuDNN version: 10.1