Distributed training on the Sun Grid Engine (SGE)
Has anyone ever managed to get PyTorch Lightning's distributed training to work on the Sun Grid Engine (or also called Oracle Grid Engine)? PyTorch Lightning only supports distributed training on multiple nodes with SLURM.
In SLURM you can also easily set the number of nodes in the submission script, I believe the closest SGE has to that is using array jobs (qsub -t <n>) but then nodes are lacking the necessary communication interface for distributed training.
One would probably have to implement some kind of synchronization mechanism for the individual jobs in the job array to get them to share gradients, which could be placed into a custom pytorch_lightning.core.LightningModule.configure_ddp. Additionally, a custom subclass of torch.nn.parallel.DistributedDataParallel seems to be necessary, similar to PyTorch Lightning's LightningDistributedDataParallel.
Is anyone aware of an implementation for distributed training with PyTorch Lightning on the Sun Grid Engine?