slurm auto re-queue inconsistency
Hi! I submitted a slurm job-array with pytorch lightning functionality. I used the suggested signal (#SBATCH --signal=SIGUSR1@90) and set distributed_backend to 'ddp' in the Trainer call. I did notice successful auto-resubmission this morning whenever my jobs were pre-emptied; however, I now notice that several of them have not completed and are not queued either. Wondering if this has been reported by someone earlier and any clue why this could happen? Is there a maximum to the number of times the jobs would be re-queued or other slurm rules that may prevent requeuing, etc.? My jobs might have been pre-emptied several times as I was running them on the low priority "non-capped" queue so as to occupy maximum number of gpus whenever they become available (already using my quota of high/medium priority queues). Thanks in advance!