Wall time auto-resubmit not working
Hi :)
I fear the wall time auto-resubmit is not working for me.
I'm using this submit script:
#SBATCH --job-name=NORA_test_MNIST
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=12000
#SBATCH --gres=gpu:1
### #SBATCH --mem-per-gpu=11000
#SBATCH -o /home/%u/%x-%j-on-%N.out
#SBATCH -e /home/%u/%x-%j-on-%N.err
#SBATCH --mail-type=ALL

#Timelimit format: "hours:minutes:seconds" -- max is 24h
#SBATCH --time=00:15:00

#Resubmit (provided by pytorch lightning)
#SBATCH --signal=SIGUSR1@90

echo "Your job is running on" $(hostname)
python3 mnist_model.py

mnist_model.py is the model from https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09
Pytorch Lightning 1.2.4 is installed on the cluster.
In the error file I get:
TPU available: None, using: 0 TPU cores
2021-03-22 11:19:52.797360: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
/home/gourmelon/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:51: UserWarning: GPU available but not used. Set the --gpus flag when calling the script.
  warnings.warn(*args, **kwargs)
Set SLURM handle signals.

  | Name    | Type   | Params
-----------------------------------
0 | layer_1 | Linear | 100 K
1 | layer_2 | Linear | 33.0 K
2 | layer_3 | Linear | 2.6 K
-----------------------------------
136 K     Trainable params
0         Non-trainable params
136 K     Total params
0.544     Total estimated model params size (MB)
slurmstepd-lme171: error: *** JOB 656086 ON lme171 CANCELLED AT 2021-03-22T11:34:49 DUE TO TIME LIMIT ***

In the out file I simply see the progress of training.
I also assured that the SIGUSR1 is actually sent using this script:
#!/bin/bash
#
#SBATCH --job-name=pers_job_debug
#SBATCH --dependency=singleton
#SBATCH --signal=B:SIGUSR1@90
#SBATCH --time=00:05:00
#SBATCH -o /cluster/%u/%x-%j-on-%N.out
#SBATCH -e /cluster/%u/%x-%j-on-%N.err
# catch the SIGUSR1 signal
_resubmit() {
    ## Resubmit the job for the next execution
    echo "$(date): job $SLURM_JOBID received SIGUSR1 at $(date), re-submitting"
    sbatch $0
}
trap _resubmit SIGUSR1

echo "$(date): job $SLURM_JOBID starting on $SLURM_NODELIST"
while true; do
    echo "$(date): normal execution"
    sleep 60
done

This script gives me:
Mon 22 Mar 2021 11:35:52 AM CET: normal execution
Mon 22 Mar 2021 11:36:52 AM CET: normal execution
Mon 22 Mar 2021 11:37:52 AM CET: normal execution
Mon 22 Mar 2021 11:38:52 AM CET: normal execution
Mon 22 Mar 2021 11:39:52 AM CET: job 656090 received SIGUSR1 at Mon 22 Mar 2021 11:39:52 AM CET, re-submitting
Submitted batch job 656092
Mon 22 Mar 2021 11:39:52 AM CET: normal execution

Any help is highly appreciated! Please also tell me if the example is not minimal enough (this is my first minimal example :D) or if I should try something else first.