ModelCheckpoint with monitor='loss' and save_best_only=True crashes
checkpoint_callback = ModelCheckpoint(checkpoint_path, monitor='loss', save_best_only=True)
trainer = Trainer(gpus=[0], checkpoint_callback=checkpoint_callback)
trainer.fit(model)
The above code crashes with the following output:
gpu available: True, used: True
VISIBLE GPUS: 0
                Name        Type    Params
0     encoder_layers  ModuleList  29461600
1   encoder_layers.0      Linear  26910000
2   encoder_layers.1      Linear   2001000
3   encoder_layers.2      Linear    500500
4   encoder_layers.3      Linear     50100
5     decoder_layers  ModuleList  29474954
6   decoder_layers.0      Linear     50500
7   decoder_layers.1      Linear    501000
8   decoder_layers.2      Linear   2002000
9   decoder_layers.3      Linear  26921454
10              loss      L1Loss         0
100%|█| 100/100 [00:19<00:00,  5.50it/s, batch_nb=99, epoch=0, gpu=0, loss=201.344, v_nb=3]save callback...
Traceback (most recent call last):
  File "/home/akonstantinov/.pycharm_helpers/pydev/pydevd.py", line 1758, in <module>
    main()
  File "/home/akonstantinov/.pycharm_helpers/pydev/pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/akonstantinov/.pycharm_helpers/pydev/pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/akonstantinov/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/akonstantinov/cross_platform_norm/dae/main.py", line 15, in <module>
    trainer.fit(model)
  File "/home/akonstantinov/.miniconda3/envs/cpn/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 742, in fit
    self.__single_gpu_train(model)
  File "/home/akonstantinov/.miniconda3/envs/cpn/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 790, in __single_gpu_train
    self.__run_pretrain_routine(model)
  File "/home/akonstantinov/.miniconda3/envs/cpn/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1025, in __run_pretrain_routine
    self.__train()
  File "/home/akonstantinov/.miniconda3/envs/cpn/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1053, in __train
    self.run_training_epoch()
  File "/home/akonstantinov/.miniconda3/envs/cpn/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1105, in run_training_epoch
    self.__run_evaluation(test=self.testing)
  File "/home/akonstantinov/.miniconda3/envs/cpn/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1467, in __run_evaluation
    logs=self.__training_tqdm_dict)
  File "/home/akonstantinov/.miniconda3/envs/cpn/lib/python3.7/site-packages/pytorch_lightning/callbacks/pt_callbacks.py", line 238, in on_epoch_end
    if self.monitor_op(current, self.best):
TypeError: ufunc 'less' did not contain a loop with signature matching types dtype('<U32') dtype('<U32') dtype('bool')
Quick debugging shows that the crash happens because current variable contains loss in form of a string instead of a float.