Accuracy RuntimeError: cannot infer num_classes when target is all zero
🐛 Bug
ptl.metrics.Accuracy can't infer num_classes from the target vector.
Throws RuntimeError: cannot infer num_classes when target is all zero is each GPU, but inspection of attributes shows that should work.
The error is triggered in this call:
File /aidio/lightning_modules.py", line 340, in validation_step
    self.val_acc(y_pred, y_target)
Where y_pred is:
tensor([[ 0.0339, -0.0203,  0.0025, -0.1014,  0.0925,  0.0877,  0.0401, -0.1383,
          0.0699, -0.0062, -0.0090,  0.0610, -0.1358, -0.0682, -0.0389,  0.1113,
         -0.0719,  0.0433,  0.1169,  0.0307,  0.0602, -0.1659, -0.1272,  0.1022,
          0.0464,  0.0248, -0.0447, -0.0293,  0.0052, -0.1008, -0.0643, -0.0774],
        [ 0.0339, -0.0203,  0.0025, -0.1014,  0.0925,  0.0877,  0.0401, -0.1383,
          0.0699, -0.0062, -0.0090,  0.0610, -0.1358, -0.0682, -0.0389,  0.1113,
         -0.0719,  0.0433,  0.1169,  0.0307,  0.0602, -0.1660, -0.1272,  0.1022,
          0.0464,  0.0248, -0.0447, -0.0293,  0.0052, -0.1009, -0.0643, -0.0774],
        [ 0.0339, -0.0203,  0.0025, -0.1014,  0.0925,  0.0877,  0.0401, -0.1383,
          0.0699, -0.0062, -0.0090,  0.0610, -0.1358, -0.0681, -0.0389,  0.1113,
         -0.0718,  0.0432,  0.1169,  0.0307,  0.0602, -0.1659, -0.1271,  0.1022,
          0.0464,  0.0248, -0.0447, -0.0292,  0.0052, -0.1008, -0.0643, -0.0773],
        [ 0.0339, -0.0203,  0.0025, -0.1014,  0.0925,  0.0877,  0.0401, -0.1383,
          0.0699, -0.0062, -0.0090,  0.0610, -0.1358, -0.0682, -0.0389,  0.1113,
         -0.0719,  0.0433,  0.1169,  0.0307,  0.0602, -0.1660, -0.1272,  0.1022,
          0.0464,  0.0248, -0.0447, -0.0293,  0.0052, -0.1009, -0.0643, -0.0774],
        [ 0.0339, -0.0203,  0.0025, -0.1014,  0.0925,  0.0877,  0.0401, -0.1383,
          0.0699, -0.0062, -0.0090,  0.0610, -0.1358, -0.0682, -0.0389,  0.1113,
         -0.0718,  0.0433,  0.1169,  0.0307,  0.0602, -0.1659, -0.1272,  0.1022,
          0.0464,  0.0248, -0.0447, -0.0293,  0.0052, -0.1008, -0.0643, -0.0774],
        [ 0.0339, -0.0203,  0.0025, -0.1014,  0.0925,  0.0877,  0.0401, -0.1383,
          0.0699, -0.0062, -0.0090,  0.0610, -0.1358, -0.0682, -0.0389,  0.1113,
         -0.0719,  0.0433,  0.1169,  0.0307,  0.0602, -0.1660, -0.1272,  0.1022,
          0.0464,  0.0248, -0.0447, -0.0293,  0.0052, -0.1009, -0.0643, -0.0774]],
       device='cuda:3')
and y_target is:
tensor([0, 0, 0, 0, 0, 0], device='cuda:3')
To Reproduce
I can't reproduce the error in a CPU environment with the given values, but is somehow triggering the error on DDP mode on 4 GPUs.
But the error is triggering on evaluation loop before the epoch 0 even starts, so i guess it's like a health check or something.
This is my pythroch lighning scheme:
    def __init__(self, hparams, num_classes, train_dataset, eval_dataset, test_dataset, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.hparams = hparams
        self.wd = hparams.weight_decay
        self.lr = hparams.learning_rate
        self.batch_size = hparams.batch_size
        self.loss = torch.nn.CrossEntropyLoss()
        self.train_dataset = train_dataset
        self.eval_dataset = eval_dataset
        self.test_dataset = test_dataset
        self.train_acc = ptl.metrics.Accuracy()
        self.val_acc = ptl.metrics.Accuracy()
        self.test_acc = ptl.metrics.Accuracy()
        # After this constructor should define self.model and self.optimizer

    def forward(self, x):
        """
        No special modification required for lightning, define as you normally would
        :param x:
        :return:
        """
        return self.model(x)

    def training_step(self, batch, batch_idx):
        """
        Lightning calls this inside the training loop
        :param batch:
        :return:
        """
        # forward pass
        x, y_target = batch['x'], batch['y']
        y_pred = self.forward(x)
        # calculate metrics
        loss = self.loss(y_pred, y_target)
        self.train_acc(y_pred, y_target)
        # log metrics
        self.log('train_loss', loss, prog_bar=True, )
        self.log('train_acc', self.train_acc, prog_bar=True, )
        return loss

    def validation_step(self, batch, batch_idx):
        """
        Lightning calls this inside the validation loop
        :param batch:
        :return:
        """
        x, y_target = batch['x'], batch['y']
        y_pred = self.forward(x)
        # calculate metrics
        loss = self.loss(y_pred, y_target)
        self.val_acc(y_pred, y_target)
        # gather results
        self.log('val_loss', loss, prog_bar=True, )
        self.log('val_acc', self.val_acc, prog_bar=True, )
        return loss
Expected behavior
Accuracy should be able to calculate the number of classes from y_pred instead of y_target. As, Accuracy parameters are:
    preds (float or long tensor): (N, ...) or (N, C, ...) where C is the number of classes
    target (long tensor): (N, ...)
Even if the y_target are all 0s, the number of classes could be inferred from y_pred.shape[1].
This error shouldn't be appearing.
Environment

PyTorch Version (e.g., 1.0): 1.6.1
OS (e.g., Linux): Linux
Pytorch-Lighting Version: 1.0.1
Python version: 3.8.5
GPU models and configuration: 4 x GPU  ~10 GB on DDP
Any other relevant information:

Additional context
The model I'm running has a high number of classes and a low batch size. The model is huge, so I need to distribute small batches across many GPUs. So my case of use will be always a classification training with a batch size of ~10 elements and with ~ 50 classes. Thus, there will be many times that I'll need to get the accuracy with a target vector that doesn't have all the possible classes.
I think the number of classes shouldn't be inferred from the target vector in runtime in the case of use.
The complete stack trace for each GPU is:
Traceback (most recent call last):
  File "/mnt/ialabnas/homes/voyanedel/aidio/model_manager.py", line 271, in <module>
    helper.train()
  File "/mnt/ialabnas/homes/voyanedel/aidio/model_manager.py", line 106, in train
    self.trainer.fit(self.module)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1046, in fit
    self.accelerator_backend.train(model)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/accelerators/ddp_backend.py", line 57, in train
    self.ddp_train(process_idx=self.task_idx, mp_queue=None, model=model)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/accelerators/ddp_backend.py", line 224, in ddp_train
    results = self.trainer.run_pretrain_routine(model)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1224, in run_pretrain_routine
    self._run_sanity_check(ref_model, model)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1257, in _run_sanity_check
    eval_results = self._evaluate(model, self.val_dataloaders, max_batches, False)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 333, in _evaluate
    output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 661, in evaluation_forward
    output = model(*args)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/overrides/data_parallel.py", line 174, in forward
    output = self.module.validation_step(*inputs[0], **kwargs[0])
  File "/mnt/ialabnas/homes/voyanedel/aidio/lightning_modules.py", line 344, in validation_step
    raise e
  File "/mnt/ialabnas/homes/voyanedel/aidio/lightning_modules.py", line 340, in validation_step
    self.val_acc(y_pred, y_target)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/metric.py", line 84, in __call__
    return apply_to_collection(self._orig_call(*args, **kwargs), torch.Tensor,
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/converters.py", line 81, in new_func
    result = function_to_decorate(*args, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/converters.py", line 81, in new_func
    result = function_to_decorate(*args, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/converters.py", line 58, in new_func
    return func_to_decorate(*args, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/classification.py", line 87, in forward
    return accuracy(pred=pred, target=target,
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/functional/classification.py", line 270, in accuracy
    raise RuntimeError("cannot infer num_classes when target is all zero")
RuntimeError: cannot infer num_classes when target is all zero
Traceback (most recent call last):
  File "/mnt/ialabnas/homes/voyanedel/aidio/model_manager.py", line 271, in <module>
    helper.train()
  File "/mnt/ialabnas/homes/voyanedel/aidio/model_manager.py", line 106, in train
    self.trainer.fit(self.module)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1058, in fit
    results = self.accelerator_backend.spawn_ddp_children(model)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/accelerators/ddp_backend.py", line 123, in spawn_ddp_children
    results = self.ddp_train(local_rank, mp_queue=None, model=model, is_master=True)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/accelerators/ddp_backend.py", line 224, in ddp_train
    results = self.trainer.run_pretrain_routine(model)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1224, in run_pretrain_routine
    self._run_sanity_check(ref_model, model)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1257, in _run_sanity_check
    eval_results = self._evaluate(model, self.val_dataloaders, max_batches, False)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 333, in _evaluate
    output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 661, in evaluation_forward
    output = model(*args)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/overrides/data_parallel.py", line 174, in forward
    output = self.module.validation_step(*inputs[0], **kwargs[0])
  File "/mnt/ialabnas/homes/voyanedel/aidio/lightning_modules.py", line 344, in validation_step
    raise e
  File "/mnt/ialabnas/homes/voyanedel/aidio/lightning_modules.py", line 340, in validation_step
    self.val_acc(y_pred, y_target)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/metric.py", line 84, in __call__
    return apply_to_collection(self._orig_call(*args, **kwargs), torch.Tensor,
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/converters.py", line 81, in new_func
    result = function_to_decorate(*args, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/converters.py", line 81, in new_func
    result = function_to_decorate(*args, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/converters.py", line 58, in new_func
    return func_to_decorate(*args, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/classification.py", line 87, in forward
    return accuracy(pred=pred, target=target,
  File "/mnt/ialabnas/homes/voyanedel/miniconda3/envs/py385/lib/python3.8/site-packages/pytorch_lightning/metrics/functional/classification.py", line 270, in accuracy
    raise RuntimeError("cannot infer num_classes when target is all zero")
RuntimeError: cannot infer num_classes when target is all zero