Metric names when using multiple dataloader for validation
Hi guys,
I am using multiple dataloaders for validation. This works great so far, but I have some questions regarding the logged metrics:
As far as I understand, lightning will automatically assign a metric suffix (/dataloader_idx_X). Is there any way for me to control that behaviour? Wandb assumes groups with that slash, and it means, that it groups by metric name and not validation set. However, there are scenarios where it makes much more sense to group by validation set, as they might look at completely different metrics. On the bottom, there is a screenshot on how it looks without any further wanbd configuration.
Apart from my question, this is more of a feature proposal: Can't we supply a (ordered) dictionary instead of a list via val_dataloader() in DataModules? So that it looks like this:
DataModule:
def val_dataloader(self) -> Union[Dict[str, DataLoader], List[Dataloader], DataLoader]:
   return {
      'scenario1': DataLoader('...'),
      'scenario2': DataLoader('...'),
  }
LightningModule
def validation_step(self, batch, batch_idx: int, loader: Union[str, int]):
   # do something
   self.log('my_metric', x)
Which then logs scenarioX/my_metric in case of the example or dataloader_idx_X/my_metric when a list is provided.