A model interpretability feature - visualize losses and data samples
🚀 Feature
An interpretability feature that allows you to log top model losses and visualize examples with the losses.
Motivation
To better understand the working of a trained model, it can be useful to analyze the examples in which your losses are doing well/bad. It would work particularly well with data that's interpretable like images and audio. Targeted actions can also be taken after such an analysis, such as creating specific data augmentation or training more on the harder examples.
Pitch
I don't have any solid information yet on the implementation or usefulness. Creating this issue to start a discussion on a potentially useful feature. Shared some thoughts on implementation below

A method in which the trainer runs validation while doing the appropriate logging of top losses and logging the data sample itself or index of the sample in the dataset. After this, the results can be saved as files or plotted
A separate class that handles interpretability. Something like what fastai has - https://docs.fast.ai/interpret