Dynamic Data Loaders
🚀 A dataloader with changable sampling behavior
A DataLoader that takes in a confusion matrix, or class specific recall at the end of the epoch. It then oversamples classes that are the least known to the network in the next epoch and hereby helps the network train more efficiently.
Motivation
I often work with highly imbalanced datasets, which I have previously been dealing with artificially oversampling classes that are less prevalent (label-balancing). Also I have been experimenting with extending this stratification from the plain labels to other confounders aswell (confounder-balancing). (* example provided at the end)
It would be great to make this process dynamic! Since the data stays the same and duplicating images seems a waste of memory and only the distribution changes over which samples are drawn from the data.
The idea is the following:
At the end of the epoch, I check how well the model is doing on the validation set and see if there are any subgroups (either on label or confounder level) and would pass the information on to my train dataloader (or its pytorch.utils.data.Sampler). Sticking to the example in the footnotes, I'd get the following validation errors:
Dog pics taken on the inside 0.4 MSE
Dog pics taken on the outside 0.3 MSE
Cat pics taken on the inside 0.5 MSE
Cat pics taken on the outside 0.2 MSE
Then I would call something like:
self.train_dataloader.sampler.distribution = [0.4, 0.3, 0.5, 0.2]
and self.train_dataloader.sampler would yield indices antiproportional to that distribution from its data_source, e.g. via torch.nn.functional.softmax(-torch.Tensor([0.4, 0.3, 0.5, 0.2])) = 
My concern
While pytorch offeres many strategies to tackle this, e.g. subclassing torch.utils.data.DataLoader or torch.utils.data.Sampler, I am not sure, how Pytorch Lightning is handling the dataloaders in the back, where they are stored and what the implications of that would be for the awesome functionalities Pytorch Lightning offeres (especially data-parallel methods etc.)
As this would be my first Pytorch Lightning project, I am yet unfamiliar with the inside functionality, so any ideas from experts with the API are highly appreaciated. I'll keep you posted with my progress! Best wishes and thanks for your input!

Example stratification (label-/confounder-balancing):
If I tried to classify cats vs. dogs from images and I have 20% dogs in my dataset, I would duplicate each dog picture 4 times and had a balanced dataset (label-balancing). If the pictures in my dataset had one of two backgrounds (outside/inside) and I had an imbalance like 25% dogs-inside 25% dogs-outside 40% cats-inside 10% cats-outside, then I had 50/50 labels, but my network is more likely to classify pictures taken inside to be from cats, since the label is more prevalent in that subclass. Knowing that this irregularity exists, I would oversample cat pictures taken outside and a few more dog pics from both classes so I'd have a balance dataset (confounder-balancing). Of course, these imbalances are not exclusive and for me it just boils down to, finding out as much as I can over my data and trying my best to reduce the all imbalances I find, unless the groups are super small, or the clientdoesn't care and thinks classifying images based on the background of the image is something he wants to pay me for (happen's more often than you think).