Multiple Trainloaders in a Sequential Mode.
🚀 Feature

Add new mode to multiple_trainloader_mode of the Trainer in order to support training on multiple dataloaders in a sequential way.
Motivation

Say we want to train on two dataloaders A, and B.
Currently, the modes support max/min size of all trainloaders and provide a list/dict containing batches from loader A and B at each step. But it is not possible to define loaders such that it returns batches from loader A until it is exhausted, and then returns batches from loader B until exhausted.
Pitch

Adding a new mode, called sequential loading to the Trainer. At each time, the batch list/dict can have only one batch from one of the loaders.
Example application: Finetuning
Say we want at each epoch, train a model on one dataset and then finetune it on another.
loaders = {'train': loader_train, 'finetune': loader_finetune} 

for epoch in epochs:
   train(model, loaders['train'])
   train(model, loaders['finetune'])
Alternatives


Chaining dataloaders together as a single dataloader seems possible but then additional book-keeping is required to figure out which dataloader a specific batch belongs to.
If we can call the train_loop of the LightningModule, we can add a callback that is called at the end of each training epoch and then finetune the model on the finetune loader.

Additional Context
To the best of my knowledge, there is a CombinedLoader that takes care of parallel sampling. I suppose a new mode can be added to this class to support sequential loading.
But I could be wrong and there is a better alternative. In that case, please let me know.
Thanks.