LightningDataModule seems to do some dataloader operations on CPU, which was not the case with LightningModule loader methods
🐛 Bug
While using LightningDataModule as lit_model(datamodule=datamodule) the models waits for some time using 1 CPU core before beginging training, and periodically stops training (every 50 train steps) GPU util goes 0% and 1 CPU core is in use. This behaviour continues till training finishes.
To Reproduce
Steps to reproduce the behavior:

Create a lightning model which takes a datamodule as input. __init__ containsthis.datamodule=datamodule
In the LightningDataModule I'm using PyTorch's UCF101 dataset


Code sample
class UCF101DataModule(LightningDataModule):
    def setup(self, stage=None):
        if stage == "fit" or stage is None:
            self.train_dataset = datasets.UCF101(
                UCF101_ROOT_PATH,
                UCF101_ANNO_PATH,
                frames_per_clip=5,
                step_between_clips=30,
                num_workers=UCF101_WORKERS,
                train=True,
                fold=self.fold,
            )

    def train_dataloader(self):
        print("Train Dataloader Called")
        return DataLoader(
            self.train_dataset,
            batch_size=self.batch_size,
            num_workers=DATALOADER_WORKERS,
            collate_fn=custom_collate,
            shuffle=True,
        )

class LitModel(LightningModule):
    def __init__(datamodule):
        self.datamodule = datamodule
Expected behavior
Training loop should start immediately
Environment

CUDA:

GPU:

GeForce RTX 2080 Ti


available:         True
version:           10.2


Packages:

numpy:             1.18.1
pyTorch_debug:     False
pyTorch_version:   1.6.0
pytorch-lightning: 0.9.0
tqdm:              4.46.0


System:

OS:                Linux
architecture:

64bit
ELF


processor:         x86_64
python:            3.8.3
version:          113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020