How to prepare list of files for dataloader, avoiding the duplicated work?
I do
self.image_paths = sorted(Path(self.hparams["image_path"]).rglob("*.jpg"))

in setup
and use self.image_paths to initialize the data loader in train_dataloader
I have 10m+ files and rglob takes some time.
My model is trained on 4 GPUs and as I understand I do rglob 4 times.
What is the best way to do it, so that it is done once and not 4 times?
__init__
setup ?
train_dataloader

somewhere else?