DDP is incompatible with large datasets
I'm trying to stream a large data file instead of loading it so it doesn't have to be pickled for multi-processing. However, open-file objects give a TypeError: cannot serialize '_io.TextIOWrapper' object error, so I have to open it within a subprocess instead-- but train_dataloader and val_dataloader methods get called in the main process of pytorch-lightning! How can I bypass issue without changing the source code?