[load_dataset] shard and parallelize the process
I need the same feature for distributed training I think @TevenLeScao is exploring adding multiprocessing in `GeneratorBasedBuilder._prepare_split` - feel free to post updates here :) Posted a PR to address the building side, still needs something to load sharded arrow files + tests Closing as this feature has been implemented in #5107
- Some huge datasets take forever to build the first time. (e.g. oscar/en) as it's done in a single cpu core.
- If the build crashes, everything done up to that point gets lost

Request: Shard the build over multiple arrow files, which would enable:
- much faster build by parallelizing the build process
- if the process crashed, the completed arrow files don't need to be re-built again

Thank you!

@lhoestq 