`map` with any joblib backend
#self-assign

Hi @lhoestq 👋🏼

I’d like to work on this!

Planning to support progress tracking with `map()` using any joblib backend (like "loky") by replacing the Queue-based approach in `iflatmap_unordered` with a file-based progress tracking mechanism (e.g. shared temp file with periodic updates).

This would allow the progress bar to work even in backends where inter-process Queues aren't supported. Let me know if this sounds good — I’ll get started!
 I think ideally it should have a general solution, since some joblib backends don't have a shared filesystem
We recently enabled the (experimental) parallel backend switch for data download and extraction but not for `map` yet.

Right now we're using our `iflatmap_unordered` implementation for multiprocessing that uses a shared Queue to gather progress updates from the subprocesses and show a progress bar in the main process.

If a Queue implementation that would work on any joblib backend by leveraging the filesystem that is shared among workers, we can have `iflatmap_unordered` for joblib and therefore a `map` with any joblib backend with a progress bar !

Note that the Queue doesn't need to be that optimized though since we can choose a small frequency for progress updates (like 1 update per second).