5537 speedup load
Awesome ! cc @mariosasko :) The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5704). All of your documentation changes will be reflected on that endpoint. Hi, thanks for working on this!

Your solution only works if the `root` is `""`, e.g., this would yield an incorrect result:
```python
dset = load_dataset("user/hf-dataset-repo", data_dir="path/to/data_dir")
```

Also, the `HfFileSystem` implementation in `datasets` will be replaced with the more powerful [one](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py) from `huggingface_hub` soon (I plan to open a PR that makes `find` much faster in the coming days). 

So I don't think we want to merge this PR in the current state, but thanks again for the effort.

 (I'll comment on the original issue to propose a cleaner solution) Ooof. Sorry, I should have checked that more thoroughly then! I would say we could just add that check and only use my approach if the root is "", which would still be faster in many cases, but it sounds like you have a better solution on the way. Thanks for the feedback Mario.
I reimplemented fsspec.spec.glob() in `hffilesystem.py` as `_glob`,  used it in `_resolve_single_pattern_in_dataset_repository` only, and saw a 20% speedup in times to load the config, on average. 

That's not much when usually this step takes only 2-3 seconds for most datasets, but in this particular case,  `bigcode/the-stack-dedup` , the loading time to get the config (not download the entire 6tb dataset, of course), went from ~170 secs to ~20 secs. 

What makes this work is this code in `_glob`:

```  
      if self.dir_cache is not None:
            allpaths = self.dir_cache
        else:
            allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
```

I also had to `import glob.has_magic( )` for `_glob()` (confusing, I know).

I hope there is no issue with copying most of the code from `fsspec.spec.glob`, as it is a BSD 3-Clause License, 
and I left a comment about this in the docstring of` _glob()`, that we may want to delete.

As mentioned, I evaluated the speedup across a random selection of about 1000 datasets (not all 27k+), and verified that old_config.eq(new_method_config) with the build in method, but deleted this test and related code changes on the subsequent commit. It's in the commit history if anyone wants to see it. (Note this does not include the outlier of  `bigcode/the-stack-dedup`

|   |   old_time     |    new _time |     diff   |  pct_diff |
| -- | -- | -- | -- | -- |
| mean  |    3.340 |    2.642 |    0.698 |   18.404 |
| min   |    2.024   |  1.976   |  -0.840  | -37.634 |
| max   |  66.582  |   41.517 |   30.927  |  85.538 |