Only two cores are getting used in sagemaker with pytorch 3.10 kernel
I think it's more likely that this issue is related to PyTorch than Datasets, as PyTorch (on import) registers functions to execute when forking a process. Maybe this is the culprit: https://github.com/pytorch/pytorch/issues/99625 From reading that ticket, it may be down in mkl?  Is it worth hotfixing in the meantime, with the express intention of turning it off?  I know that's a horribly crufty solution, but it's also deeply frustrating to be limited to 2 cores for operations as simple as filtration. This is too specific and unrelated to `datasets`, so this shouldn't be fixed here. @mariosasko @mmr-crexi  I had the exact same problem on my kubernetes cluster. the datasets subprocess only user 1 and 17 core
### Describe the bug

When using the newer pytorch 3.10 kernel, only 2 cores are being used by huggingface filter and map functions.  The Pytorch 3.9 kernel would use as many cores as specified in the num_proc field.

We have solved this in our own code by placing the following snippet in the code that is called inside subprocesses:

```os.sched_setaffinity(0, {i for i in range(1000)})```

The problem, as near as we can tell, us that once upon a time, cpu affinity was set using a bitmask ("0xfffff" and the like), and affinity recently changed to a list of processors rather than to using the mask.  As such, only processors 1 and 17 are shown to be working in htop.
![Selection_072](https://github.com/huggingface/datasets/assets/107141022/04c5a824-5321-4531-afca-7bc84dff36b4)


When running functions via `map`, the above resetting of affinity works to spread across the cores.  When using `filter`, however, only two cores are active.

### Steps to reproduce the bug

Repro steps:

1.  Create an aws sagemaker instance
2. use the pytorch 3_10 kernel
3. Load a dataset
4. run a filter operation
5. watch as only 2 cores are used when num_proc > 2
6. run a map operation
7. watch as only 2 cores are used when num_proc > 2
8. run a map operation with processor affinity reset inside the function called via map
9. Watch as all cores run



### Expected behavior

All specified cores are used via the num_proc argument.

### Environment info

AWS sagemaker with the following init script run in the terminal after instance creation:

conda init bash
bash
conda activate pytorch_p310
pip install Wand PyPDF pytesseract datasets seqeval pdfplumber transformers pymupdf sentencepiece timm donut-python accelerate optimum xgboost
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
sudo yum -y install htop
sudo yum -y update
sudo yum -y install wget libstdc++ autoconf automake libtool autoconf-archive pkg-config gcc gcc-c++ make libjpeg-devel libpng-devel libtiff-devel zlib-devel