Update navbar chapter titles color

Consistency with the color change that was done in transformers at https://github.com/huggingface/transformers/pull/7423
It makes the background-color of the chapter titles in the docs navbar darker, to differentiate them from the inner sections.

see changes [here](https://691-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html)


Adding missing @property (+2 small flake8 fixes).

Fixes #678


Fix bug related to boolean in GAP dataset.

Hi !

Good catch, thanks for creating this PR :)

Could you also regenerate the metadata for this dataset using
```
datasets-cli test ./datasets/gap --save_infos --all_configs
```

That'd be awesome

@lhoestq Thank you for your revieing!!!

I've performed it and have read CONTRIBUTING.md now!

### Why I did
The value in `row["A-coref"]` and `row["B-coref"]` is `'TRUE'` or `'FALSE'`.
This type is `string`, then `bool('FALSE')` is equal to `True` in Python.
So, both rows are transformed into `True` now.

So, I modified this problem.

### What I did
I modified `bool(row["A-coref"])` and `bool(row["B-coref"])` to `row["A-coref"] == "TRUE"` and `row["B-coref"] == "TRUE"`.

Thank you!


Fix negative ids when slicing with an array

```python
from datasets import Dataset

d = ds.Dataset.from_dict({"a": range(10)})
print(d[[0, -1]])
# OverflowError
```

raises an error because of the negative id.

This PR fixes that.
Fix #668


The download instructions for c4 datasets are not contained in the error message

Good catch !
Indeed the `@property` is missing.

Feel free to open a PR :)

Also not that C4 is a dataset that needs an Apache Beam runtime to be generated.
For example Dataflow, Spark, Flink etc.

Usually we generate the dataset on our side once and for all, but we haven't done it for C4 yet.
More info about beam datasets [here](https://huggingface.co/docs/datasets/beam_dataset.html)

Let me know if you have any questions

The manual download instructions are not clear
```The dataset c4 with config en requires manual data.
 Please follow the manual download instructions: <bound method C4.manual_download_instructions of <datasets_modules.datasets.c4.830b0c218bd41fed439812c8dd19dbd4767d2a3faa385eb695cf8666c982b1b3.c4.C4 object at 0x7ff8c5969760>>.
 Manual data can be loaded with `datasets.load_dataset(c4, data_dir='<path/to/manual/data>')
```

Either `@property`  could be added to C4.manual_download_instrcutions (or make it a real property), or the manual_download_instructions function needs to be called I think.

Let me know if you want a PR for this, but I'm not sure which possible fix is the correct one.


Move cache dir root creation in builder's init

We use lock files in the builder initialization but sometimes the cache directory where they're supposed to be was not created. To fix that I moved the builder's cache dir root creation in the builder's init.

Fix #671


train_test_split returns empty dataset item

The problem still exists after removing the cache files.

Can you reproduce this example in a Colab so we can investigate? (or give more information on your software/hardware config)

Thanks for reporting.
I just found the issue, I'm creating a PR

We'll do a release pretty soon to include the fix :)
In the meantime you can install the lib from source if you want to

I try to split my dataset by `train_test_split`, but after that the item in `train` and `test` `Dataset` is empty.
The codes:
```
yelp_data = datasets.load_from_disk('/home/ssd4/huanglianzhe/test_yelp')
    print(yelp_data[0])
    yelp_data = yelp_data.train_test_split(test_size=0.1)
    print(yelp_data)
    print(yelp_data['test'])
    print(yelp_data['test'][0])
```
The outputs:
```
{'stars': 2.0, 'text': 'xxxx'}
Loading cached split indices for dataset at /home/ssd4/huanglianzhe/test_yelp/cache-f9b22d8b9d5a7346.arrow and /home/ssd4/huanglianzhe/test_yelp/cache-4aa26fa4005059d1.arrow
DatasetDict({'train': Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 7219009), 'test': Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 802113)})
Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 802113)
{}    # yelp_data['test'][0] is empty
```


Add custom dataset to NLP?

Yes you can have a look here: https://huggingface.co/docs/datasets/loading_datasets.html#csv-files

Is it possible to add a custom dataset such as a .csv to the NLP library?

Thanks.


load_dataset() won't download in Windows

I have the same issue. Tried to download a few of them and not a single one is downloaded successfully.

This is the output:
```
>>> dataset = load_dataset('blended_skill_talk', split='train')
Using custom data configuration default               <-- This step never ends
```

This was fixed in #644
I'll do a new release soon :)

In the meantime you can run it by installing from source

Closing since version 1.1.0 got released with Windows support :)
Let me know if it works for you now

I don't know if this is just me or Windows. Maybe other Windows users can chime in if they don't have this problem. I've been trying to get some of the tutorials working on Windows, but when I use the load_dataset() function, it just stalls and the script keeps running indefinitely without downloading anything. I've waited upwards of 18 hours to download the 'multi-news' dataset (which isn't very big), and still nothing. I've tried running it through different IDE's and the command line, but it had the same behavior. I've also tried it with all virus and malware protection turned off. I've made sure python and all IDE's are exceptions to the firewall and all the requisite permissions are enabled.

Additionally, I checked to see if other packages could download content such as an nltk corpus, and they could. I've also run the same script using Ubuntu and it downloaded fine (and quickly). When I copied the downloaded datasets from my Ubuntu drive to my Windows .cache folder it worked fine by reusing the already-downloaded dataset, but it's cumbersome to do that for every dataset I want to try in my Windows environment.

Could this be a bug, or is there something I'm doing wrong or not thinking of?

Thanks.


blog_authorship_corpus crashed

Thanks for reporting !
We'll free some memory

This is just to report that When I pick blog_authorship_corpus in
https://huggingface.co/nlp/viewer/?dataset=blog_authorship_corpus
I get this:
![image](https://user-images.githubusercontent.com/7553188/94349542-4364f300-0013-11eb-897d-b25660a449f0.png)


Questions about XSUM

We should try to regenerate the data using the official script.
But iirc that's what we used in the first place, so not sure why it didn't match in the first place.

I'll let you know when the dataset is updated

Thanks, looking forward to hearing your update on this thread.

This is a blocking issue for us; would appreciate any progress on this front. We can also help with the fix, if you deem it appropriately.

I just started the generation on my side, I'll let you know how it goes :)

Hmm after a first run I'm still missing 136668/226711 urls.
I'll relaunch it tomorrow to try to get the remaining ones.

Update: I'm missing 36/226711 urls but I haven't managed to download them yet

Thanks! That sounds like a reasonable number!

So I managed to download them all but when parsing only 226,181/226,711 worked.
Not sure if it's worth digging and debugging parsing at this point :/

Maybe @sshleifer can help, I think he's already played with xsum at one point

Thanks @lhoestq
It would be great to improve coverage, but IDs are the really crucial part for us. We'd really appreciate an update to the dataset with IDs either way!

I gave up at an even earlier point. The dataset I use has 204,017 train examples.

@lhoestq @sshleifer like @jbragg said earlier, the main issue for us is that the current XSUM dataset (in your package) does not have IDs suggested by the original dataset ([here is the file](https://raw.githubusercontent.com/EdinburghNLP/XSum/master/XSum-Dataset/XSum-TRAINING-DEV-TEST-SPLIT-90-5-5.json).) Would appreciate if you update the XSUM dataset to include the instance IDs.

The missing instances is also a problem, but likely not worth pursuing given its relatively small scale.

>So I managed to download them all but when parsing only 226,181/226,711 worked.

@lhoestq any chance we could update the HF-hosted dataset with the IDs in your new version? Happy to help if there's something I can do.

Well I couldn't parse what I downloaded.
Unfortunately I think I won't be able to take a look at it this week.
I can try to send you what I got if you want to give it a shot @jbragg
Otherwise feel free to re-run the xsum download script, maybe you'll be luckier than me

Hi there ✋

I'm looking into your `xsum` dataset and I have several questions on that.
So here is how I loaded the data:
```
>>> data = datasets.load_dataset('xsum', version='1.0.1')
>>> data['train']
Dataset(features: {'document': Value(dtype='string', id=None), 'summary': Value(dtype='string', id=None)}, num_rows: 204017)
>>> data['test']
Dataset(features: {'document': Value(dtype='string', id=None), 'summary': Value(dtype='string', id=None)}, num_rows: 11333)
```

The first issue is, the instance counts don’t match what I see on [the dataset's website](https://github.com/EdinburghNLP/XSum/tree/master/XSum-Dataset#what-builds-the-xsum-dataset) (11,333 vs 11,334 for test set; 204,017 vs 204,045 for training set)
```
 … training (90%, 204,045), validation (5%, 11,332), and test (5%, 11,334) set.
```
Any thoughts why? Perhaps @mariamabarham could help here, since she recently had a PR on this dataaset https://github.com/huggingface/datasets/pull/289  (reviewed by @patrickvonplaten)

Another issue is that the instances don't seem to have IDs. The original datasets provides IDs for the instances: https://github.com/EdinburghNLP/XSum/blob/master/XSum-Dataset/XSum-TRAINING-DEV-TEST-SPLIT-90-5-5.json but to be able to use them, the dataset sizes need to match.

CC @jbragg


[BUG] No such file or directory

This happens when both
1. Huggingface datasets cache dir does not exist
2. Try to load a local dataset script

builder.py throws an error when trying to create a filelock in a directory (cache/datasets) that does not exist
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L177

Tested on v1.0.2

@lhoestq


Fix SQuAD metric kwargs description

The `answer_start` field was missing in the kwargs docstring.

This should fix #657

FYI another fix was proposed by @tshrjn in #658 and suggests to remove this field.
However IMO `answer_start` is useful to match the squad dataset format for consistency, even though it is not used in the metric computation. I think it's better to keep it this way, so that you can just give references=squad["answers"] to .compute().

Let me know what sounds the best for you


How to skip a example when running dataset.map

Hi @xixiaoyao,
Depending on what you want to do you can:
- use a first step of `filter` to filter out the invalid examples: https://huggingface.co/docs/datasets/processing.html#filtering-rows-select-and-filter
- or directly detect the invalid examples inside the callable used with `map` and return them unchanged or even remove them at the same time if you are using `map` in batched mode. Here is an example where we use `map` in batched mode to add new rows on the fly but you can also use it to remove examples on the fly (that's what `filter` actually do under-the-hood): https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset

Closing this one.
Feel free to re-open if you have other questions

in processing func, I process examples and detect some invalid examples, which I did not want it to be added into train dataset. However I did not find how to skip this recognized invalid example when doing dataset.map.


OverflowError when slicing with an array containing negative ids

```python
from datasets import Dataset

d = ds.Dataset.from_dict({"a": range(10)})

print(d[0])
# {'a': 0}

print(d[-1])
# {'a': 9}

print(d[[0, -1]])
# OverflowError
```
results in
```
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-5-863dc3555598> in <module>
----> 1 d[[0, -1]]

~/Desktop/hf/nlp/src/datasets/arrow_dataset.py in __getitem__(self, key)
   1070             format_columns=self._format_columns,
   1071             output_all_columns=self._output_all_columns,
-> 1072             format_kwargs=self._format_kwargs,
   1073         )
   1074

~/Desktop/hf/nlp/src/datasets/arrow_dataset.py in _getitem(self, key, format_type, format_columns, output_all_columns, format_kwargs)
   1025                 indices = key
   1026
-> 1027             indices_array = pa.array([int(i) for i in indices], type=pa.uint64())
   1028
   1029             # Check if we need to convert indices

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

OverflowError: can't convert negative value to unsigned int
```


Loss not decrease with Datasets and Transformers

And I tested it on T5ForConditionalGeneration, that works no problem.

Hi did you manage to fix your issue ?

If so feel free to share your fix and close this thread

HI,

The following script is used to fine-tune a BertForSequenceClassification model on SST2.

The script is adapted from [this colab](https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb) that presents an example of fine-tuning BertForQuestionAnswering using squad dataset. In that colab, loss works fine. When I adapt it to SST2, the loss fails to decrease as it should. I attach the adapted script below and appreciate anyone pointing out what I miss?

```python
import torch
from datasets import load_dataset
from transformers import BertForSequenceClassification
from transformers import BertTokenizerFast
# Load our training dataset and tokenizer
dataset = load_dataset("glue", 'sst2')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
del dataset["test"] # let's remove it in this demo

# Tokenize our training dataset
def convert_to_features(example_batch):
    encodings = tokenizer(example_batch["sentence"])
    encodings.update({"labels": example_batch["label"]})
    return encodings

encoded_dataset = dataset.map(convert_to_features, batched=True)
# Format our dataset to outputs torch.Tensor to train a pytorch model
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'labels']
encoded_dataset.set_format(type='torch', columns=columns)

# Instantiate a PyTorch Dataloader around our dataset
# Let's do dynamic batching (pad on the fly with our own collate_fn)
def collate_fn(examples):
    return tokenizer.pad(examples, return_tensors='pt')

dataloader = torch.utils.data.DataLoader(encoded_dataset['train'], collate_fn=collate_fn, batch_size=8)
# Now let's train our model
device = 'cuda' if torch.cuda.is_available() else 'cpu'
# Let's load a pretrained Bert model and a simple optimizer
model = BertForSequenceClassification.from_pretrained('bert-base-cased', return_dict=True)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)
model.train().to(device)
for i, batch in enumerate(dataloader):
    batch.to(device)
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    model.zero_grad()
    print(f'Step {i} - loss: {loss:.3}')

```
In case needed.

- datasets == 1.0.2
- transformers == 3.2.0


Does both 'bookcorpus' and 'wikipedia' belong to the same datasets which Google used for pretraining BERT?

No they are other similar copies but they are not provided by the official Bert models authors.


runing dataset.map, it raises TypeError: can't pickle Tokenizer objects

Hi !
It works on my side with both the LongFormerTokenizer and the LongFormerTokenizerFast.

Which version of transformers/datasets are you using ?

transformers and datasets are both the latest

Then I guess you need to give us more informations on your setup (OS, python, GPU, etc) or a Google Colab reproducing the error for us to be able to debug this error.

And your version of `dill` if possible :)

I have the same issue with `transformers/BertJapaneseTokenizer`.

```python
# train_ds = Dataset(features: {
#     'title': Value(dtype='string', id=None),
#     'score': Value(dtype='float64', id=None)
# }, num_rows: 99999)

t = BertJapaneseTokenizer.from_pretrained('bert-base-japanese-whole-word-masking')
encoded = train_ds.map(lambda examples: {'tokens': t.encode(examples['title'])}, batched=True)
```

<details><summary>Error Message</summary>

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-35-2b7d66b291c1> in <module>
      2
      3 encoded = train_ds.map(lambda examples:
----> 4   {'tokens': t.encode(examples['title'])}, batched=True)

/usr/local/lib/python3.6/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1242                 fn_kwargs=fn_kwargs,
   1243                 new_fingerprint=new_fingerprint,
-> 1244                 update_data=update_data,
   1245             )
   1246         else:

/usr/local/lib/python3.6/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    151             "output_all_columns": self._output_all_columns,
    152         }
--> 153         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    154         if new_format["columns"] is not None:
    155             new_format["columns"] = list(set(new_format["columns"]) & set(out.column_names))

/usr/local/lib/python3.6/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    156                         kwargs_for_fingerprint["fingerprint_name"] = fingerprint_name
    157                         kwargs[fingerprint_name] = update_fingerprint(
--> 158                             self._fingerprint, transform, kwargs_for_fingerprint
    159                         )
    160

/usr/local/lib/python3.6/site-packages/datasets/fingerprint.py in update_fingerprint(fingerprint, transform, transform_args)
    103     for key in sorted(transform_args):
    104         hasher.update(key)
--> 105         hasher.update(transform_args[key])
    106     return hasher.hexdigest()
    107

/usr/local/lib/python3.6/site-packages/datasets/fingerprint.py in update(self, value)
     55     def update(self, value):
     56         self.m.update(f"=={type(value)}==".encode("utf8"))
---> 57         self.m.update(self.hash(value).encode("utf-8"))
     58
     59     def hexdigest(self):

/usr/local/lib/python3.6/site-packages/datasets/fingerprint.py in hash(cls, value)
     51             return cls.dispatch[type(value)](cls, value)
     52         else:
---> 53             return cls.hash_default(value)
     54
     55     def update(self, value):

/usr/local/lib/python3.6/site-packages/datasets/fingerprint.py in hash_default(cls, value)
     44     @classmethod
     45     def hash_default(cls, value):
---> 46         return cls.hash_bytes(dumps(value))
     47
     48     @classmethod

/usr/local/lib/python3.6/site-packages/datasets/utils/py_utils.py in dumps(obj)
    365     file = StringIO()
    366     with _no_cache_fields(obj):
--> 367         dump(obj, file)
    368     return file.getvalue()
    369

/usr/local/lib/python3.6/site-packages/datasets/utils/py_utils.py in dump(obj, file)
    337 def dump(obj, file):
    338     """pickle an object to a file"""
--> 339     Pickler(file, recurse=True).dump(obj)
    340     return
    341

/usr/local/lib/python3.6/site-packages/dill/_dill.py in dump(self, obj)
    444             raise PicklingError(msg)
    445         else:
--> 446             StockPickler.dump(self, obj)
    447         stack.clear()  # clear record of 'recursion-sensitive' pickled objects
    448         return

/usr/local/lib/python3.6/pickle.py in dump(self, obj)
    407         if self.proto >= 4:
    408             self.framer.start_framing()
--> 409         self.save(obj)
    410         self.write(STOP)
    411         self.framer.end_framing()

/usr/local/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    474         f = self.dispatch.get(t)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return
    478

/usr/local/lib/python3.6/site-packages/dill/_dill.py in save_function(pickler, obj)
   1436                                 globs, obj.__name__,
   1437                                 obj.__defaults__, obj.__closure__,
-> 1438                                 obj.__dict__, fkwdefaults), obj=obj)
   1439         else:
   1440             _super = ('super' in getattr(obj.func_code,'co_names',())) and (_byref is not None) and getattr(pickler, '_recurse', False)

/usr/local/lib/python3.6/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    608         else:
    609             save(func)
--> 610             save(args)
    611             write(REDUCE)
    612

/usr/local/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    474         f = self.dispatch.get(t)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return
    478

/usr/local/lib/python3.6/pickle.py in save_tuple(self, obj)
    749         write(MARK)
    750         for element in obj:
--> 751             save(element)
    752
    753         if id(obj) in memo:

/usr/local/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    474         f = self.dispatch.get(t)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return
    478

/usr/local/lib/python3.6/site-packages/dill/_dill.py in save_module_dict(pickler, obj)
    931             # we only care about session the first pass thru
    932             pickler._session = False
--> 933         StockPickler.save_dict(pickler, obj)
    934         log.info("# D2")
    935     return

/usr/local/lib/python3.6/pickle.py in save_dict(self, obj)
    819
    820         self.memoize(obj)
--> 821         self._batch_setitems(obj.items())
    822
    823     dispatch[dict] = save_dict

/usr/local/lib/python3.6/pickle.py in _batch_setitems(self, items)
    850                 k, v = tmp[0]
    851                 save(k)
--> 852                 save(v)
    853                 write(SETITEM)
    854             # else tmp is empty, and we're done

/usr/local/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    519
    520         # Save the reduce() output and finally memoize the object
--> 521         self.save_reduce(obj=obj, *rv)
    522
    523     def persistent_id(self, obj):

/usr/local/lib/python3.6/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    632
    633         if state is not None:
--> 634             save(state)
    635             write(BUILD)
    636

/usr/local/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    474         f = self.dispatch.get(t)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return
    478

/usr/local/lib/python3.6/site-packages/dill/_dill.py in save_module_dict(pickler, obj)
    931             # we only care about session the first pass thru
    932             pickler._session = False
--> 933         StockPickler.save_dict(pickler, obj)
    934         log.info("# D2")
    935     return

/usr/local/lib/python3.6/pickle.py in save_dict(self, obj)
    819
    820         self.memoize(obj)
--> 821         self._batch_setitems(obj.items())
    822
    823     dispatch[dict] = save_dict

/usr/local/lib/python3.6/pickle.py in _batch_setitems(self, items)
    845                 for k, v in tmp:
    846                     save(k)
--> 847                     save(v)
    848                 write(SETITEMS)
    849             elif n:

/usr/local/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    519
    520         # Save the reduce() output and finally memoize the object
--> 521         self.save_reduce(obj=obj, *rv)
    522
    523     def persistent_id(self, obj):

/usr/local/lib/python3.6/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    632
    633         if state is not None:
--> 634             save(state)
    635             write(BUILD)
    636

/usr/local/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    474         f = self.dispatch.get(t)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return
    478

/usr/local/lib/python3.6/site-packages/dill/_dill.py in save_module_dict(pickler, obj)
    931             # we only care about session the first pass thru
    932             pickler._session = False
--> 933         StockPickler.save_dict(pickler, obj)
    934         log.info("# D2")
    935     return

/usr/local/lib/python3.6/pickle.py in save_dict(self, obj)
    819
    820         self.memoize(obj)
--> 821         self._batch_setitems(obj.items())
    822
    823     dispatch[dict] = save_dict

/usr/local/lib/python3.6/pickle.py in _batch_setitems(self, items)
    845                 for k, v in tmp:
    846                     save(k)
--> 847                     save(v)
    848                 write(SETITEMS)
    849             elif n:

/usr/local/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    494             reduce = getattr(obj, "__reduce_ex__", None)
    495             if reduce is not None:
--> 496                 rv = reduce(self.proto)
    497             else:
    498                 reduce = getattr(obj, "__reduce__", None)

TypeError: can't pickle Tagger objects
```

</details>

trainsformers: 2.10.0
datasets: 1.0.2
dill: 0.3.2
python: 3.6.8

OS: ubuntu 16.04 (Docker Image) on [Deep Learning VM](https://console.cloud.google.com/marketplace/details/click-to-deploy-images/deeplearning) (GCP)
GPU: Tesla P100 (CUDA 10)

> I have the same issue with `transformers/BertJapaneseTokenizer`.

It looks like it this tokenizer is not supported unfortunately.
This is because `t.word_tokenizer.mecab` is a `fugashi.fugashi.GenericTagger` which is not compatible with pickle nor dill.

We need objects passes to `map` to be picklable for our caching system to work properly.
Here it crashes because the caching system is not able to pickle the GenericTagger.

\> Maybe you can create an issue on [fugashi](https://github.com/polm/fugashi/issues) 's repo and ask to make `fugashi.fugashi.GenericTagger` compatible with pickle ?

What you can do in the meantime is use a picklable wrapper of the tokenizer:

```python
from transformers import BertJapaneseTokenizer, MecabTokenizer

class PicklableTokenizer(BertJapaneseTokenizer):

    def __getstate__(self):
        state = dict(self.__dict__)
        state["do_lower_case"] = self.word_tokenizer.do_lower_case
        state["never_split"] = self.word_tokenizer.never_split
        del state["word_tokenizer"]
        return state

    def __setstate__(self, state):
        do_lower_case = state.pop("do_lower_case")
        never_split = state.pop("never_split")
        self.__dict__ = state
        self.word_tokenizer = MecabTokenizer(
            do_lower_case=do_lower_case, never_split=never_split)
        )

t = PicklableTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-whole-word-masking")
encoded = train_ds.map(lambda examples: {'tokens': t.encode(examples['title'])}, batched=True)  # it works
```

We can also update the `BertJapaneseTokenizer` in `transformers` as you just shown @lhoestq to make it compatible with pickle. It will be faster than asking on fugashi 's repo and good for the other users of `transformers` as well.

I'm currently working on `transformers` I'll include it in the https://github.com/huggingface/transformers/pull/7141 PR and the next release of `transformers`.

Thank you for the rapid and polite response!

@lhoestq Thanks for the suggestion! I've passed the pickle phase, but another `ArrowInvalid` problem occored. I created another issue #687 .

@thomwolf Wow, really fast work. I'm looking forward to the next release 🤗

I load squad dataset. Then want to process data use following function with `Huggingface Transformers LongformerTokenizer`.

```
def convert_to_features(example):
    # Tokenize contexts and questions (as pairs of inputs)
    input_pairs = [example['question'], example['context']]
    encodings = tokenizer.encode_plus(input_pairs, pad_to_max_length=True, max_length=512)
    context_encodings = tokenizer.encode_plus(example['context'])

    # Compute start and end tokens for labels using Transformers's fast tokenizers alignement methodes.
    # this will give us the position of answer span in the context text
    start_idx, end_idx = get_correct_alignement(example['context'], example['answers'])
    start_positions_context = context_encodings.char_to_token(start_idx)
    end_positions_context = context_encodings.char_to_token(end_idx-1)

    # here we will compute the start and end position of the answer in the whole example
    # as the example is encoded like this <s> question</s></s> context</s>
    # and we know the postion of the answer in the context
    # we can just find out the index of the sep token and then add that to position + 1 (+1 because there are two sep tokens)
    # this will give us the position of the answer span in whole example
    sep_idx = encodings['input_ids'].index(tokenizer.sep_token_id)
    start_positions = start_positions_context + sep_idx + 1
    end_positions = end_positions_context + sep_idx + 1

    if end_positions > 512:
      start_positions, end_positions = 0, 0

    encodings.update({'start_positions': start_positions,
                      'end_positions': end_positions,
                      'attention_mask': encodings['attention_mask']})
    return encodings
```

Then I run `dataset.map(convert_to_features)`, it raise
```
In [59]: a.map(convert_to_features)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-59-c453b508761d> in <module>
----> 1 a.map(convert_to_features)

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1242                 fn_kwargs=fn_kwargs,
   1243                 new_fingerprint=new_fingerprint,
-> 1244                 update_data=update_data,
   1245             )
   1246         else:

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    151             "output_all_columns": self._output_all_columns,
    152         }
--> 153         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    154         if new_format["columns"] is not None:
    155             new_format["columns"] = list(set(new_format["columns"]) & set(out.column_names))

/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    156                         kwargs_for_fingerprint["fingerprint_name"] = fingerprint_name
    157                         kwargs[fingerprint_name] = update_fingerprint(
--> 158                             self._fingerprint, transform, kwargs_for_fingerprint
    159                         )
    160

/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py in update_fingerprint(fingerprint, transform, transform_args)
    103     for key in sorted(transform_args):
    104         hasher.update(key)
--> 105         hasher.update(transform_args[key])
    106     return hasher.hexdigest()
    107

/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py in update(self, value)
     55     def update(self, value):
     56         self.m.update(f"=={type(value)}==".encode("utf8"))
---> 57         self.m.update(self.hash(value).encode("utf-8"))
     58
     59     def hexdigest(self):

/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py in hash(cls, value)
     51             return cls.dispatch[type(value)](cls, value)
     52         else:
---> 53             return cls.hash_default(value)
     54
     55     def update(self, value):

/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py in hash_default(cls, value)
     44     @classmethod
     45     def hash_default(cls, value):
---> 46         return cls.hash_bytes(dumps(value))
     47
     48     @classmethod

/opt/conda/lib/python3.7/site-packages/datasets/utils/py_utils.py in dumps(obj)
    365     file = StringIO()
    366     with _no_cache_fields(obj):
--> 367         dump(obj, file)
    368     return file.getvalue()
    369

/opt/conda/lib/python3.7/site-packages/datasets/utils/py_utils.py in dump(obj, file)
    337 def dump(obj, file):
    338     """pickle an object to a file"""
--> 339     Pickler(file, recurse=True).dump(obj)
    340     return
    341

/opt/conda/lib/python3.7/site-packages/dill/_dill.py in dump(self, obj)
    444             raise PicklingError(msg)
    445         else:
--> 446             StockPickler.dump(self, obj)
    447         stack.clear()  # clear record of 'recursion-sensitive' pickled objects
    448         return

/opt/conda/lib/python3.7/pickle.py in dump(self, obj)
    435         if self.proto >= 4:
    436             self.framer.start_framing()
--> 437         self.save(obj)
    438         self.write(STOP)
    439         self.framer.end_framing()

/opt/conda/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506

/opt/conda/lib/python3.7/site-packages/dill/_dill.py in save_function(pickler, obj)
   1436                                 globs, obj.__name__,
   1437                                 obj.__defaults__, obj.__closure__,
-> 1438                                 obj.__dict__, fkwdefaults), obj=obj)
   1439         else:
   1440             _super = ('super' in getattr(obj.func_code,'co_names',())) and (_byref is not None) and getattr(pickler, '_recurse', False)

/opt/conda/lib/python3.7/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    636         else:
    637             save(func)
--> 638             save(args)
    639             write(REDUCE)
    640

/opt/conda/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506

/opt/conda/lib/python3.7/pickle.py in save_tuple(self, obj)
    787         write(MARK)
    788         for element in obj:
--> 789             save(element)
    790
    791         if id(obj) in memo:

/opt/conda/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506

/opt/conda/lib/python3.7/site-packages/dill/_dill.py in save_module_dict(pickler, obj)
    931             # we only care about session the first pass thru
    932             pickler._session = False
--> 933         StockPickler.save_dict(pickler, obj)
    934         log.info("# D2")
    935     return

/opt/conda/lib/python3.7/pickle.py in save_dict(self, obj)
    857
    858         self.memoize(obj)
--> 859         self._batch_setitems(obj.items())
    860
    861     dispatch[dict] = save_dict

/opt/conda/lib/python3.7/pickle.py in _batch_setitems(self, items)
    883                 for k, v in tmp:
    884                     save(k)
--> 885                     save(v)
    886                 write(SETITEMS)
    887             elif n:

/opt/conda/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    547
    548         # Save the reduce() output and finally memoize the object
--> 549         self.save_reduce(obj=obj, *rv)
    550
    551     def persistent_id(self, obj):

/opt/conda/lib/python3.7/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    660
    661         if state is not None:
--> 662             save(state)
    663             write(BUILD)
    664

/opt/conda/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506

/opt/conda/lib/python3.7/site-packages/dill/_dill.py in save_module_dict(pickler, obj)
    931             # we only care about session the first pass thru
    932             pickler._session = False
--> 933         StockPickler.save_dict(pickler, obj)
    934         log.info("# D2")
    935     return

/opt/conda/lib/python3.7/pickle.py in save_dict(self, obj)
    857
    858         self.memoize(obj)
--> 859         self._batch_setitems(obj.items())
    860
    861     dispatch[dict] = save_dict

/opt/conda/lib/python3.7/pickle.py in _batch_setitems(self, items)
    883                 for k, v in tmp:
    884                     save(k)
--> 885                     save(v)
    886                 write(SETITEMS)
    887             elif n:

/opt/conda/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    547
    548         # Save the reduce() output and finally memoize the object
--> 549         self.save_reduce(obj=obj, *rv)
    550
    551     def persistent_id(self, obj):

/opt/conda/lib/python3.7/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    660
    661         if state is not None:
--> 662             save(state)
    663             write(BUILD)
    664

/opt/conda/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506

/opt/conda/lib/python3.7/site-packages/dill/_dill.py in save_module_dict(pickler, obj)
    931             # we only care about session the first pass thru
    932             pickler._session = False
--> 933         StockPickler.save_dict(pickler, obj)
    934         log.info("# D2")
    935     return

/opt/conda/lib/python3.7/pickle.py in save_dict(self, obj)
    857
    858         self.memoize(obj)
--> 859         self._batch_setitems(obj.items())
    860
    861     dispatch[dict] = save_dict

/opt/conda/lib/python3.7/pickle.py in _batch_setitems(self, items)
    883                 for k, v in tmp:
    884                     save(k)
--> 885                     save(v)
    886                 write(SETITEMS)
    887             elif n:

/opt/conda/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    522             reduce = getattr(obj, "__reduce_ex__", None)
    523             if reduce is not None:
--> 524                 rv = reduce(self.proto)
    525             else:
    526                 reduce = getattr(obj, "__reduce__", None)

TypeError: can't pickle Tokenizer objects
```


load_dataset from local squad.py, raise error: TypeError: 'NoneType' object is not callable

Hi !
Thanks for reporting.
It looks like no object inherits from `datasets.GeneratorBasedBuilder` (or more generally from `datasets.DatasetBuilder`) in your script.

Could you check that there exist at least one dataset builder class ?

Hi @xixiaoyao did you manage to fix your issue ?

version: 1.0.2

```
train_dataset  = datasets.load_dataset('squad')
```

The above code can works. However, when I download the squad.py from your server, and saved as `my_squad.py` to local. I run followings raise errors.
```
train_dataset  = datasets.load_dataset('./my_squad.py')
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-25a84b4d1581> in <module>
----> 1 train_dataset  = nlp.load_dataset('./my_squad.py')

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    602         hash=hash,
    603         features=features,
--> 604         **config_kwargs,
    605     )
    606

TypeError: 'NoneType' object is not callable


Created dataset card snli.md

Dataset discussion

Discussions on the datasets

Adding a direct link to the rendered markdown:
https://github.com/mcmillanmajora/datasets/blob/add_dataset_documentation/datasets/snli/README.md

It would be amazing if we ended up with this much information on all of our datasets :)

I don't think there's too much repetition, everything that is in here is relevant. The main challenge will be to figure out how to structure the sheet so that all of the information can be presented without overwhelming the reader. We'll also want to have as much of it as possible in structured form so it can be easily navigated.

@mcmillanmajora for now can you remove the prompts / quoted blocks so we can see what the datasheet would look like on its own?

Would also love to hear if @sgugger has some first impressions

I removed the prompts. It's definitely a little easier to read without them!

Should we name the file `README.md` for consistency with models?

Asked @sleepinyourhat for some insights too :)

Thank you for taking the time to look through the card and for all your comments @sleepinyourhat ! I've incorporated them in the latest update.

Be careful to keep the ‘sa’ term in the license. It’s something we
inherited from the Flickr captions.

On Thu, Oct 1, 2020 at 10:09 AM Julien Chaumond <notifications@github.com>
wrote:

> *@julien-c* commented on this pull request.
> ------------------------------
>
> In datasets/snli/README.md
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663-23discussion-5Fr498273172&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=WbEkKXCbL6j5Ui3sox_WqvzrbShbJn2WW-51SENL2ZQ&e=>
> :
>
> > +---
> +language:
> +- en
> +task:
> +- text-classification
> +purpose:
> +- NLI
> +size:
> +- ">100k"
> +language producers:
> +- crowdsourced
> +annotation:
> +- crowdsourced
> +tags:
> +- extended-from-other-datasets
> +license: "CC BY-SA 4.0"
>
> ⬇ Suggested change
>
> -license: "CC BY-SA 4.0"
> +license: cc-by-4.0
>
> For models (documented at
> https://huggingface.co/docs#what-metadata-can-i-add-to-my-model-card
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__huggingface.co_docs-23what-2Dmetadata-2Dcan-2Di-2Dadd-2Dto-2Dmy-2Dmodel-2Dcard&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=ck3x8c_ujrwKReDTSGuWWgD9W6REHEPbZaO7S4GFRd4&e=>)
> we use the License keywords listed by GitHub at
> https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/licensing-a-repository#searching-github-by-license-type
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.github.com_en_free-2Dpro-2Dteam-40latest_github_creating-2Dcloning-2Dand-2Darchiving-2Drepositories_licensing-2Da-2Drepository-23searching-2Dgithub-2Dby-2Dlicense-2Dtype&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=dWBP-ZvtMErD-egoBiBTCKA4500mjDXVSk03oW1g16U&e=>
>
> (Hopefully we'll plug some sort of form validation for users at some point)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663-23pullrequestreview-2D500386385&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=HU2Hwi7HH9W2NtMoCIiQlhXxxEULLi8L9gnWU5PBAPY&e=>,
> or unsubscribe
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAJZSWL63W2LB7SBICA2GMTSISEPZANCNFSM4RWKAZRA&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=086__lKQLxTanHfjE8kOIpaJbaWPzBB9gGIt_prWeH8&e=>
> .
>

@sleepinyourhat You're right, wrong copy/paste

Question: Where does this standard come from? It looks similar to both
'Data Statements' and 'Datasheets for Datasets', but it doesn't look quite
like either.

On Mon, Oct 12, 2020 at 4:27 PM Yacine Jernite <notifications@github.com>
wrote:

> Merged #663
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=D34WbiHBTYHOdXsI9JV9wJqSieP6zAPGqGKDziM5uKU&s=s4_X-BSEnTKgGg9rPLBt3cyVptyMX_iWD5Ql3UMBi-I&e=>
> into master.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663-23event-2D3868180429&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=D34WbiHBTYHOdXsI9JV9wJqSieP6zAPGqGKDziM5uKU&s=elcM4umqReQfIrgHhpey9W_wPaq5QRgq7xNlubM47QI&e=>,
> or unsubscribe
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAJZSWJVGQRCR4OTTV27VTTSKNRBXANCNFSM4RWKAZRA&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=D34WbiHBTYHOdXsI9JV9wJqSieP6zAPGqGKDziM5uKU&s=NB6nEROnTPgwNyF3ZklOmHnvP7kOkOm7sEa740KbVCs&e=>
> .
>

@sleepinyourhat The schema is definitely drawing from Data Statements and Datasheets for Datasets but we also wanted to include some more general information to introduce the dataset to new users. If you have any suggestions for changes to the schema itself, please let us know!

First draft of a dataset card using the SNLI corpus as an example.

This is mostly based on the [Google Doc draft](https://docs.google.com/document/d/1dKPGP-dA2W0QoTRGfqQ5eBp0CeSsTy7g2yM8RseHtos/edit), but I added a few sections and moved some things around.

- I moved **Who Was Involved** to follow **Language**, both because I thought the authors should be presented more towards the front and because I think it makes sense to present the speakers close to the language so it doesn't have to be repeated.

- I created a section I called **Data Characteristics** by pulling some things out of the other sections. I was thinking that this would be more about the language use in context of the specific task construction. That name isn't very descriptive though and could probably be improved.
-- Domain and language type out of **Language**. I particularly wanted to keep the Language section as simple and as abstracted from the task as possible.
-- 'How was the data collected' out of **Who Was Involved**
-- Normalization out of **Features/Dataset Structure**
-- I also added an annotation process section.

- I kept the **Features** section mostly the same as the Google Doc, but I renamed it **Dataset Structure** to more clearly separate it from the language use, and added some links to the documentation pages.

- I also kept **Tasks Supported**,  **Known Limitations**, and **Licensing Information** mostly the same. Looking at it again though, maybe **Tasks Supported** should come before **Data Characteristics**?

The trickiest part about writing a dataset card for the SNLI corpus specifically is that it's built on datasets which are themselves built on datasets so I had to dig in a lot of places to find information. I think this will be easier with other datasets and once there is more uptake of dataset cards so they can just link to each other. (Maybe that needs to be an added section?)

I also made an effort not to repeat information across the sections or to refer to a previous section if the information was relevant in a later one. Is there too much repetition still?


Created dataset card snli.md

Discussions on the datasets

First draft of a dataset card using the SNLI corpus as an example


Replace pa.OSFile by open

It should fix #643


add openwebtext

> BTW, is there a one-line command to make our building scripts pass flake8 test? (included code quality test), I got like trailing space or mixed space and tab warning and error, and fixed them manually.

I don't think so.
We have a command for black and isort but not flake8 as far as I know.

Thanks for your awesome work too.
BTW a little reminder, this solves #132

This adds [The OpenWebText Corpus](https://skylion007.github.io/OpenWebTextCorpus/), which is a clean and large text corpus for nlp pretraining. It is an open source effort to reproduce OpenAI’s WebText dataset used by GPT-2, and it is also needed to reproduce ELECTRA.

It solves #132 .

### Besides dataset building script, I made some changes to the library.

1. Extract large amount of compressed files with multi processing
I add a `num_proc` argument to `DownloadManager.extract` and pass this `num_proc` to `map_nested`. So I can decompress 20 thousands compressed files faster. `num_proc` I add is default to `None`, so it shouldn't break any other thing.

2. In `cached_path`, I change the order to deal with different kind of compressed files (zip, tar, gzip)
Because there is no way to 100% detect a file is a zip file (see [this](https://stackoverflow.com/questions/18194688/how-can-i-determine-if-a-file-is-a-zip-file)), I found it wrongly detect `'./datasets/downloads/extracted/58764bd6898fa339b25d92e7fbbc3d8dbf64fb504edff1a30a1d7d99d1561027/openwebtext/urlsf_subset13-630_data.xz'` as a zip and try decompress it with zip, sure it will get error. So I made it detect wheter the file is tar or gzip first and detect zip in the last.

3. `MockDownloadManager.extract`
Cuz I pass `num_proc` to `DownloadManager.extract`, I also have to make `MockDownloadManager.extract` to accept extra keywork arguments. So I make it `extract(path, *args, **kwargs)`, but just return the path as original implementation.

**Note**: If there is better way for points mentioned above, thought I would like to help, unless we can solve point4 (make dataset building fast), I may not be able to afford rebuild the dataset again because of change of the dataset script (Building the dataset cost me 4 days).

### There is something I think we can improve

4. Long time to decompress compressed files
Even I decompress those 20 thousands compressed files with 12 process on my 16 core 3.x Ghz server. It still took about 3 ~ 4days to complete dataset building. Most of time spent on decompress those files.

### Info about the source data
The source data is an  tar.xz file with following structure, files/directory beyond compressed file is what can we get after decompress it.
```
openwebtext.tar.xz
  |__ openwebtext
         |__subset000.xz
         |     |__ ....txt
         |     |__ ....txt
         |     ...
         |__ subset001.xz
         |
         ....
```
And this the structure of dummy data, same as the original one.
```
dummy_data.zip
  |__ dummy_data
         |__ openwebtext
              |__fake_subset-1_data-dirxz # actually it is a directory
              |     |__ ....txt
              |     |__ ....txt
              |__ fake_subset-2_data-dirxz
                    |__ ....txt
                    |__ ....txt
```


Keep new columns in transmit format

When a dataset is formatted with a list of columns that `__getitem__` should return, then calling `map` to add new columns doesn't add the new columns to this list.

It caused `KeyError` issues in #620

I changed the logic to add those new columns to the list that `__getitem__` should return.


Fix squad metric's Features

Closing this one in favor of #670

Thanks again for reporting the issue and proposing this fix !
Let me know if you have other remarks

Resolves issue [657](https://github.com/huggingface/datasets/issues/657).


Squad Metric Description & Feature Mismatch

Thanks for reporting !
There indeed a mismatch between the features and the kwargs description

I believe `answer_start` was added to match the squad dataset format for consistency, even though it is not used in the metric computation. I think I'd rather keep it this way, so that you can just give `references=squad["answers"]` to `.compute()`.
Maybe we can just fix the description then.

But then providing the `answer_start` becomes mandatory since the format of the features is checked against the one provided in the squad [file](https://github.com/huggingface/datasets/pull/658/files).

The [description](https://github.com/huggingface/datasets/blob/master/metrics/squad/squad.py#L39) doesn't mention `answer_start` in squad. However the `datasets.features` require [it](https://github.com/huggingface/datasets/blob/master/metrics/squad/squad.py#L68). It's also not used in the evaluation.


Use multiprocess from pathos for multiprocessing

We can just install multiprocess actually, I'll change that

Just an FYI: I remember that I wanted to try pathos a couple of years back and I ran into issues considering cross-platform; the code would just break on Windows. If I can verify this PR by running CPU tests on Windows, let me know!

That's good to know thanks
I guess we can just wait for #644 to be merged first. I'm working on fixing the tests for windows

Looks like all the CI jobs on windows passed !
I also tested locally on my windows and it works great :)

I think this is ready to merge, let me know if you have any remarks @thomwolf @BramVanroy

[Multiprocess](https://github.com/uqfoundation/multiprocess) (from the [pathos](https://github.com/uqfoundation/pathos) project) allows to use lambda functions in multiprocessed map.
It was suggested to use it by @kandorm.

We're already using dill which is its only dependency.


added Winogrande debiased subset

To fix the CI you just have to copy the dummy data to the 1.1.0 folder, and maybe create the dummy ones for the `debiased` configuration

The [Winogrande](https://arxiv.org/abs/1907.10641) paper mentions a `debiased` subset that wasn't in the first release; this PR adds it.


Allow empty inputs in metrics

There was an arrow error when trying to compute a metric with empty inputs. The error was occurring when reading the arrow file, before calling metric._compute.


handle data alteration when trying type

Fix #649

The bug came from the type inference that didn't handle a weird case in Pyarrow.
Indeed this code runs without error but alters the data in arrow:
```python
import pyarrow as pa

type = pa.struct({"a": pa.struct({"b": pa.string()})})
array_with_altered_data = pa.array([{"a": {"b": "foo", "c": "bar"}}] * 10, type=type)
print(array_with_altered_data[0].as_py())
# {'a': {'b': 'foo'}} -> the sub-field "c" is missing
```
(I don't know if this is intended in pyarrow tbh)

We didn't take this case into account during type inference. By default it was keeping old features and maybe alter data.
To fix that I added a line that checks that the first element of the array is not altered.


handle connection error in download_prepared_from_hf_gcs

Fix #647


Problem with JSON dataset format

Currently the `json` dataset doesn't support this format unfortunately.
However you could load it with
```python
from datasets import Dataset
import pandas as pd

df = pd.read_json("path_to_local.json", orient="index")
dataset = Dataset.from_pandas(df)
```

or you can make a custom dataset script as explained in doc here: https://huggingface.co/docs/datasets/add_dataset.html

I have a local json dataset with the following form.

{
    'id01234': {'key1': value1, 'key2': value2, 'key3': value3},
    'id01235': {'key1': value1, 'key2': value2, 'key3': value3},
    .
    .
    .
    'id09999': {'key1': value1, 'key2': value2, 'key3': value3}
}
Note that instead of a list of records it's basically a dictionary of key value pairs with the keys being the record_ids and the values being the corresponding record.

Reading this with json:

```
data = datasets.load('json', data_files='path_to_local.json')
```
Throws an error and asks me to chose a field. What's the right way to handle this?


dummy data testing can't test datasets using `dl_manager.extract` in `_split_generators`

Hi :)
In your dummy data zip file you can just have `subset000.xz` as directories instead of compressed files.
Let me know if it helps

Thanks for your comment @lhoestq ,
Just for confirmation, changing dummy data like this won't make dummy test test the functionality to extract `subsetxxx.xz` but actually kind of circumvent it. But since we will test the real data so it is ok ?

Yes it's fine for now. We plan to add a job for slow tests.
And at one point we'll also do another pass on the dummy data handling and consider extracting files.

Thanks for the confirmation.
Also the suggestion works. Thank you.

Hi, I recently want to add a dataset whose source data is like this
```
openwebtext.tar.xz
  |__ openwebtext
         |__subset000.xz
         |     |__ ....txt
         |     |__ ....txt
         |     ...
         |__ subset001.xz
         |
         ....
```
So I wrote `openwebtext.py` like this
```
 def _split_generators(self, dl_manager):
        dl_dir = dl_manager.download_and_extract(_URL)
        owt_dir = os.path.join(dl_dir, 'openwebtext')
        subset_xzs = [
            os.path.join(owt_dir, file_name) for file_name in os.listdir(owt_dir) if file_name.endswith('xz') # filter out ...xz.lock
        ]
        ex_dirs = dl_manager.extract(subset_xzs, num_proc=round(os.cpu_count()*0.75))
        nested_txt_files = [
          [
            os.path.join(ex_dir,txt_file_name) for txt_file_name in os.listdir(ex_dir) if txt_file_name.endswith('txt')
          ] for ex_dir in ex_dirs
        ]
        txt_files = chain(*nested_txt_files)
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"txt_files": txt_files}
            ),
        ]
```
All went good, I can load and use real openwebtext, except when I try to test with dummy data. The problem is  `MockDownloadManager.extract` do nothing, so `ex_dirs = dl_manager.extract(subset_xzs)` won't decompress `subset_xxx.xz`s for me.

How should I do ? Or you can modify `MockDownloadManager` to make it like a real `DownloadManager` ?


Inconsistent behavior in map

bug

Something isn't working

Thanks for reporting !

This issue must have appeared when we refactored type inference in `nlp`
By default the library tries to keep the same feature types when applying `map` but apparently it has troubles with nested structures. I'll try to fix that next week

I'm observing inconsistent behavior when applying .map(). This happens specifically when I'm incrementally adding onto a feature that is a nested dictionary. Here's a simple example that reproduces the problem.

```python
import datasets

# Dataset with a single feature called 'field' consisting of two examples
dataset = datasets.Dataset.from_dict({'field': ['a', 'b']})
print(dataset[0])
# outputs
{'field': 'a'}

# Map this dataset to create another feature called 'otherfield', which is a dictionary containing a key called 'capital'
dataset = dataset.map(lambda example: {'otherfield': {'capital': example['field'].capitalize()}})
print(dataset[0])
# output is okay
{'field': 'a', 'otherfield': {'capital': 'A'}}

# Now I want to map again to modify 'otherfield', by adding another key called 'append_x' to the dictionary under 'otherfield'
print(dataset.map(lambda example: {'otherfield': {'append_x': example['field'] + 'x'}})[0])
# printing out the first example after applying the map shows that the new key 'append_x' doesn't get added
# it also messes up the value stored at 'capital'
{'field': 'a', 'otherfield': {'capital': None}}

# Instead, I try to do the same thing by using a different mapped fn
print(dataset.map(lambda example:  {'otherfield': {'append_x': example['field'] + 'x', 'capital': example['otherfield']['capital']}})[0])
# this preserves the value under capital, but still no 'append_x'
{'field': 'a', 'otherfield': {'capital': 'A'}}

# Instead, I try to pass 'otherfield' to remove_columns
print(dataset.map(lambda example:  {'otherfield': {'append_x': example['field'] + 'x', 'capital': example['otherfield']['capital']}}, remove_columns=['otherfield'])[0])
# this still doesn't fix the problem
{'field': 'a', 'otherfield': {'capital': 'A'}}

# Alternately, here's what happens if I just directly map both 'capital' and 'append_x' on a fresh dataset.

# Recreate the dataset
dataset = datasets.Dataset.from_dict({'field': ['a', 'b']})
# Now map the entire 'otherfield' dict directly, instead of incrementally as before
print(dataset.map(lambda example:  {'otherfield': {'append_x': example['field'] + 'x', 'capital': example['field'].capitalize()}})[0])
# This looks good!
{'field': 'a', 'otherfield': {'append_x': 'ax', 'capital': 'A'}}
```

This might be a new issue, because I didn't see this behavior in the `nlp` library.

Any help is appreciated!


offset overflow when multiprocessing batched map on large datasets.

Something isn't working

This should be fixed with #645
Feel free to re-open if it still occurs

It only happened when "multiprocessing" + "batched" + "large dataset" at the same time.

```
def bprocess(examples):
  examples['len'] = []
  for text in examples['text']:
    examples['len'].append(len(text))
  return examples
wiki.map(brpocess, batched=True, num_proc=8)
```
```
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/yisiang/miniconda3/envs/ml/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/yisiang/datasets/src/datasets/arrow_dataset.py", line 153, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/yisiang/datasets/src/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/yisiang/datasets/src/datasets/arrow_dataset.py", line 1486, in _map_single
    batch = self[i : i + batch_size]
  File "/home/yisiang/datasets/src/datasets/arrow_dataset.py", line 1071, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/home/yisiang/datasets/src/datasets/arrow_dataset.py", line 972, in _getitem
    data_subset = self._data.take(indices_array)
  File "pyarrow/table.pxi", line 1145, in pyarrow.lib.Table.take
  File "/home/yisiang/miniconda3/envs/ml/lib/python3.7/site-packages/pyarrow/compute.py", line 268, in take
    return call_function('take', [data, indices], options)
  File "pyarrow/_compute.pyx", line 298, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 192, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
"""

The above exception was the direct cause of the following exception:

ArrowInvalid                              Traceback (most recent call last)
 in
     30   owt = datasets.load_dataset('/home/yisiang/datasets/datasets/openwebtext/openwebtext.py', cache_dir='./datasets')['train']
     31   print('load/create data from OpenWebText Corpus for ELECTRA')
---> 32   e_owt = ELECTRAProcessor(owt, apply_cleaning=False).map(cache_file_name=f"electra_owt_{c.max_length}.arrow")
     33   dsets.append(e_owt)
     34

~/Reexamine_Attention/electra_pytorch/_utils/utils.py in map(self, **kwargs)
    126       writer_batch_size=10**4,
    127       num_proc=num_proc,
--> 128       **kwargs
    129     )
    130

~/hugdatafast/hugdatafast/transform.py in my_map(self, *args, **kwargs)
     21     if not cache_file_name.endswith('.arrow'): cache_file_name += '.arrow'
     22     if '/' not in cache_file_name: cache_file_name = os.path.join(self.cache_directory(), cache_file_name)
---> 23   return self.map(*args, cache_file_name=cache_file_name, **kwargs)
     24
     25 @patch

~/datasets/src/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1285                 logger.info("Spawning {} processes".format(num_proc))
   1286                 results = [pool.apply_async(self.__class__._map_single, kwds=kwds) for kwds in kwds_per_shard]
-> 1287                 transformed_shards = [r.get() for r in results]
   1288                 logger.info("Concatenating {} shards from multiprocessing".format(num_proc))
   1289                 result = concatenate_datasets(transformed_shards)

~/datasets/src/datasets/arrow_dataset.py in (.0)
   1285                 logger.info("Spawning {} processes".format(num_proc))
   1286                 results = [pool.apply_async(self.__class__._map_single, kwds=kwds) for kwds in kwds_per_shard]
-> 1287                 transformed_shards = [r.get() for r in results]
   1288                 logger.info("Concatenating {} shards from multiprocessing".format(num_proc))
   1289                 result = concatenate_datasets(transformed_shards)

~/miniconda3/envs/ml/lib/python3.7/multiprocessing/pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658
    659     def _set(self, i, obj):

ArrowInvalid: offset overflow while concatenating arrays
```


Cannot download dataset_info.json

Thanks for reporting !
We should add support for servers without internet connection indeed
I'll do that early next week

Thanks, @lhoestq !
Please let me know when it is available.

Right now the recommended way is to create the dataset on a server with internet connection and then to save it and copy the serialized dataset to the server without internet connection.

#652 should allow you to load text/json/csv/pandas datasets without an internet connection **IF** you've the dataset script locally.

Example:
If you have `datasets/text/text.py` locally, then you can do `load_dataset("./datasets/text", data_files=...)`

I am running my job on a cloud server where does not provide for connections from the standard compute nodes to outside resources. Hence, when I use `dataset.load_dataset()` to load data, I got an error like this:

```
ConnectionError: Couldn't reach https://storage.googleapis.com/huggingface-nlp/cache/datasets/text/default-53ee3045f07ba8ca/0.0.0/dataset_info.json
```

I tried to open this link manually, but I cannot access this file. How can I download this file and pass it through `dataset.load_dataset()` manually?

Versions:
Python version 3.7.3
PyTorch version 1.6.0
TensorFlow version 2.3.0
datasets version: 1.0.1


Fix docs typos

This PR fixes few typos in the docs and the error in the code snippet in the set_format section in docs/source/torch_tensorflow.rst. `torch.utils.data.Dataloader` expects padded batches so it throws an error due to not being able to stack the unpadded tensors. If we follow the Quick tour from the docs where they add the `truncation=True, padding='max_length'` arguments to the tokenizer before passing data to Dataloader, we can easily fix the issue.


Don't use take on dataset table in pyarrow 1.0.x

I tried lower batch sizes and it didn't accelerate filter (quite the opposite actually).
The slow-down also appears for pyarrow 0.17.1 for some reason, not sure it comes from these changes

I just checked the benchmarks of other PRs and some of them had 300s (!!) for filter. This needs some investigation..

Merging this one since it's not the cause of the the slow down

Fix #615


Better windows support

This PR is ready :)
It brings official support for windows.

Some tests `AWSDatasetTest` are failing.
This is because I had to fix a few datasets that were not compatible with windows.
These test will pass once they got merged on master :)

There are a few differences in the behavior of python and pyarrow on windows.

For example there are restrictions when accessing/deleting files that are open

Fix #590


Caching processed dataset at wrong folder

Something isn't working

Thanks for reporting !
It uses a temporary file to write the data.
However it looks like the temporary file is not placed in the right directory during the processing

Well actually I just tested and the temporary file is placed in the same directory, so it should work as expected.
Which version of `datasets` are you using ?

`datasets-1.0.1`
Here you can reproduce it here:
https://colab.research.google.com/drive/1O0KcepTFsmpkBbrbLLMq42iwTKmQh8d5?usp=sharing

It looks like a pyarrow issue with google colab.
For some reason this code increases the disk usage of google colab while it actually writes into google drive:

```python
import pyarrow as pa

stream = pa.OSFile("/content/drive/My Drive/path/to/file.arrow", "wb")
writer = pa.RecordBatchStreamWriter(stream, schema=pa.schema({"text": pa.string()}))
writer.write_table(pa.Table.from_pydict({"text": ["a"*511 + "\n"] * ((1 << 30) // 512)}))  # 1GiB
writer.close()
stream.close()
```

Moreover if I `rm` the file on google drive, it frees disk space on google colab.

It looks like replacing `pa.OSFile` by `open` fixes it, I'm going to open a PR

Actually I did more tests it doesn't >.<
I'll let you know if I find a way to fix that

Actually I also have the issue when writing a regular text file

```python
f = open("/content/drive/My Drive/path/to/file", "w")
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.close()
```

Is that supposed to happen ?

The code you wrote should write a 1GB file in the Google Drive folder. Doesn't it?

Yes it does, but the disk usage of google colab also increases by 1GB

I could check it and as you say as I write to te Drive disk the colab disk also increases...

To reproduce it:
```bash
!df -h | grep sda1
```
```python
f = open("/content/drive/My Drive/test_to_remove.txt", "w")
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.close()
```
```bash
!ls -lh /content/drive/My\ Drive/test_to_remove.txt

!df -h | grep sda1

!rm -rf /content/drive/My\ Drive/test_to_remove.txt

```
[Colab](https://colab.research.google.com/drive/1D0UiweCYQwwWZ65EEhuqqbaDDbhJYXfm?usp=sharing)

Hi guys, I run this on my Colab (PRO):

```python
from datasets import load_dataset
dataset = load_dataset('text', data_files='/content/corpus.txt', cache_dir='/content/drive/My Drive', split='train')

def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')

dataset = dataset.map(encode, batched=True)
```
The file is about 4 GB, so I cannot process it on the Colab HD because there is no enough space. So I decided to mount my Google Drive fs and do it on it.
The dataset is cached in the right place but by processing it (applying `encode` function) seems to use a different folder because Colab HD starts to grow and it crashes when it should be done in the Drive fs.

What gets me crazy, it prints it is processing/encoding the dataset in the right folder:
```
Testing the mapped function outputs
Testing finished, running the mapping function on the dataset
Caching processed dataset at /content/drive/My Drive/text/default-ad3e69d6242ee916/0.0.0/7e13bc0fa76783d4ef197f079dc8acfe54c3efda980f2c9adfab046ede2f0ff7/cache-b16341780a59747d.arrow
```


Rename wnut fields

As mentioned in #641 it would be cool to have it follow the naming of the other NER datasets


Add Polyglot-NER Dataset

Hi @joeddav thanks for adding this! (I did a long webarchive.org session to actually find that dataset a while ago).

One question: should we manually correct the labeling scheme to (at least) IOB1?

That means "LOC" will be converted to "I-LOC". IOB1 is not explict. mentioned in the paper, but it is used in the documentation:

https://polyglot.readthedocs.io/en/latest/NamedEntityRecognition.html

@stefan-it I went back and forth on this. My biggest problem with it is that once you are in IOB, there is the expectation that the beginning of new entities are marked with a `B-` (at least in the case of two back-to-back entities):
```
Today    O
Alice    I-PER
Bob      B-PER
and      O
I        O
ate      O
lasagna  O
```
If we just prepend `I-` to everything, `Bob` would be incorrectly tagged `I-PER`, meaning `Bob Alice` is a single entity. The current format is bad but is at least clear that it does not contain that information.

But I could go either way if someone has a strong opinion.

Indeed I'm not sure we can convert them to IOB because of this issue. I'm fine with keeping it like that

I'll do a release later today, hopefully we can include this dataset in the release :)

Let me know if you need help with the dummy data

@lhoestq cool thanks, I think I've got it right now – just zipped them wrong. I'm running tests locally now and then will push.

@joeddav I'm fine with keeping the original labeling scheme :)

Adds the [Polyglot-NER dataset](https://sites.google.com/site/rmyeid/projects/polylgot-ner) with named entity tags for 40 languages. I include separate configs for each language as well as a `combined` config which lumps them all together.


Make shuffle compatible with temp_seed

This code used to return different dataset at each run
```python
import dataset as ds

dataset = ...

with ds.temp_seed(42):
    shuffled = dataset.shuffle()
```

Now it returns the same one since the seed is set


Update glue QQP checksum

Fix #638


GLUE/QQP dataset: NonMatchingChecksumError

Hi @lhoestq , I know you are busy and there are also other important issues. But if this is easy to be fixed, I am shamelessly wondering if you can give me some help , so I can evaluate my models and restart with my developing cycle asap. 😚

datasets version: editable install of master at 9/17

`datasets.load_dataset('glue','qqp', cache_dir='./datasets')`

```
Downloading and preparing dataset glue/qqp (download: 57.73 MiB, generated: 107.02 MiB, post-processed: Unknown size, total: 164.75 MiB) to ./datasets/glue/qqp/1.0.0/7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4...
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
 in
----> 1 datasets.load_dataset('glue','qqp', cache_dir='./datasets')

~/datasets/src/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    609         download_config=download_config,
    610         download_mode=download_mode,
--> 611         ignore_verifications=ignore_verifications,
    612     )
    613

~/datasets/src/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    467                     if not downloaded_from_gcs:
    468                         self._download_and_prepare(
--> 469                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    470                         )
    471                     # Sync info

~/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    527         if verify_infos:
    528             verify_checksums(
--> 529                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    530             )
    531

~/datasets/src/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     37     if len(bad_urls) > 0:
     38         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 39         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     40     logger.info("All the checksums matched successfully" + for_verification_name)
     41

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://dl.fbaipublicfiles.com/glue/data/QQP-clean.zip']
```


Add MATINF


Consistent ner features

As discussed in #613 , this PR aims at making NER feature names consistent across datasets.

I changed the feature names of LinCE and XTREME/PAN-X


Loglevel

I think it's ready now @stas00, did you want to add something else ?
This PR includes your changes but with the level set to warning

Continuation of #618


Add ConLL-2000 dataset

Adds ConLL-2000 dataset used for text chunking. See https://www.clips.uantwerpen.be/conll2000/chunking/ for details and [motivation](https://github.com/huggingface/transformers/pull/7041#issuecomment-692710948) behind this PR


Load large text file for LM pre-training resulting in OOM

Not sure what could cause that on the `datasets` side. Could this be a `Trainer` issue ? cc @julien-c @sgugger ?

There was a memory leak issue fixed recently in master. You should install from source and see if it fixes your problem.

@lhoestq @sgugger Thanks for your comments. I have install from source code as you told, but the problem is still there.
To reproduce the issue, just replace [these lines](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L241-L258) with:
(load_dataset and DataCollatorForDatasetsLanguageModeling as [above mentioned](https://github.com/huggingface/datasets/issues/633#issue-702440484))
```python
    dataset = load_dataset("bookcorpus")
    dataset = dataset.train_test_split(test_size=0.1)
    train_dataset = dataset['train']
    eval_dataset = dataset['test'] if training_args.do_eval else None

    data_collator = DataCollatorForDatasetsLanguageModeling(
        tokenizer=tokenizer,
        mlm=data_args.mlm,
        mlm_probability=data_args.mlm_probability,
        block_size=data_args.block_size
    )
```
and run by:
```bash
python run_language_modeling.py
--output_dir=output \
--model_type=bert \
--model_name_or_path=bert-base-uncased \
--do_train \
--do_eval \
--mlm
```

Same here. Pre-training on wikitext-103 to do some test. At the end of the training it takes 32GB of RAM + ~30GB of SWAP. I installed dataset==1.1.0, not built from source. I will try uninstalling and building from source when it finish.

This seems to be on the `transformers` library side.

If you have more informations (pip env) or even better, a colab reproducing the error we can investigate.

It seems like it's solved with freshed versions of transformers. I have tried to replicate the error doing a fresh pip install transformers & datasets on colab and the error doesn't continue. On colab it keeps stable on 5GB! (Y)

Edit: **Thanks for your great work**. Have a good day.

@gaceladri witch version transformers and datasets  are you using now? I want to try again. Thanks.

doing some modifications to mobilebert
https://colab.research.google.com/drive/1ba09ZOpyHGAOQLcsxiQAHRXl10qnMU5o?usp=sharing

It does not happen to me anymore. Can we close? @leethu2012

It's happening to me again. After 4 hours of pre-training, my ram memory gets full and the kernel dies. I am using the last transformers version as today. 4.4.0 and the last version of datasets 1.2.1, both installed from master. The memory consumption keeps increasing.

It looks like it is something from pytorch/python itself :face_with_head_bandage:  https://github.com/pytorch/pytorch/issues/13246

Thanks for the investigation @gaceladri

Apparently this happens when `num_workers>0` and has to do with objects being copied-on-write.
Did you try setting num_workers to 0 @gaceladri ?
If the issue doesn't happen with `num_workers=0` then this would confirm that it's indeed related to this python/pytorch issue.

Since a `Dataset` object is a wrapper of a pyarrow Table, we should investigate if the data being copied comes from the Table itself or from metadata in the `Dataset` object. If it comes from the metadata in the `Dataset` object, we should be able to implement a workaround. But if it comes from the Table, we'll need to see with the pyarrow team what we can do...

@lhoestq I have tried and it keeps increasing also with `dataloader_num_workers=0`

Hmmm so this might come from another issue...
Since it doesn't seem to be related to multiprocessing it should be easier to investigate though.
Do you have some ideas @gaceladri ?

@lhoestq I looked quickly to a previously spoted bug in my env wandb /sdk/interface/interface.py, because sometimes when I load the dataset I got a multiprocessing error at line 510 in wandb...interface.py

This bug is reported here https://github.com/huggingface/datasets/issues/847

```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<timed eval> in <module>

~/anaconda3/envs/tfm/lib/python3.6/site-packages/transformers/trainer.py in train(self, model_path, trial)
    877             print(len(epoch_iterator))
    878
--> 879             for step, inputs in enumerate(epoch_iterator):
    880
    881                 start_step = time.time()

~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    433         if self._sampler_iter is None:
    434             self._reset()
--> 435         data = self._next_data()
    436         self._num_yielded += 1
    437         if self._dataset_kind == _DatasetKind.Iterable and \

~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _next_data(self)
   1083             else:
   1084                 del self._task_info[idx]
-> 1085                 return self._process_data(data)
   1086
   1087     def _try_put_index(self):

~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_data(self, data)
   1109         self._try_put_index()
   1110         if isinstance(data, ExceptionWrapper):
-> 1111             data.reraise()
   1112         return data
   1113

~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/_utils.py in reraise(self)
    426             # have message field
    427             raise self.exc_type(message=msg)
--> 428         raise self.exc_type(msg)
    429
    430

AssertionError: Caught AssertionError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1083, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1070, in _getitem
    format_kwargs=format_kwargs,
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 886, in _convert_outputs
    v = map_nested(command, v, **map_nested_kwargs)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 216, in map_nested
    return function(data_struct)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 847, in command
    return torch.tensor(x, **format_kwargs)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/warnings.py", line 101, in _showwarnmsg
    _showwarnmsg_impl(msg)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/warnings.py", line 30, in _showwarnmsg_impl
    file.write(text)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/lib/redirect.py", line 100, in new_write
    cb(name, data)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/wandb_run.py", line 729, in _console_callback
    self._backend.interface.publish_output(name, data)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 186, in publish_output
    self._publish_output(o)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 191, in _publish_output
    self._publish(rec)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 510, in _publish
    if self._process and not self._process.is_alive():
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/multiprocessing/process.py", line 134, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
```

My workaround was to just comment those lines without looking to much into consecuences:

```
def _publish(self, record: pb.Record, local: bool = None) -> None:
        #if self._process and not self._process.is_alive():
        #    raise Exception("The wandb backend process has shutdown")
```

It worked so far... I need to try running without wandb and see if it could be causing something wrong with multiprocessing. I am going to try to launch the training setting wandb to false and I will let you know again.

@lhoestq But despite this, I got lost into the [class Dataset()](https://huggingface.co/docs/datasets/_modules/datasets/arrow_dataset.html#Dataset) reading the pyarrow files.

Edit: but you should be rigth, that it does not have to be related to multiprocessing since it keeps happening when `num_workers=0`

Or maybe wandb uses multiprocessing ? One process for wandb logging and one for actual training ? If this is the case then even setting `num_workers=0` would cause the process to be forked for wandb and therefore cause the memory issue.

@lhoestq could be, but if we set wandb to false this should not happen. I am going to try.

@lhoestq It keeps happening. I have uninstalled wandb from my env, setted `%env WANDB_DISABLED=true` on my notebook, and commented this func:

```
def get_available_reporting_integrations():
    integrations = []
    if is_azureml_available():
        integrations.append("azure_ml")
    if is_comet_available():
        integrations.append("comet_ml")
    if is_mlflow_available():
        integrations.append("mlflow")
    if is_tensorboard_available():
        integrations.append("tensorboard")
    # if is_wandb_available():
    #     integrations.append("wandb")
    return integrations
```
As a fast test and it keeps increasing the ram memory. Wandb could not be the blameworthy here.

Thanks for checking @gaceladri . Let's investigate the single process setting then.
If you have some sort of colab notebook with a minimal code example that shows this behavior feel free to share it @gaceladri so that we can play around with it to find what causes this. Otherwise I'll probably try to reproduce on my side at one point

@lhoestq sure. Here you have https://colab.research.google.com/drive/1ba09ZOpyHGAOQLcsxiQAHRXl10qnMU5o?usp=sharing let me know if the link works and it reproduces the issue. To me, it reproduces the issue, since if you start the training the ram memory keeps increasing.

Let me know. Thanks!

Could the bug be comming from tokenizers?

I got this warning at the terminal from my jupyter notebook:
```
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
```

I've never experienced memory issues with tokenizers so I don't know
Cc @n1t0 are you aware of any issue that would cause memory to keep increasing when the tokenizer is used in the Data Collator for language modeling ?

@lhoestq Thanks for pointing to n1t0, just to clarify. That warning was doing fine-tuning, without collator:
```

from datasets import load_dataset, load_metric
import numpy as np

GLUE_TASKS = [
    "cola",
    "mnli",
    "mnli-mm",
    "mrpc",
    "qnli",
    "qqp",
    "rte",
    "sst2",
    "stsb",
    "wnli",
]
task = "mnli"
actual_task = "mnli" if task == "mnli-mm" else task
dataset = load_dataset("glue", actual_task)
metric = load_metric("glue", actual_task)
batch_size = 16
attention_type = "linear"

from transformers.models.mobilebert_mod import (
    MobileBertForSequenceClassification,
    MobileBertTokenizerFast,
)
from transformers.models.mobilebert_mod.configuration_mobilebert import (
    MobileBertConfigMod,
)
from transformers import TrainingArguments, Trainer

num_labels = 3 if task.startswith("mnli") else 1 if task == "stsb" else 2
tokenizer = MobileBertTokenizerFast.from_pretrained(
    "/media/ad/00b5422b-9d54-4449-8b5d-08eab5cdac8c/training_trfm/big_linear_layerdrop_shared/checkpoint-23000/",
    max_len=512,
)
model = MobileBertForSequenceClassification.from_pretrained(
    "/media/ad/00b5422b-9d54-4449-8b5d-08eab5cdac8c/training_trfm/big_linear_layerdrop_shared/checkpoint-23000/",
    num_labels=num_labels,
)
print(model.num_parameters())

task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mnli-mm": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

sentence1_key, sentence2_key = task_to_keys[task]
if sentence2_key is None:
    print(f"Sentence: {dataset['train'][0][sentence1_key]}")
else:
    print(f"Sentence 1: {dataset['train'][0][sentence1_key]}")
    print(f"Sentence 2: {dataset['train'][0][sentence2_key]}")

def preprocess_function(examples):
    if sentence2_key is None:
        return tokenizer(examples[sentence1_key], truncation=True)
    return tokenizer(examples[sentence1_key], examples[sentence2_key], truncation=True)

encoded_dataset = dataset.map(preprocess_function, batched=True)
metric_name = (
    "pearson"
    if task == "stsb"
    else "matthews_correlation"
    if task == "cola"
    else "accuracy"
)

args = TrainingArguments(
    f"test-glue/{task}_{attention_type}",
    evaluation_strategy="steps",
    learning_rate=1e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    logging_steps=200,
    num_train_epochs=5,
    gradient_accumulation_steps=1,
    warmup_steps=10000,
    fp16=True,
    dataloader_num_workers=10,
    weight_decay=0.1,
    load_best_model_at_end=True,
    metric_for_best_model=metric_name,
)

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    if task != "stsb":
        predictions = np.argmax(predictions, axis=1)
    else:
        predictions = predictions[:, 0]
    return metric.compute(predictions=predictions, references=labels)

validation_key = (
    "validation_mismatched"
    if task == "mnli-mm"
    else "validation_matched"
    if task == "mnli"
    else "validation"
)

trainer = Trainer(
    model,
    args,
    train_dataset=encoded_dataset["train"],
    eval_dataset=encoded_dataset[validation_key],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

trainer.train()
```

Now, I have come back to pre-training. The changes that I think I have done are: not formatting the dataset to torch: ~~`big_dataset.set_format(type='torch', columns=["text", "input_ids", "attention_mask", "token_type_ids"])`~~ so maybe some column is dropped and not freezed in memory and now I have not setted any validation dataset in the trainer.

My validation dataset before:
```
book_corpus_eval = load_dataset(
    "bookcorpus",
    "plain_text",
    cache_dir="/home/ad/Desktop/bookcorpus",
    split="train[98:99%]",
)
book_corpus_eval = book_corpus_eval.map(encode, batched=True)
book_corpus_eval.set_format(
    type="torch", columns=["text", "input_ids", "attention_mask", "token_type_ids"]
)
**book_corpus_eval = book_corpus_eval.select([i for i in range(1500)])**
```
Maybe _selecting_ or indexing the dataset before feeding it to the trainer, do something strange.

My trainer now:
```

big_dataset = load_from_disk("/home/ad/Desktop/35percent_data.arrow/")

from transformers import DataCollatorForWholeWordMask

data_collator = DataCollatorForWholeWordMask(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./big_linear_layerdrop_shared_silu_secondtry",
    overwrite_output_dir=True,
    per_device_train_batch_size=60,
    per_device_eval_batch_size=60,
    save_steps=500,
    save_total_limit=10,
    logging_first_step=True,
    logging_steps=100,
#     evaluation_strategy='steps',
#     eval_steps=250,
    gradient_accumulation_steps=8,
    fp16=True,
    dataloader_num_workers=10,
    warmup_steps=15000,
    learning_rate=6e-4,
    adam_epsilon=1e-6,
    adam_beta2=0.98,
    weight_decay=0.01,
    max_grad_norm=1.0,
    max_steps=500000,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=big_dataset,
#     eval_dataset=book_corpus_eval,
    tokenizer=tokenizer)

import wandb
wandb.login()

trainer.train()
```

And surprisingly, the ram now keeps going up and down. The training is up now for 12h without collapse the ram. I don't know what could cause the leakage. :mag:

Edit: I didn't see the swap memory, that keeps increasing. So the problem persist.

Thanks for sharing your results.
So you still had the issue for fine-tuning ?
And the issue still appears with a bare-bone dataset from an arrow file...

Yes, on both cases. Fine-tuning a pre-trained model and pre-training from scratch with a local arrow file already pre-processed.

I tried to pretrain Longformer using transformers and datasets. But I got OOM issues with loading a large text file. My script is almost like this:

```python
from datasets import load_dataset

@dataclass
class DataCollatorForDatasetsLanguageModeling(DataCollatorForLanguageModeling):
    """
    Data collator used for language modeling based on DataCollatorForLazyLanguageModeling
    - collates batches of tensors, honoring their tokenizer's pad_token
    - preprocesses batches for masked language modeling
    """

    block_size: int = 512

    def __call__(self, examples: List[dict]) -> Dict[str, torch.Tensor]:
        examples = [example['text'] for example in examples]
        batch, attention_mask = self._tensorize_batch(examples)
        if self.mlm:
            inputs, labels = self.mask_tokens(batch)
            return {"input_ids": inputs, "labels": labels}
        else:
            labels = batch.clone().detach()
            if self.tokenizer.pad_token_id is not None:
                labels[labels == self.tokenizer.pad_token_id] = -100
            return {"input_ids": batch, "labels": labels}

    def _tensorize_batch(self, examples: List[str]) -> Tuple[torch.Tensor, torch.Tensor]:

        if self.tokenizer._pad_token is None:
            raise ValueError(
                "You are attempting to pad samples but the tokenizer you are using"
                f" ({self.tokenizer.__class__.__name__}) does not have one."
            )

        tensor_examples = self.tokenizer.batch_encode_plus(
            [ex for ex in examples if ex],
            max_length=self.block_size,
            return_tensors="pt",
            pad_to_max_length=True,
            return_attention_mask=True,
            truncation=True,
        )

        input_ids, attention_mask = tensor_examples["input_ids"], tensor_examples["attention_mask"]
        return input_ids, attention_mask

dataset = load_dataset('text', data_files='train.txt',cache_dir="./", , split='train')
data_collator = DataCollatorForDatasetsLanguageModeling(tokenizer=tokenizer, mlm=True,
                      mlm_probability=0.15, block_size=tokenizer.max_len)
trainer = Trainer(model=model, args=args, data_collator=data_collator,
                      train_dataset=train_dataset, prediction_loss_only=True, )
trainer.train(model_path=model_path)
```
This train.txt is about 1.1GB and has 90k lines where each line is a sequence of 4k words.
During training, the memory usage increased fast as the following graph and resulted in OOM before the finish of training.

![image](https://user-images.githubusercontent.com/29704017/93292112-5576b280-f817-11ea-8da2-b2db9bf35665.png)

Could you please give me any suggestions on why this happened and how to fix it?
Thanks.


Fix typos in the loading datasets docs

This PR fixes two typos in the loading datasets docs, one of them being a broken link to the `load_dataset` function.


Fix text delimiter

> Which OS are you using ?

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Do you mind sharing the data you used (or part of it), so I can try to reproduce ?
Or at least some info about the text file you're using ? (size, n of lines, encoding)

Lot of data, difficult to share. There are 46 shards, each having about 256000 lines. using `file` command gives this - `ASCII text, with very long lines`.

Ok I see, no problem :)
I'll see what I can do

Could you just test with one single dummy text file with a few lines to see if you're having the issue ?
Also which version of `datasets` do you have ?

I changed the delimiter in the `text` dataset script.
It should fix the `pyarrow.lib.ArrowInvalid: CSV parse error` from #622

I changed the delimiter to an unused ascii character that is not present in text files : `\b`