| --- |
| language: |
| - ca |
| license: apache-2.0 |
| tags: |
| - catalan |
| - masked-lm |
| - distilroberta |
| widget: |
| - text: El Català és una llengua molt <mask>. |
| - text: Salvador Dalí va viure a <mask>. |
| - text: La Costa Brava té les millors <mask> d'Espanya. |
| - text: El cacaolat és un batut de <mask>. |
| - text: <mask> és la capital de la Garrotxa. |
| - text: Vaig al <mask> a buscar bolets. |
| - text: Antoni Gaudí vas ser un <mask> molt important per la ciutat. |
| - text: Catalunya és una referència en <mask> a nivell europeu. |
| --- |
| |
| # DistilRoBERTa-base-ca |
|
|
| ## Model description |
|
|
| This model is a distilled version of [projecte-aina/roberta-base-ca-v2](https://huggingface.co/projecte-aina/roberta-base-ca-v2). |
|
|
| It follows the same training procedure as [DistilBERT](https://arxiv.org/abs/1910.01108), using the implementation of Knowledge Distillation |
| from the paper's [official repository](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation). |
|
|
| The resulting architecture consists of 6 layers, 768 dimensional embeddings and 12 attention heads. |
| This adds up to a total of 82M parameters, which is considerably less than the 125M of standard RoBERTa-base models. |
| This makes the model lighter and faster than the original, at the cost of a slightly lower performance. |
|
|
| ## Training |
|
|
| ### Training procedure |
|
|
| This model has been trained using a technique known as Knowledge Distillation, |
| which is used to shrink networks to a reasonable size while minimizing the loss in performance. |
|
|
| It basically consists in distilling a large language model (the teacher) into a more |
| lightweight, energy-efficient, and production-friendly model (the student). |
|
|
| So, in a “teacher-student learning” setup, a relatively small student model is trained to mimic the behavior of a larger teacher model. |
| As a result, the student has lower inference time and the ability to run in commodity hardware. |
|
|
| ### Training data |
|
|
| The training corpus consists of several corpora gathered from web crawling and public corpora, as shown in the table below: |
|
|
| | Corpus | Size (GB) | |
| |--------------------------|-----------:| |
| | Catalan Crawling | 13.00 | |
| | RacoCatalá | 8.10 | |
| | Catalan Oscar | 4.00 | |
| | CaWaC | 3.60 | |
| | Cat. General Crawling | 2.50 | |
| | Wikipedia | 1.10 | |
| | DOGC | 0.78 | |
| | Padicat | 0.63 | |
| | ACN | 0.42 | |
| | Nació Digital | 0.42 | |
| | Cat. Government Crawling | 0.24 | |
| | Vilaweb | 0.06 | |
| | Catalan Open Subtitles | 0.02 | |
| | Tweets | 0.02 | |
|
|
| ## Evaluation |
|
|
| ### Evaluation benchmark |
|
|
| This model has been fine-tuned on the downstream tasks of the [Catalan Language Understanding Evaluation benchmark (CLUB)](https://club.aina.bsc.es/), which includes the following datasets: |
|
|
| | Dataset | Task| Total | Train | Dev | Test | |
| |:----------|:----|:--------|:-------|:------|:------| |
| | AnCora | NER | 13,581 | 10,628 | 1,427 | 1,526 | |
| | AnCora | POS | 16,678 | 13,123 | 1,709 | 1,846 | |
| | STS-ca | STS | 3,073 | 2,073 | 500 | 500 | |
| | TeCla | TC | 137,775 | 110,203| 13,786| 13,786| |
| | TE-ca | RTE | 21,163 | 16,930 | 2,116 | 2,117 | |
| | CatalanQA | QA | 21,427 | 17,135 | 2,157 | 2,135 | |
| | XQuAD-ca | QA | - | - | - | 1,189 | |
|
|
| ### Evaluation results |
|
|
| This is how it compares to its teacher when fine-tuned on the aforementioned downstream tasks: |
|
|
| | Model \ Task |NER (F1)|POS (F1)|STS-ca (Comb.)|TeCla (Acc.)|TEca (Acc.)|CatalanQA (F1/EM)| XQuAD-ca <sup>1</sup> (F1/EM) | |
| | ------------------------|:-------|:-------|:-------------|:-----------|:----------|:----------------|:------------------------------| |
| | RoBERTa-base-ca-v2 | 89.29 | 98.96 | 79.07 | 74.26 | 83.14 | 89.50/76.63 | 73.64/55.42 | |
| | DistilRoBERTa-base-ca | 87.88 | 98.83 | 77.26 | 73.20 | 76.00 | 84.07/70.77 | 62.93/45.08 | |
|
|
| <sup>1</sup> : Trained on CatalanQA, tested on XQuAD-ca (no train set). |
