% 0 Conference Proceedings% TA Large DataBase of Hypernymy Relations Extracted from the Web.% A Seitner, Julian% A Bizer, Christian% A Eckert, Kai

… for many word understanding applications. We present a publicly available database containing more than 400 million hypernymy relations we extracted from the CommonCrawl web corpus. We describe the infrastructure we …


A 6-month Analysis of Factors Impacting Web Browsing Quality for QoE Prediction

A Saverimoutou, B Mathieu, S Vaton - Computer Networks, 2019

… Fourth Party [15] instruments the Mozilla-Firefox browser and Web Xray [16] is a PhantomJS based tool for measuring HTTP traffic. XRay [17] and AdFisher [18] run automated personalization detection experiments and …


A Bilingual Adversarial Autoencoder for Unsupervised Bilingual Lexicon Induction

X Bai, H Cao, K Chen, T Zhao - IEEE/ACM Transactions on Audio, Speech, and …, 2019

… [29]. This dataset consists of gold dictionaries and 300-dimensional CBOW5 embeddings trained on WacKy crawling corpora (English, Italian, German), Common Crawl (Finish) and WMT News Crawl (Spanish). We report the results 2We set k = 10 …


A Combined Approach to Automatic Taxonomy Extraction

… [10] presented publicly available database of hypernym relations called WebIsA. This database was created using Hearst-like patterns on CommonCrawl web corpus. They extracted more than 400 million hypernymy relations …


A Common Semantic Space for Monolingual and Cross-Lingual Meta-Embeddings

I García, R Agerri, G Rigau - arXiv preprint arXiv:2001.06381, 2020

… From GloVe (GV) [34], the Common Crawl vectors (600 billion words) … The WS353 dataset is di- vided in two subsets [1]. In this section all the meta-embeddings have been mapped to the vector space of the English FastText (Common Crawl, 600B tokens) …


A COMPARATIVE STUDY ON END-TO-END SPEECH TO TEXT TRANSLATION

… We select our checkpoints based on the dev set. For the MT training, we use the TED, OpenSubtitles2018, Europarl, ParaCrawl, CommonCrawl, News Commentary, and Rapid corpora resulting in 32M sentence pairs after filtering noisy samples …


A Comparison of Context-sensitive Models for Lexical Substitution

AG Soler, A Cocos, M Apidianaki, C Callison-Burch

… to two context-insensitive baselines that solely rely on the target-to-substitute similarity of standard, pre-trained word embeddings: 300-dimensional GloVe vectors (Pennington et al., 2014)5 and 300-dimensional FastText vectors …


A Comparison of Neural Document Classification Models

… Bojanowski et al. (2016). It is available for 294 languages. An updated version, trained on Common Crawl in addition to Wikipedia and with adapted parameters is available for 157 languages (Grave et al., 2018). These are the …


A Comparison on Fine-grained Pre-trained Embeddings for the WMT19Chinese-English News Translation Task

Z Li, L Specia - Proceedings of the Fourth Conference on Machine …, 2019

… In addition, the Common Crawl Corpus from WMT is used as monolingual data to pre-train the embeddings … We trained the embeddings on the Common Crawl Corpus provided by WMT19 and fine-tuned them on the task data when training the RNN …


A Contextualized Word Representation Approach for Irony Detection

L Garcıa, D Moctezuma, V Muniz - Proceedings of the Iberian Languages Evaluation …, 2019

… 2.2 Word Embeddings We use the ELMo pre-trained word embeddings provided by [4], which were trained with a corpus of 20 million-words randomly sampled from the raw text released by the CoNLL 2018 shared …


A Dataset for Content Error Detection in Web Archives

J Kiesel, F Hubricht, B Stein, M Potthast

… The Webis Web Archive 2017 [9] contains 10,000 web pages sampled from the Common Crawl [11] in a way which ensured that both well-known and less … January 2017 Common Crawl Archive, 2017. http: //commoncrawl.org …


A Deep Dive into Supervised Extractive and Abstractive Summarization from Text

M Dey, D Das - Data Visualization and Knowledge Engineering, 2020

… 7.1. As mentioned in the algorithm, estimated frequency p(w) of every words have been found from datasets (enwiki, poliblogs, commoncrawl, text8) [1]. The parameter “a” for our task is fixed at \(3 * 10^{-3}\). The vector of all …


A Deep Learning Approach for Identification of Confusion in Unstructured Crowdsourced Annotations

… improvements in model performance. We also compared the VQR binary classification model based on GLoVe embeddings with a model based on FastText embeddings trained on Common Crawl (3; 14). Since the FastText …


A Dynamic Evolutionary Framework for Timeline Generation based on Distributed Representations

D Liang, G Wang, J Nie - arXiv preprint arXiv:1905.05550, 2019

… To learn the distributed representations, we use pre-trained word vectors2, trained on Common Crawl and Wikipedia by fastText tookit [2]. Furthermore, each v(q) is only embedded by the name of the topic q as a experimental control …


A Framework to Estimate the Nutritional Value of Food in Real Time Using Deep Learning Techniques

R Yunus, O Arif, H Afzal, MF Amjad, H Abbas… - IEEE Access, 2019

… First is Common-Crawl, which is an archive hosted on an Amazon S3 bucket … is more relevant as it return pages corresponding to precise labels while the text from Common-Crawl is more generic. The raw text data obtained …


A Language Invariant Neural Method for TimeML Event Detection

S Prabhu, P Goel, A Debnath, M Shrivastava

… The CNN uses 40 filters with a window size of 3. For our contextual word embeddings, we use fastText embeddings for English (Bojanowski et al., 2017) which are pretrained on commonCrawl and the Wikipedia corpus. FastText …


A Massive Collection of Cross-Lingual Web-Document Pairs

A El-Kishky, V Chaudhary, F Guzman, P Koehn - arXiv preprint arXiv:1911.06154, 2019

… Other works (Smith et al., 2013) have mined Common Crawl for bitexts for machine … and restrictions, we mined 54 million aligned documents across 12 Common Crawl snapshots … 2: NMT performance on comparable directions …


A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations

04/02/19 - We propose a generative model for a sentence that uses two latent variables, with one intended to represent the syntax of the sent...


A New Corpus for Low-Resourced Sindhi Language with Word Embeddings

W Ali, J Kumar, J Lu, Z Xu - arXiv preprint arXiv:1911.12579, 2019

… 3Available online at https://rdrr.io/cran/wordspace/man/WordSim353.html 4We denote Sindhi word representations as (SdfastText) recently revealed by fastText, available at (https://fasttext.cc/docs/en/crawl-vectors.html) trained …


A New Hybrid Ensemble Feature Selection Framework for Machine Learning-based Phishing Detection System

KL Chiew, CL Tan, KS Wong, KSC Yong, WK Tiong - Information Sciences, 2019

… June 2017. Specifically, we selected 5000 phishing webpages based on URLs from PhishTank 2 and OpenPhish 3 , and another 5000 legitimate webpages based on URLs from Alexa 4 and the Common Crawl 5 archive. The …


A Question-Entailment Approach to Question Answering

AB Abacha, D Demner-Fushman - arXiv preprint arXiv:1901.08079, 2019

… We use the pretrained common crawl version with 840B tokens and 300d vectors, which are not updated during training. 3.3 Logistic Regression Classifier In this feature-based approach, we use Logistic Regression to …


A Recurrent Deep Neural Network Model to measure Sentence Complexity for the Italian

… The authors have used FastText [3], a library for efficient learning of word representations and sentence classification, trained on Common Crawl [21] and Wikipedia to create a pre-trained word vector representation for …


A Robust Abstractive System for Cross-Lingual Summarization

… about 23k sentences for Somali and Swahili and 51k for Tagalog); noisy, web-crawled parallel data (So- mali only, about 354k sentences); and synthetic, backtranslated parallel data created from monolingual sources including …


A Study of Neural Networks Models applied to Natural Language Inference

… word vectors of different size, in order to check whether the word space dimension plays an important role on the final results: – A 100d version trained on the Wikipedia 2014 + Gigaword 5 corpus, with 6B tokens; – The 300d …


A Survey of URL-based Phishing Detection

… 2017 [40] 98.76 98.60 98.93 98.76 99.91 Common Crawl PhishTank 1M 1M Balanced Path length, URL entropy, length ratio, '@' and '-' count, punctuation count, TLDs count, IP address, suspicious words count …


S Maruf, F Saleh, G Haffari - arXiv preprint arXiv:1912.08494, 2019

Page 1. A Survey on Document-level Machine Translation: Methods and Evaluation Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari Faculty of Information Technology, Monash University, Clayton VIC, Australia {firstname.lastname}@monash.edu …


A System to Monitor Cyberbullying based on Message Classification and Social Network Analysis

S Menini, G Moretti, M Corazza, E Cabrio, S Tonelli… - Proceedings of the Third …, 2019

… timestep. We use English Fasttext embeddings1 trained on Common Crawl with a size of 300. Concerning hy- perparameters, our model uses no dropout and no batch normalization on the outputs of the hidden layer. Instead …


A Systematic Comparison Between SMT and NMT on Translating User-Generated Content

… Morever, as the Europarl corpus is a fix-domain and did not work well for our experiments, we plan to utilise other types of mix-domain parallel resource such as common crawl corpus9 … WASSA '12 (2012) 52–60 9 http://www.statmt …


A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Computer Science

Page 1. COMPUTATIONAL ANALYSIS OF ARGUMENTS AND PERSUASIVE STRATEGIES IN POLITICAL DISCOURSE by Nona Naderi A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy …


A Vector Worth a Thousand Counts

DS Hain, R Jurowetzkiφ, T Buchmannψ, P Wolfψ

… task convolutional neural network model trained on OntoNotes, with GloVe vectors (685k unique vectors with 300 dimensions) trained on Common Crawl. Given a patent abstract, spaCy predicts the meaning of each term in the document …


Abstract Text Summarization: A Low Resource Challenge

… We propose an iterative data augmentation approach which uses synthetic data along with the real summarization data for the German language. To generate synthetic data, the Common Crawl (German) dataset is exploited, which covers different domains …


AC-Net: Assessing the Consistency of Description and Permission in Android Apps

Y Feng, L Chen, A Zheng, C Gao, Z Zheng - IEEE Access, 2019

… Compared to the somewhat popular embeddings such as GloVe [22] (400 thousand word vectors trained on Wikipedia) and crawl [23] (2 million word vectors trained on Common Crawl), ours fully retains domain-specific characteristics of statements …


Acquiring Knowledge from Pre-trained Model to Neural Machine Translation

R Weng, H Yu, S Huang, S Cheng, W Luo - arXiv preprint arXiv:1912.01774, 2019

… Following Song et al. (2019) , on the English and German, we use the monolingual data from WMT News Crawl. We select 50M sentence from year 2007 to 2017 for English and German respectively. Then, we choose 50M sentence from Common Crawl for Chinese …


Adapting Transformer-XL Techniques to QANet Architecture for SQuAD 2.0 Challenge

… set data. • glove.840B.300dglove.840B.300d.txt: Pretrained GloVevectors. Theseare300dimensional embeddings trained on the CommonCrawl 840B corpus. • {word,char}_emb.json: Word and character embeddings. Only the …


Adaptive Cross-Modal Few-Shot Learning

C Xing, N Rostamzadeh, BN Oreshkin, PO Pinheiro - arXiv preprint arXiv:1902.07104, 2019

… category labels. GloVe is an unsupervised approach based on wordword co-occurrence statistics from large text corpora. We use the Common Crawl version trained on 840B tokens. The embeddings are of dimension 300. When …


Advanced Deep learning Methods and Applications in Open-domain Question Answering

Page 1. VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY Nguyen Minh Trang ADVANCED DEEP LEARNING METHODS AND APPLICATIONS IN OPEN-DOMAIN QUESTION ANSWERING MASTER THESIS …


Adversarial NLI: A New Benchmark for Natural Language Understanding

Y Nie, A Williams, E Dinan, M Bansal, J Weston… - arXiv preprint arXiv …, 2019

… transfer. In addition to contexts from Wikipedia for Round 3, we also included contexts from the following domains: News (ex- tracted from Common Crawl), fiction (extracted from Mostafazadeh et al. 2016, Story Cloze, and Hill et al …


Adverse drug event detection from electronic health records using hierarchical recurrent neural networks with dual-level embedding

S Wunnava, X Qin, T Kakar, C Sen, EA Rundensteiner… - Drug Safety, 2019

… compared the results from DLADE, which uses domainand task-specific MADE1.0 word embedding trained using wiki, and Pittsburgh EHR and PubMed articles (1,352,550 word vectors) [10, 21], with two systems that use …


AELA-DLSTMs: Attention-Enabled and Location-Aware Double LSTMs for Aspect-level Sentiment Classification

K Shuang, X Ren, Q Yang, R Li, J Loo - Neurocomputing, 2018


Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition

O Arshad, I Gallo, S Nawaz, A Calefati - arXiv preprint arXiv:1904.01356, 2019

… Page 5. B. Word embeddings We used 300D fasttext crawl embeddings. It contains 2 million word vectors trained with subword information on Common Crawl (600B tokens). However, we do not apply fine-tuning on these embeddings during the training stage …


Algorithmic Bias and the Biases of the Bias Catchers

D Rozado - arXiv preprint arXiv:1905.11985, 2019

… This work systematically analyzed 3 popular word embedding methods: Word2vec (Skipgram) (4), Glove (9) and FastText (10), externally pretrained on a wide array of corpora such as Google News, Wikipedia, Twitter and Common Crawl …


All-in-One: Emotion, Sentiment and Intensity Prediction using a Multi-task Ensemble Framework

S Akhtar, D Ghosal, A Ekbal, P Bhattacharyya… - IEEE Transactions on …, 2019

… IEEE TRANSACTIONS ON AFFECTIVE COMPUTING. 4 A. Deep Learning Models We employ the architecture of Figure 1a to train and tune all the deep learning models using pre-trained GloVe (common crawl 840 billion) word embeddings [32] …


AMR-to-Text Generation with Cache Transition Systems

L Jin, D Gildea - arXiv preprint arXiv:1912.01682, 2019

… 6.2 Setup The word embeddings are initialized with 300-dimensional GloVe embeddings (Penningtonetal., 2014) from the Common Crawl and are fixed during training. This embedding vocabulary consists of both English tokens and AMR concept labels …


An effective approach to candidate retrieval for cross-language plagiarism detection: A fusion of conceptual and keyword-based schemes

M Roostaee, MH Sadreddini, SM Fakhrahmad - Information Processing & …, 2020


An Efficient Framework for Processing and Analyzing Unstructured Text to Discover Delivery Delay and Optimization of Route Planning in Realtime

Page 1. HAL Id: tel-02310852 https://tel.archives-ouvertes.fr/tel-02310852 Submitted on 10 Oct 2019 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not …


An Empirical Evaluation of Text Representation Schemes on Multilingual Social Web to Filter the Textual Aggression

S Modha, P Majumder - arXiv preprint arXiv:1904.08770, 2019

… Glove pre-trained model available with different embed size and trained on common crawl, Twitter. We have use Glove pre-trained model with vocabulary size 2.2 million and trained on common crawl. fastText pretrained models are available in 157 language …


An Empirical study on Pre-trained Embeddings and Language Models for Bot Detection

A Garcia-Silva, C Berrio, JM Gómez-Pérez - Proceedings of the 4th Workshop on …, 2019

… scratch. We use pre-trained embeddings learned from Twitter it- self, urban dictionary definitions to accommodate the informal vocabulary often used in the social network, and common crawl as a general source of information …


An Ensemble Method for Producing Word Representations for the Greek Language

M Lioudakis, S Outsios, M Vazirgiannis - arXiv preprint arXiv:1912.04965, 2019

… In addition, we show that CBOS outperforms the CBOW and Skip-gram models when they are trained on the same data. The future work of this research could include training of our newly proposed model with the Common Crawl dataset for the Greek language …


An Exploration of Sarcasm Detection Using Deep Learning

… not appear in the training data (”out-of-vocabulary” words). It also supports 157 different languages. In our research we use 300-dimensional word vectors pre-trained on Common Crawl3 (600B tokens). 4.3 ELMo ELMo (Embeddings …


An Extended CLEF eHealth Test Collection for Cross-Lingual Information Retrieval in the Medical Domain

S Saleh, P Pecina - European Conference on Information Retrieval, 2019

… However, an additional assessment was performed [15]. CLEF eHealth 2018 Consumer Health Search Task released a document collection created using CommonCrawl platform [8] containing more than five million documents from more than thousand websites …


An integrated neural decoder of linguistic and experiential meaning

AJ Anderson, JR Binder, L Fernandino, CJ Humphries… - Journal of Neuroscience, 2019

… occurrence 210 matrix (vocabulary size is 2.2million words and co-occurrences were measured across 840 billion 211 tokens from Common Crawl https://commoncrawl.org). GloVe in particular was used because it yielded 212 state-of …


An Open-Domain System for Retrieval and Visualization of Comparative Arguments from Text

… parts of the Page 23. 3.1. Retrieval of Sentences 19 CommonCrawl: Full text search index Retrieval of Sentences: Elasticsearch API Sentence Classification: Keyword or ML approach Sentence Ranking: Ordering of sentences


Analysing Coreference in Transformer Outputs

ELKC Espana-Bonet, J van Genabith - DiscoMT 2019, 2019

… Recent NMT systems that include context deal with both phenomena, coreference and coherence, but usually context is limited to the previous sen- # lines S1, S3 S2 Common Crawl 2394878 x1 x4 Europarl 1775445 …


Analysing Representations of Memory Impairment in a Clinical Notes Classification Model

M Ormerod, J Martínez-del-Rincón, N Robertson… - Proceedings of the 18th …, 2019

… In this study we use 300-dimensional fastText word embeddings (Bojanowski et al., 2017) which were pretrained on the Common Crawl dataset using the skipgram schema (Mikolov et al., 2013), which in- volves predicting a target word based on nearby words …


Analysis of Joint Multilingual Sentence Representations and Semantic K-Nearest Neighbor Graphs

H Schwenk, D Kiela, M Douze - Proceedings of the AAAI Conference on Artificial …, 2019

… All the statistics in this section are calculated on ten million sentences of Common Crawl data. Please remember that we apply romanization for the languages which use a Cyrillic script (Greek, Bosnian, Bulgarian, Macedonian, Serbian and Russian) …


Analysis of Positional Encodings for Neural Machine Translation

… We train our models on the data from the De→En and the Zh→En news translation task of WMT 2019. For the De→En task we train on CommonCrawl, Europarl, NewsCommentary and Rapid summing up …


Annotating and Recognising Visually Descriptive Language

T Alrashid, J Wang, R Gaizauskas - … on Interoperable Semantic Annotation (ISA-15), 2019

… Stop words were only removed for the word embedding representation. We did not remove stop words for tf-idf because the approach 5https://spacy. io/. We use the model en vectors web lg. 6http://commoncrawl. org/the-data/ 7http://www. scikit-learn. org/ 27 Page 34 …


Answering Comparative Questions: Better than Ten-Blue-Links?

M Schildwächter, A Bondarenko, J Zenker, M Hagen… - arXiv preprint arXiv …, 2019

… CommonCrawl: Full text search index … Clicking on a result sentence reveals its Common Crawl context—by default the ±3 sentences around it, with the … underlying corpus of our CAM system and the keyword-based search …


Architecture for semantic search over encrypted data in the cloud

J Woodworth, MA Salehi - US Patent App. 16/168,919, 2019

… The dataset has a total size of 357 MB and is made up of 6,942 text files. To evaluate S3C under large scale datasets, a second dataset, the Common Crawl Corpus from AWS (a web crawl composed of over five billion web pages) was used …


Are we consistently biased? Multidimensional analysis of biases in distributional word vectors

A Lauscher, G Glavaš - arXiv preprint arXiv:1904.11783, 2019

… 4 we compare the biases of em- beddings trained with the same model (GLOVE) but on different corpora: Common Crawl (ie, noisy … Table 4: WEAT bias effects for GLOVE embeddings trained on different corpora: Wikipedia …


Are We Safe Yet? The Limitations of Distributional Features for Fake News Detection

T Schuster, R Schuster, DJ Shah, R Barzilay - arXiv preprint arXiv:1908.09805, 2019

… 2019). The generator is trained with a LM objective on a large news corpus from Common Crawl dumps. The fake news detector is a simple linear classifier on top of the last hidden state of Grover's LM on the examined article …


Argument Generation with Retrieval, Planning, and Realization

X Hua, Z Hu, L Wang - arXiv preprint arXiv:1906.03717, 2019

… Wachsmuth et al., 2017b, 2018b). Recent work by Stab et al. (2018) in- dexes all web documents collected in Common Crawl, which inevitably incorporates noisy, lowquality content. Besides, existing work treats individual …


Argument Search: Assessing Argument Relevance

M Potthast, L Gienapp, F Euchner, N Heilenkötter… - 2019

… online debating portals. Thereafter, ArgumenText [19], which retrieves argumentative sentences from the Common Crawl, and “multi-perspective answers” in the US version of Bing3 have been published. Another loosely related …


Articles Classification in Myanmar Language

MS Phyu, KT Nwet - 2019 International Conference on Advanced …, 2019

… Fortunately, Grave et al. [3] recently released pretrained vectors for 246 languages trained on Wikipedia and common crawl … Data source Method Number of word vectors Dimension Wikipedia fastText (skip-gram) 91,497 …


Artificial Intelligence: An Overview

Page 1. Chapter 1 Artificial Intelligence: An Overview Peter Grogono Department of Computer Science and Software Engineering Concordia University Montréal, Québec The smart devices that we have become so familiar with …


Aspect Detection using Word and Char Embeddings with (Bi) LSTM and CRF

Ł Augustyniak, T Kajdanowicz, P Kazienko - 2019 IEEE Second International …, 2019

… Glove 840B - Global Vectors for Word Representation proposed by Stanford NLP Group, trained based on Common Crawl. fastText - Distributed Word Representation proposed by Facebook, trained on Common Crawl as well …


Page 1. DEPARTMENT OF INFORMATICS TECHNISCHE UNIVERSITÄT MÜNCHEN Master's Thesis in Informatics Aspect-Based Sentiment Analysis Using Deep Neural Networks and Transfer Learning Sumit Dugar Page 2. DEPARTMENT OF INFORMATICS …


YC Tan, LE Celis - arXiv preprint arXiv:1911.01485, 2019

Page 1. Assessing Social and Intersectional Biases in Contextualized Word Representations Yi Chern Tan, L. Elisa Celis Yale University {yichern.tan, elisa.celis}@yale.edu Abstract Social bias in machine learning has drawn …


Assessing the Impact of Contextual Embeddings for Portuguese Named Entity Recognition

J Santos, B Consoli, C dos Santos, J Terra, S Collonini… - 2019 8th Brazilian …, 2019

… 2019. [19] C. Buck, K. Heafield, and B. van Ooyen, “N-gram counts and language models from the common crawl,” in Proceedings of the Language Resources and Evaluation Conference, Reykjavik, Iceland, May 2014. [20] N …


Assessing the Lexico-Semantic Relational Knowledge Captured by Word and Concept Embeddings

R Denaux, JM Gomez-Perez - arXiv preprint arXiv:1909.11042, 2019

… three different corpora, which we chose to study whether relation prediction capacity varies depending on the corpus size: the English United Nations corpus[Ziemski et al., 2016] (517M tokens), the En- glish Wikipedia (just under …


Assessment of text coherence using an ontology‐based relatedness measurement method

Abstract This paper proposes a novel method for assessing text coherence. Central to this approach is an ontology‐based representation of text, which captures the level of relatedness between conse...


AStylometric INVESTIGATION OF CHARACTER VOICES IN LITERARY FICTION

Page 1. ASTYLOMETRIC INVESTIGATION OF CHARACTER VOICES IN LITERARY FICTION by Krishnapriya Vishnubhotla A thesis submitted in conformity with the requirements for the degree of Master of Science …


Asymmetry Sensitive Architecture for Neural Text Matching

T Belkacem, JG Moreno, T Dkaki, M Boughanem - European Conference on …, 2019

… We adopted a cross-validation with \(80\%\) to train, \(10\%\) to test and \(10\%\) to validate the different models. We used a public pre-trained 300-dimensional word vectors of GloVe 3 , which are trained in a Common crawl dataset …


Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures

PJ Ortiz Suárez, B Sagot, L Romary - 2019

… 7http://commoncrawl.org/about/ 8http://microformats.org/wiki/ rel-nofollow 9https://www.robotstxt.org … In order to download, extract, filter, clean and classify Common Crawl we base ourselves on … Each of these processes first …


At the Lower End of Language—Exploring the Vulgar and Obscene Side of German

E Eder, U Krieg-Holz, U Hahn - Proceedings of the Third Workshop on Abusive …, 2019

… 123 FASTTEXT (Grave et al., 2018) word embeddings, the latter being based on COMMON CRAWL and WIKIPEDIA … et al., 2017) trained on German tweets (TWITTER) and, finally, FASTTEXT word embeddings (Grave et al., 2018) …


Attending the Emotions to Detect Online Abusive Language

NS Samghabadi, A Hatami, M Shafaei, S Kar, T Solorio - arXiv preprint arXiv …, 2019

… Then, we extract 5For ask.fm data, we use 300-dimensional Common Crawl Glove pre-trained embeddings, since it works better than the Twitter embedding. the DeepMoji vector for each sentence and calculate the average vector per post …


Attention Guided Graph Convolutional Networks for Relation Extraction

Z Guo, Y Zhang, W Lu - arXiv preprint arXiv:1906.07510, 2019

… 5https://nlp.stanford.edu/projects/ tacred/ 6We use the 300-dimensional Glove word vectors trained on the Common Crawl corpus https://nlp. stanford.edu/projects/glove/ 7The results are produced by the open implementation of Zhang et al. (2018). Page 6. Model …


Attenuating Bias in Word Vectors

S Dev, J Phillips - arXiv preprint arXiv:1901.07656, 2019

Page 1. Attenuating Bias in Word Vectors Sunipa Dev Jeff Phillips University of Utah University of Utah Abstract Word vector representations are well developed tools for various NLP and Machine Learning tasks and are known …


Attribute Sentiment Scoring With Online Text Reviews: Accounting for Language Structure and Attribute Self-Selection

Page 1. Attribute Sentiment Scoring with Online Text Reviews: Accounting for Language Structure and Attribute Self-Selection Ishita Chakraborty, Minkyung Kim, K. Sudhir Yale School of Management March 2019 We thank the …


Augmenting Neural Machine Translation through Round-Trip Training Approach

… For the high-resource scenario (En- Es) we utilize the English-Spanish bilingual corpora from WMT'18² [29] which contains 10M sentence pairs extracting from Europarl, News-Commentary, UN and Common Crawl collections …


AutoEncoder Guided Bootstrapping of Semantic Lexicon

C Hu, M Nakano, M Okumura - Pacific Rim International Conference on Artificial …, 2019

… The top-5 best candidate instances were then added to the expanded seed list for the next iteration. For the inputs of the AutoEncoder model, Glove embeddings (300 dimensions) trained on Common Crawl were used (Pennington et al …


Automated Dictionary Creation for Analyzing Text: An Illustration from Stereotype Content

… The Glove word embeddings used here were trained using around 840 billion words from the common crawl (a very large database of web text), and it has word-vectors with 300 dimensions for 2.2 million words (available …


Automated extraction of attributes from natural language attribute-based access control (ABAC) Policies

M Alohaly, H Takabi, E Blanco - Cybersecurity, 2019

The National Institute of Standards and Technology (NIST) has identified natural language policies as the preferred expression of policy and implicitly called for an automated translation of ABAC...


Automated Grading of Short Text Answers: Preliminary Results in a Course of Health Informatics

G De Gasperis, S Menini, S Tonelli, P Vittorini - International Conference on Web …, 2019

… vectors representing both words and sub-words. To generate these embeddings we start from the pre-computed Italian language model 3 , trained on Common Crawl and Wikipedia. The latter, in particular, is suitable for our …


Automated lifelog moment retrieval based on image segmentation and similarity scores

S Taubert, S Kahl, D Kowerko, M Eibl - CLEF2019 Working Notes. CEUR Workshop …, 2019

… Page 8. 4 Resources We only used resources which were open source. Our word vectors were pretrained GloVe vectors from Common Crawl which had 300 dimensions and a vocabulary of 2.2 million tokens [27]. Furthermore …


Automated organ-level classification of free-text pathology reports to support a radiology follow-up tracking engine

JM Steinkamp, CM Chambers, D Lalevic, HM Zafar… - Radiology: Artificial …, 2019

… GloVe word embeddings pretrained on the Common Crawl dataset of web pages were used as input to both networks; no performance benefit was observed from continuing to train word embeddings during the experiments …


Automatic Knowledge Extraction to build Semantic Web of Things Applications

M Noura, A Gyrard, S Heil, M Gaedke - IEEE Internet of Things Journal, 2019

… The naming process used additional hypernym information derived from WebIsALOD12 the Linked Open Data version of the WebIsA Database, a database containing 11.7 million hypernymy re- lations extracted from the CommonCrawl web corpus …


Automatic stance detection on political discourse in Twitter

Page 1. Automatic Stance Detection on Political Discourse in Twitter Author: Elena Zotova Advisors: Rodrigo Agerri and German Rigau Hizkuntzaren Azterketa eta Prozesamendua Language Analysis and Processing Master's Thesis …


Automatic Text Difficulty Estimation Using Embeddings and Neural Networks

A Filighera, T Steuer, C Rensing - European Conference on Technology Enhanced …, 2019

… Next, the resulting tokens are embedded. The following pre-trained embedding models were used in our experiments: the word2vec [14], the uncased Common Crawl GloVe [15], the original ELMo [16], the uncased …


Automatic Text Summarization of News Articles in Serbian Language

D Kosmajac, V Kešelj - 2019 18th International Symposium INFOTEH …, 2019

… 1). A. Auxiliary word2vec generated from Bosnian Wikidumps For input representation we used pre-trained glove word embeddings for Bosnian language2. They were trained on Common Crawl and Wikipedia using fastText [20] …


Automating Analysis and Feedback to Improve Mathematics' Teachers' Classroom Discourse

A Suresh, T Sumner, J Jacobs, B Foland, W Ward - Paper submitted to the ninth …, 2019

… GloVe or Global vectors for word representation is an unsupervised learning algorithm trained on aggregated word-word co-occurrence statistics from a corpus. In our model, we use the vectors trained on Common Crawl with 840 billion tokens and 300 dimensions …


Page 1. Automating the Fact-Checking Task: Challenges and Directions Dissertation zur Erlangung des Doktorgrades (Dr. rer. nat.) der Mathematisch-Naturwissenschaftlichen Fakultät der Rheinischen Friedrich-Wilhelms-Universität Bonn …


Backlink Analyser using Apache Spark

M Zeeshan, S Asim, A Nadeem Anwar - 2019

… Google Page Rank is assigned by Google based on different website factors (Design, Visitors, and Quality of content). Common Crawl [1] is an open repository for web crawl data … We will use subset of Common Crawl dataset good enough to demonstrate our system …


BEING PROFILED: COGITAS ERGO SUM: 10 Years of Profiling the European Citizen

I Baraliuc, E Bayamlioglu, M Hildebrandt, L Janssens - 2019


Beyond Bag-of-Concepts: Vectors of Locally Aggregated Concepts

… Word2Vec pre-trained embeddings were trained on the Google News data set and contain vectors for 3 million English words.1 GloVe pre-trained embeddings were trained on the Common Crawl data set and contain vectors for 1.9 million English words.2 Pre-trained …


P Dinklage, J Ellert, J Fischer, D Köppl, M Penschuck - arXiv preprint arXiv …, 2019

Page 1. Bidirectional Text Compression in External Memory Patrick Dinklage Technische Universität Dortmund, Department of Computer Science patrick.dinklage@ tu-dortmund.de Jonas Ellert Technische Universität Dortmund …


L Li, W Chan - arXiv preprint arXiv:1910.13034, 2019

… Eu- Page 3. Figure 1: Big Bidirectional Insertion Representations for Documents roparl, Rapid, News-Commentary) and parallel sentence-level data (WikiTitles, Common Crawl, Paracrawl). The test set is newstest2019. The …


Big BiRD: A Large, Fine-Grained, Bigram Relatedness Dataset for Examining Semantic Composition

… Word representations: We use GloVe word embeddings pre-trained on 840B-token CommonCrawl corpus16 and fastText word embeddings pre-trained on Common Crawl and Wikipedia using CBOW.17 For the …


Big Data Competence Center ScaDS Dresden/Leipzig: Overview and selected research activities

E Rahm, WE Nagel, E Peukert, R Jäkel, F Gärtner… - Datenbank-Spektrum

… devise multiple research activities. The most promising direction resulted in the publication of the “Dresden WebTable Corpus” (DWTC) 2 [9] based on the freely available web crawl “CommonCrawl”. The DWTC corpus consists …


Boosting Implicit Discourse Relation Recognition with Connective-based Word Embeddings

C Wu, J Su, Y Chen, X Shi - Neurocomputing, 2019


Bornholmsk Natural Language Processing: Resources and Tools

L Derczynski, ITU Copenhagen, AS Kjeldsen - Proceedings of the Nordic Conference …, 2019

… of compensating for the high data sparsity. Embeddings are induced with 300 dimensions, in order to be compatible with the public Common Crawl-based FastText embeddings. Having induced these embeddings for Bornholmsk …


BTC-2019: The 2019 Billion Triple Challenge Dataset

JM Herrera, A Hogan, T Käfer - International Semantic Web Conference, 2019

… Meusel et al. [43] have published the WebDataCommons, extracting RDFa, Microdata and Microformats from the massive Common Crawl dataset; the result is a collection of 17,241,313,916 RDF triples, which, to the best of …


Building and using parallel text for translation

M Simard - The Routledge Handbook of Translation and …, 2019


Building Knowledge Base through Deep Learning Relation Extraction and Wikidata

… Very large high-quality training data set is then generated automatically by matching Common Crawl data with relation keywords extracted from knowledge database … We solve this problem by matching Common Crawl …


BUILDING TYPE CLASSIFICATION FROM SOCIAL MEDIA TEXTS VIA GEO-SPATIAL TEXT MINING

… We applied the ReLU activation function [18] after each hidden layer and the softmax function after the output layer. The neural network has been trained for 100 epochs and a batch size of 64 samples. 2http …


Building Unbiased Comment Toxicity Classification Model with Natural Language Processing

… For this project, we investigated word embeddings GloVe300D (11) and fastText300D (12), where both are pretrained on Common Crawl. We also implemented character level embedding layer introduced in the paper (13) …


Building Unbiased Comment Toxicity Classification Model

… combined them together to mitigate the potential biases in any embeddings. We experimented with GloVe 300D trained on Common Crawl [3] and fastText 300D trained on Common Crawl [2]. We also implemented …


CamemBERT: a Tasty French Language Model

L Martin, B Muller, PJO Suárez, Y Dupont, L Romary… - arXiv preprint arXiv …, 2019

… Later (Grave et al., 2018) trained fastText word embeddings for 157 languages using Common Crawl and showed that using crawled data significantly increased the performance of the embeddings relatively to those trained only on Wikipedia …


Can Character Embeddings Improve Cause-of-Death Classification for Verbal Autopsy Narratives?

… Page 3. Figure 1: Embedding concatenation model architecture. d1 is the dimensionality of the word embedding (100), and d2 is the dimensionality of the character em- bedding (24). rived from GloVe vectors (Pennington et al., 2014) trained on Common Crawl …


M Kacmajor, JD Kelleher - Language Resources and Evaluation, 2019

Page 1. ORIGINAL PAPER Capturing and measuring thematic relatedness Magdalena Kacmajor1 • John D. Kelleher2 © The Author(s) 2019 Abstract In this paper we explain the difference between two aspects of semantic …


Capturing Discriminative Attributes Using Convolution Neural Network Over ConceptNet Numberbatch Embedding

V Vinayan, MA Kumar, KP Soman - … Research in Electronics, Computer Science and …, 2019

… network. There the model is represented over the various dimensions of GloVe embedding, of those the 300 dimensions (trained over a common crawl corpus of size 840B) embedding performed the best for the task. Building …


Cardiff University at SemEval-2019 Task 4: Linguistic Features for Hyperpartisan News Detection

C Pérez-Almendros, LE Anke, S Schockaert - … of the 13th International Workshop on …, 2019

… GloVe vectors (Pennington et al., 2014) for all the words occurring in them. To this end, we used the un- cased Common Crawl pretrained GloVe embeddings, with 300 dimensions and a vocabulary of 1.9 million words. The …


CatchPhish: detection of phishing websites by inspecting URLs

RS Rao, T Vaishnavi, AR Pais - Journal of Ambient Intelligence and Humanized …, 2019

… 4.1 Dataset We have collected the dataset from three different sources. Legitimate sites are collected from common-crawl and Alexa database whereas phishing sites are collected from PhishTank … D2: Legitimate sites from common-crawl and phishing sites from PhishTank …


Categorising AWS Common Crawl Dataset using MapReduce

A Chiniah, A Chummun, Z Burkutally - 2019 Conference on Next Generation …, 2019

Keeping track of websites connected to the Web is an impossible task given the amplitude and fluctuation of new sites being created and those going offline. In this paper we took the task to create a directory by categorising the websites using …


Categorizing Comparative Sentences

A Panchenko, A Bondarenko, M Franzek, M Hagen…

… containing both items from a web-scale corpus. Our sentence source is the publicly available in- dex of the DepCC (Panchenko et al., 2018), an index of more then 14 billion dependency-parsed English sentences from the Common Crawl filtered for duplicates …


Categorizing Emails Using Machine Learning with Textual Features

F Rudzicz, K Malikov - Advances in Artificial Intelligence: 32nd Canadian …

H Zhang, J Rangrej, S Rais, M Hillmer, F Rudzicz… - Canadian Conference on …, 2019

… Lastly, domain-specific email inboxes such as DDM-Support will contain highly specific subject-matter terminology such as organization acronyms or hospital names, which are generally not present in large text corpora such as common crawl …


CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB

H Schwenk, G Wenzek, S Edunov, E Grave, A Joulin - arXiv preprint arXiv …, 2019

… We are using ten snapshots of a curated common crawl corpus (Wenzek et al., 2019), totaling 32.7 billion unique sentences … we use the same underlying mining approach based on LASER and scale to a much larger …


CCMT 2019 Machine Translation Evaluation Report

M Yang, X Hu, H Xiong, J Wang, Y Jiaermuhamaiti… - China Conference on …, 2019

Y Jiaermuhamaiti, Z He, W Luo, S Huang - … , China, September 27–29, 2019, Revised …, 2019

… (2) English and Chinese monolingual Corpus (Europarl v7/v8, News Commentary, Common Crawl, News Crawl, News Discussions, etc.); LDC for English and Gigaword for Chinese (LDC2011T07, LDC2009T13, LDC2007T07, LDC2009T27) …


CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

G Wenzek, MA Lachaux, A Conneau, V Chaudhary… - arXiv preprint arXiv …, 2019

… (2019) used a large scale dataset based on Common Crawl to train text … Russian, Chinese and Urdu (∆ for average) for BERT-BASE models trained either on Wikipedia or CommonCrawl … We preprocess Common Crawl by …


Characterizing the impact of geometric properties of word embeddings on task performance

B Whitaker, D Newman-Griffis, A Haldar… - arXiv preprint arXiv …, 2019

… 1 3M 300-d GoogleNews vectors from https:// code.google.com/archive/p/ word2vec/ 2 2M 300-d 840B Common Crawl vectors from https: //nlp.stanford.edu/projects/glove/ 3 1M 300-d WikiNews vectors with subword …


CITIZENS IN DATA LAND

… And Indie music'. 3 https://github.com/webis-de/wasp/. 4 Consider a new service provided by The Common Crawl Foundation, http://commoncrawl org/, or, alternatively, a new community service provided via public libraries …


CLaC at clpsych 2019: Fusion of neural features and predicted class probabilities for suicide risk assessment based on online posts

E Mohammadi, H Amini, L Kosseim - Proceedings of the Sixth Workshop on …, 2019

… 2.1 Word Embeddings As shown in Figure 1, GloVe (Pennington et al., 2014) and ELMo (Peters et al., 2018) have been used as pretrained word embeddings. The 300d GloVe word embedder has been pretrained on 840B tokens of web data from Common Crawl …


Classification Approaches to Identify Informative Tweets

P Aggarwal - Proceedings of the Student Research Workshop …, 2019

… After these preprocessing steps, we represent each posting by a dense embedding, created by the mean of the individual words embeddings. We use the pretrained embeddings provided by (Mikolov et al., 2018) …


Classification of Anti-phishing Solutions

S Chanti, T Chithralekha - SN Computer Science, 2020

… WestPac. PhishTank. PIRT report. Legitimate. –. Google whitelist. Manual. Alexa, Yahoo. Web crawler. World Wide Web. WestPac. Common crawl Google search. Data set size. Phishing. 203 Archives. 200 websites. 600 …


Classification of the Answers of the OMT

Page 1. Universität Hamburg Fachbereich Informatik Fakultät für Mathematik, Informatik und Naturwissenschaften Classi cation of the Answers of the OMT Bachelor-Thesis Mensch-Computer-Interaktion Arbeitsbereich …


Classification of virtual patent marking web-pages using machine learning techniques

… Having found the best model, this was then used to predict Common Crawl instances (nonseen observations). Figure 6.1 shows a schema of the first approach. Figure 6.1 – First Approach Framework Adapted from https://www.analyticsvidhya.com …


Classification of Web History Tools Through Web Analysis

JRG Evangelista, DD de Oliveira Gatto, RJ Sassi - International Conference on …, 2019

… 02. Archive.fo. http://archive.fo. 03. CashedPages. http://www.cachedpages. com. 04. CachedView. http://cachedview.com. 05. Common Crawl http://commoncrawl.org. 06. Screenshots.com. http://www.screenshots.com …


Classifying Pastebin Content Through the Generation of PasteCC Labeled Dataset

A Riesco, E Fidalgo, MW Al-Nabki, F Jáñez-Martino… - International Conference on …, 2019

… Panchenko et al. [17] took English text from Common Crawl and constructed a large web-scale corpus using text classification … Panchenko, A., Ruppert, E., Faralli, S., Ponzetto, SP, Biemann, C.: Building a web-scale dependency-parsed corpus from commoncrawl …


Classifying Websites Using Word Vectors and Other Techniques: An Application of Zipf's Law

Page 1. CLASSIFYING WEBSITES USING WORD VECTORS AND OTHER TECHNIQUES: AN APPLICATION OF ZIPF'S LAW A THESIS Presented to the Department of Mathematics and Statistics California State University, Long Beach In Partial Fulfillment …


Clinical Data Extraction and Normalization of Cyrillic Electronic Health Records Via Deep-Learning Natural Language Processing

B Zhao - JCO Clinical Cancer Informatics, 2019

… The medical text in the hematic was modified from the actual patient record for the purposes of illustration. (B) Word embeddings used were fastText embeddings pretrained on Wikipedia and Common Crawl data for English (EN) and Bulgarian (BG) …


Cloze-driven Pretraining of Self-attention Networks

A Baevski, S Edunov, Y Liu, L Zettlemoyer, M Auli - arXiv preprint arXiv:1903.07785, 2019

… We pretrain on individual examples as they oc- cur in the training corpora (§5.1). For News Crawl this is individual sentences while on Wikipedia, Bookcorpus, and Common Crawl examples are paragraph length … Common Crawl …


SM Zobaed, S Ahmad, R Gottumukkala, MA Salehi

Page 1. ClustCrypt: Privacy-Preserving Clustering of Unstructured Big Data in the Cloud SM Zobaed∗, Sahan Ahmad∗, Raju Gottumukkala†, and Mohsen Amini Salehi∗ ∗ School of Computing & Informatics †Informatics Research …


CluWords: Exploiting Semantic Word Clustering Representation for Enhanced Topic Modeling

F Viegas, S Canuto, C Gomes, W Luiz, T Rosa, S Ribas… - Proceedings of the Twelfth …, 2019

… In this section we compare the proposed CluWords with three pre-trained word em- beddings spaces: (i) Word2Vec trained with GoogleNews [21]; (ii) FastText trained with WikiNews [22] and (iii) Fasttext trained on Common Crawl [22] …


CoFiF: A Corpus of Financial Reports in French Language

T Daudert, S Ahmadi - The First Workshop on Financial Technology and …, 2019

… com/CoFiF/Corpus [Merity et al., 2016], or CommonCrawl 2. Considering the domain of business and economics, especially for English, corpora such as the Wall Street Journal (WSJ) Corpus [Paul and Baker, 1992], the …


Cognition and the Structure of Bias

Page 1. UCLA UCLA Electronic Theses and Dissertations Title Cognition and the Structure of Bias Permalink https://escholarship.org/uc/item/7hf582vz Author Johnson, Gabbrielle Michelle Publication Date 2019 Peer reviewed|Thesis/dissertation eScholarship.org …


CogniVal: A Framework for Cognitive Word Embedding Evaluation

N Hollenstein, A de la Torre, N Langer, C Zhang - arXiv preprint arXiv:1909.09001, 2019

… et al., 2018). We evaluate the embeddings with and without subword information trained on 16 billion to- kens of Wikipedia sentences as well as the ones trained on 600 billion tokens of Common Crawl. • ELMo models both …


Collaborative Attention Network with Word and N-Gram Sequences Modeling for Sentiment Classification

J Bao, L Zhang, B Han - International Conference on Artificial Neural Networks, 2019

… words in text. In our experiments, we adopt global vectors for word representation (GloVe) [17] in the embedding layer of our model, which consists of 840 billion 300-dimension tokens trained on Common Crawl. We don't use …


Page 1. Combating Fake News with Adversarial Domain Adaptation and Neural Models by Brian Xu BS, Massachusetts Institute of Technology (2018) Submitted to the Department of Electrical Engineering and Computer Science …


Combining and learning word embedding with WordNet for semantic relatedness and similarity measurement

YY Lee, H Ke, TY Yen, HH Huang, HH Chen - Journal of the Association for …, 2019

… ahttps://nlp.stanford.edu/projects/glove/. bhttp://dumps.wikimedia.org/enwiki/20140102/. chttps://catalog.ldc.upenn.edu/LDC2011T07. dThe Common Crawl corpus contains raw web page data, extracted metadata and text extractions. http://commoncrawl.org …


Comparison of Machine Learning Approaches for Industry Classification Based on Textual Descriptions of Companies

… GloVe vector embeddings are used. Specifically the 300dimensional GloVe vectors trained on the large Common Crawl corpus of 840 billion tokens with a vocabulary of 2.2 million words. While there is no additional training …


Complex Security Policy? A Longitudinal Analysis of Deployed Content Security Policies

S Roth, T Barron, S Calzavara, N Nikiforakis, B Stock

… server response. To determine this IA-specific influence, we chose a second archive service to corroborate the IA's data. In particular, Common Crawl (CC) [10] has been collecting snapshots of popular sites since 2013. For each …


Comprehensive Analysis of Aspect Term Extraction Methods using Various Text Embeddings

Ł Augustyniak, T Kajdanowicz, P Kazienko - arXiv preprint arXiv:1909.04917, 2019

… 1. word2vec - protoplast model of any neural word embedding trained on Google News. 2. glove.840B - Global Vectors for Word Representation proposed by Stanford NLP Group, trained based on Common Crawl with …


Comprehensive trait attributions show that face impressions are organized in four dimensions

… we represented each of them with a vector of 300 computationally extracted semantic features (describing word embeddings and text classification) using a state-of-the-art neural network provided within the FastText library (61) …


Compressing Inverted Indexes with Recursive Graph Bisection: A Reproducibility Study

J Mackenzie, A Mallia, M Petri, JS Culpepper, T Suel

… tool8, – Gov2 is a crawl of .gov domains from 2004, – ClueWeb09 and ClueWeb12 both correspond to the 'B' portion of the 2009 and 2012 ClueWeb crawls of the world wide web, respectively, and – CC-News contains English …


R El Baff, H Wachsmuth, K Al-Khatib, M Stede, B Stein

Page 1. Computational Argumentation Synthesis as a Language Modeling Task Roxanne El Baff 1 Henning Wachsmuth 2 Khalid Al-Khatib 1 Manfred Stede 3 Benno Stein 1 1 Bauhaus-Universität Weimar, Weimar, Germany …


Conceptor Debiasing of Word Representations Evaluated on WEAT

S Karve, L Ungar, J Sedoc - arXiv preprint arXiv:1906.05993, 2019

… For context-independent embeddings, we used off-the-shelf Fasttext subword embeddings6, which were trained with subword information on the Common Crawl (600B tokens), the GloVe embeddings 7 trained on Wikipedia and …


P Dinklage, J Fischer, F Kurpicz - 2020 Proceedings of the Twenty-Second Workshop on …

Page 1. Constructing the Wavelet Tree and Wavelet Matrix in Distributed Memory ∗ Patrick Dinklage † Johannes Fischer† Florian Kurpicz† Abstract The wavelet tree (Grossi et al. [SODA,2003]) is a compact index for texts …


Content Similarity Analysis of Written Comments under Posts in Social Media

… The GloVe vectors were trained from 840 billion tokens of Common Crawl web data and have 300 dimensions [23]. This feature is extracted similar to the Google-word2vec similarity by using equation 6 for each post and comment pair …


Context Matters: Recovering Human Semantic Structure from Machine Learning Analysis of Large-Scale Text Corpora

MC Iordan, T Giallanza, CT Ellis, N Beckage, JD Cohen - arXiv preprint arXiv …, 2019

… We also compared performance of the four Word2Vec embedding spaces to another commonly used embedding space known as GloVe28 for two main reasons; first, the GloVe embeddings are learned from the Common …


Context-Aware Crosslingual Mapping

H Aldarmaki, M Diab - arXiv preprint arXiv:1903.03243, 2019

… data. We trained monolingual ELMo and FastText with de- fault parameters. We used the WMT'13 commoncrawl data for cross-lingual mapping, and the WMT'13 test sets for evaluating sentence translation retrieval. For all …


Continual Learning for Sentence Representations Using Conceptors

T Liu, L Ungar, J Sedoc - arXiv preprint arXiv:1904.09187, 2019

… zero-shot CA. Best results are in boldface and the second best results are underscored. dimensional GloVe vectors (trained on the 840 billion token Common Crawl) (Pennington et al., 2014). Additional experiments with Word2Vec …


CONTRIBUTIONS TO CLINICAL INFORMATION EXTRACTION IN PORTUGUESE: CORPORA, NAMED ENTITY RECOGNITION, WORD EMBEDDINGS

Page 1. Fábio André da Costa Lopes CONTRIBUTIONS TO CLINICAL INFORMATION EXTRACTION IN PORTUGUESE:CORPORA,NAMED ENTITY RECOGNITION,WORD EMBEDDINGS Thesis submitted to the Faculty of Science …


Controlling Grammatical Error Correction Using Word Edit Rate

K Hotate, M Kaneko, S Katsumata, M Komachi - … of the 57th Conference of the …, 2019

… the lowest WER). In the ranking experiment, we used a 5-gram KenLM (Heafield, 2011) with Kneser-Ney smoothing trained on the web-scale Common Crawl corpus (Junczys-Dowmunt and Grundkiewicz, 2016). As an evaluation …


Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading

L Qin, M Galley, C Brockett, X Liu, X Gao, B Dolan… - arXiv preprint arXiv …, 2019

… informative response. To enable reproducibility of our experiments, we crawled web pages using Common Crawl (http://commoncrawl.org), a service that crawls web pages and makes its historical crawls available to the public. We …


Correlation Coefficients and Semantic Textual Similarity

V Zhelezniak, A Savkov, A Shen, NY Hammerla - arXiv preprint arXiv:1905.07790, 2019

… In all experiments we rely on the following publicly available word embeddings: GloVe (Pennington et al., 2014) trained on Common Crawl (840B tokens), fastText (Bojanowski et al., 2017) trained on Common Crawl …


Correlations between Word Vector Sets

V Zhelezniak, A Shen, D Busbridge, A Savkov…

… For methods involving pretrained word embeddings, we use fastText (Bo- janowski et al., 2017) trained on Common Crawl (600B tokens), as previous evaluations have in- dicated that fastText vectors have uniformly the best …


Page 1. Topical Sentence Embeddings Creation of Sentence Embeddings Based on Topical Word Representations Phillip Wenig 160361 Master's thesis to obtain the degree of Master of Science in Information Systems University of Liechtenstein …


Cross-collection Multi-aspect Sentiment Analysis

H Kaporo - Computer Science On-line Conference, 2019

… 3.1, we set \(n=10\), \(m=5\) and run the algorithm over topic-words returned by CPTM and word vectors from glove pre-trained embeddings 2 . These embeddings are trained over common crawl (Google data) and contain 2.2 Million words …


Cross-Domain Sentiment Classification With Bidirectional Contextualized Transformer Language Models

B Myagmar, J Li, S Kimura - IEEE Access, 2019

… For pre-training data, in addition to the BookCorpus and English Wikipedia datasets, cased XLNet-Large model, refered to simply as XLNet henceforth, uses Giga5 (16GB text) [35], ClueWeb 2012-B [36] and Common Crawl [37] as part of its pre-training data …


Cross-Layer Optimization of Big Data Transfer Throughput and Energy Consumption

L Di Tacchio, MDSQZ Nine, T Kosar, MF Bulut… - 2019 IEEE 12th …, 2019

… The algorithms have been compared using four different datasets: i) a small files dataset, including 20,000 HTML files form the Common Crawl project [1]; ii) a medium files dataset, consisting of 5,000 image files from Flickr …


Page 1. Cross-Lingual Alignment of Word & Sentence Embeddings by Hanan Aldarmaki B.Sc. in Computer Engineering, May 2008, The American University of Sharjah M.Phil. in Computer Speech, Text, and Internet Technology, May 2009, University of Cambridge …


Cross-lingual Data Transformation and Combination for Text Classification

J Jiang, S Pang, X Zhao, L Wang, A Wen, H Liu… - arXiv preprint arXiv …, 2019

… There are word vectors for 157 languages1, trained on Common Crawl and Wikipedia, and these models were trained using CBOW with position-weights, in dimension 300, with character n-grams of length 5, a window size of 5 and 10 negatives …


Cross-lingual Parsing with Polyglot Training and Multi-treebank Learning: A Faroese Case Study

J Barry, J Wagner, J Foster - arXiv preprint arXiv:1910.07938, 2019

… the source languages. We use the precomputed Word2Vec embeddings11 released as part of the 2017 CoNLL shared task on UD parsing (Zeman et al., 2017) which were trained on CommonCrawl and Wikipedia. In order to …


Cross-Platform Evaluation for Italian Hate Speech Detection

M Corazza, S Menini, E Cabrio, S Tonelli, S Villata…

… Generic embeddings: we use embedding spaces obtained directly from the Fasttext website4 for Italian. In particular, we use the Italian embeddings trained on Common Crawl and Wikipedia (Grave et al., 2018) with size 300 …


CrossLang: the system of cross-lingual plagiarism detection

O Bakhteev, A Ogaltsov, A Khazov, K Safin… - 2019

… They were obtained from open-source parallel OPUS [46] corpora, but also we mine parallel sentences from Common Crawl.4 Algo … the machine translation stage generates texts that differ too much from 3https://tensorflow …


CUNI Submission for Low-Resource Languages in WMT News 2019

T Kocmi, O Bojar - Proceedings of the Fourth Conference on Machine …, 2019

… Words in English Commoncrawl Russian-English 878k 17.4M 18.8M … Words News crawl 2018 EN 15.4M 344.3M Common Crawl KK 12.5M 189.2M News commentary KK 13.0k 218.7k News crawl Kk 772.9k 10.3M Common Crawl …


CUNI System for the WMT19 Robustness Task

J Helcl, J Libovický, M Popel - arXiv preprint arXiv:1906.09246, 2019

… Corpus # Sentences P arallel 109 English-French Corpus 22,520k Europarl 2,007k News Commentary 200k UN Corpus 12,886k Common Crawl 3,224k M onoFrench News Crawl ('08–'14) 37,320k English News Crawl ('11–'17) 127,554k …


Curriculum Learning for Domain Adaptation in Neural Machine Translation

X Zhang, P Shapiro, G Kumar, P McNamee, M Carpuat… - arXiv preprint arXiv …, 2019

… and WMT 2017 (Bojar et al., 2017), which contains data from several domains, eg parliamentary proceedings (Europarl, UN Parallel Corpus), political/economic news (news commentary, Rapid corpus), and …


Customizing Neural Machine Translation for Subtitling

E Matusov, P Wilken, Y Georgakopoulou - Proceedings of the Fourth Conference on …, 2019

… These data included all other publicly available training data, including ParaCrawl, CommonCrawl, EUbookshop, JRCAcquis, EMEA, and other corpora from the OPUS collection … This was done to avoid oversampling …


D-NET: A Simple Framework for Improving the Generalization of Machine Reading Comprehension

H Li, X Zhang, Y Liu, Y Zhang, Q Wang, X Zhou, J Liu…

… by introducing two-stream self attention. Besides BooksCorpus and Wikipedia, on which the BERT is trained, XLNET uses more corpus in its pretraining, including Giga5, ClueWeb and Common Crawl. In our system, we use …


Page 1. Si ZHANG SIGMA 2018/2019 AUTHÔT 52Av. Pierre Sémard 94200 Ivry-sur-Seine Détection automatique de la thématique et adaptation des modèles de langage from 01/03/2019 to 16/08/2019 Confidentiality : yes …


AE Lillie, ER Middelboe - arXiv preprint arXiv:1907.01304, 2019

Page 1. IT University of Copenhagen MSc in Software Development Thesis project KISPECI1SE Danish Stance Classification and Rumour Resolution Authors: Anders E. Lillie aedl@itu.dk Emil R. Middelboe erem@itu.dk …


Page 1. Masterarbeit zur Erlangung des akademischen Grades Master of Arts der Philosophischen Fakultät der Universität Zürich Data Augmentation in Deep Learning for Hate Speech Detection in Lower Resource Settings …


Data4UrbanMobility: Towards Holistic Data Analytics for Mobility Applications in Urban Regions

N Tempelmeier, Y Rietz, I Lishchuk, T Kruegel… - arXiv preprint arXiv …, 2019

… Web Event-centric Web markup Annotated Web pages, eg us- ing schema.org. Web Data Commons event subset: 263 × 106 facts until November 2017 Common Crawl ToU RDFa, MicroData Focused crawls Event-centric crawls, news11 …


Debiasing Embeddings for Reduced Gender Bias in Text Classification

F Prost, N Thain, T Bolukbasi - Proceedings of the First Workshop on Gender Bias in …, 2019

… 2 Classification Task This work utilizes the BiosBias dataset introduced in (De-Arteaga et al., 2019). This dataset consists of biographies identified within the Common Crawl 397,340 biographies were extracted from sixteen crawls from 2014 to 2018 …


DebiasingWord Embeddings Improves Multimodal Machine Translation

T Hirasawa, M Komachi - arXiv preprint arXiv:1905.10464, 2019

Page 1. Debiasing Word Embeddings Improves Multimodal Machine Translation Tosho Hirasawa Tokyo Metropolitan University hirasawa-tosho@ed.tmu.ac.jp Mamoru Komachi Tokyo Metropolitan University komachi@tmu.ac.jp Abstract …


X Shi, Z Ke, Y Zhou, H Jin, L Lu, X Zhang, L He, Z Hu… - ACM Transactions on …, 2019

Page 1. 3 Deca: A Garbage Collection Optimizer for In-Memory Data Processing XUANHUA SHI and ZHIXIANG KE, Huazhong University of Science and Technology, China YONGLUAN ZHOU, University of Copenhagen, Denmark …


DECO: A Dataset of Annotated Spreadsheets for Layout and Table Recognition

E Koci, M Thiele, J Rehak, O Romero, W Lehner - the 15th IAPR International …, 2019

… 50 files). The performance is manually assessed per file. 1http://info.nuix.com/Enron. html 2http://commoncrawl.org/ 3http://lemurproject.org/clueweb09.php/ Koci et al. [15] use a dataset of 216 annotated spreadsheets. Unlike …


Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing--A Tale of Two Parsers Revisited

A Kulmizev, M de Lhoneux, J Gontrum, E Fano, J Nivre - arXiv preprint arXiv …, 2019

… 2018), who train ELMo on 20 million words randomly sampled from raw WikiDump and Common Crawl datasets for … In other words, while the standalone ELMo models were trained on the tokenized WikiDump and …


Page 1. Uday Kamath · John Liu · James Whitaker Deep Learning for NLP and Speech Recognition Page 2. Deep Learning for NLP and Speech Recognition Page 3. Uday Kamath • John Liu • James Whitaker Deep …


Deep learning for pollen allergy surveillance from twitter in Australia

J Rong, S Michalska, S Subramani, J Du, H Wang - BMC Medical Informatics and …, 2019

… embeddings - as alternative. The pre-trained Common Crawl 840B tokens GloVe embeddings were downloaded from the website 2 . Both 50 dimensions (min) and 300 dimensions (max) options were tested. The HF embeddings …


A Hannun, C Case, J Casper, B Catanzaro, G DIAMOS… - US Patent App. 16/542,243, 2019

US20190371298A1 - Deep learning models for speech recognition - Google Patents. Deep learning models for speech recognition. Download PDF Info. Publication number US20190371298A1. US20190371298A1 US16/542,243 …


Deep Learning vs. Classic Models on a New Uzbek Sentiment Analysis Dataset

… We use as input the FastText pre-trained word embeddings of size 300 (Grave et al., 2018) for Uzbek language, that were created from Wiki pages and CommonCrawl, 9 which, to our knowledge, are the only available pre-trained …


Deep Learning-based Categorical and Dimensional Emotion Recognition for Written and Spoken Text

BT Atmaja, K Shirai, M Akagi - INA-Rxiv. June, 2019

… meaning. Glove captured the global corpus statistics from the corpus, for example, a Wikipedia document or a common crawl document. In GloVe model, the cost function is given by V ∑ i,j=1 f(Xi,j)(uT i,jvj + bi + cj − log Xi,j)2 (2) …


Deep Structured Semantic Model for Recommendations in E-commerce

A Larionova, P Kazakova, N Nikitinsky - International Conference on Hybrid Artificial …, 2019

… We generated a vector representation for each text by inferring FastText embeddings [4] from their tokens and averaging them (FastText model is pretrained on the Russian language subset of the Common Crawl corpus [10]) …


Deepening Hidden Representations from Pre-trained Language Models for Natural Language Understanding

J Yang, H Zhao - arXiv preprint arXiv:1911.01940, 2019

… the other hand. In addition to BooksCorpus and English Wikipedia, it also uses Giga5, ClueWeb 2012-B and Common Crawl for pre-training. Trained with dynamic masking, large mini-batches and a larger bytelevel BPE, full …


Defending Against Neural Fake News

R Zellers, A Holtzman, H Rashkin, Y Bisk, A Farhadi… - arXiv preprint arXiv …, 2019

… Dataset. We present RealNews, a large corpus of news articles from Common Crawl … Thus, we construct one by scraping dumps from Common Crawl, limiting ourselves to the 5000 news domains indexed by Google News …


Deliverable 4.2: Data Integration (v. 1)

A Haller, JD Fernández, A Polleres, MR Kamdar - Work, 2019

Page 1. Cyber-Physical Social Systems for City-wide Infrastructures Deliverable 4.2: Data Integration (v.1) Authors : Armin Haller, Javier D. Fernández, Axel Polleres, Maulik R. Kamdar Dissemination Level : Public Due date …


Design and implementation of an open source Greek POS Tagger and Entity Recognizer using spaCy

E Partalidou, E Spyromitros-Xioufis, S Doropoulos… - IEEE/WIC/ACM International …, 2019

… 3.4 Evaluation and comparison of results In the first experiment the model was trained using pretrained vectors extracted from two different sources, Common Crawl and Wikipedia and can be found at the official FastText …


Detecting Aggression and Toxicity using a Multi Dimension Capsule Network

S Srivastava, P Khurana - Proceedings of the Third Workshop on Abusive …, 2019

… The code for tokenization was taken from (Devlin et al., 2018) which seems to properly separate the word tokens and special characters. For training all our classification models, we have used fastText embeddings of dimension 300 trained on a common crawl …


Detecting associations between dietary supplement intake and sentiments within mental disorder tweets

Y Wang, Y Zhao, J Zhang, J Bian, R Zhang - Health Informatics Journal, 2019

Many patients with mental disorders take dietary supplement, but their use patterns remain unclear. In this study, we developed a method to detect signals of associations between dietary supplement...


Detecting Clitics Related Orthographic Errors in Turkish

… For this task, GloVe was used with the dimension size of 300 and window size of 15. The pretrained word vectors for Turkish were obtained from the model trained on Common Crawl and Wikipedia using fastText (Grave et al., 2018) …


Detecting Hacker Threats: Performance of Word and Sentence Embedding Models in Identifying Hacker Communications

… model Source Dim. size MDL-1 SVM WEMB Word2vec Google News 300 MDL-2 SVM WEMB Glove Common Crawl 300 … MDL-4 SVM SEMB InferSent Wikipedia 4096 MDL-5 SVM SEMB SentEncoder Wiki, Web News, SNLI …


Detecting Incivility and Impoliteness in Online Discussions. Classification Approaches for German User Comments.

Page 1. DETECTING INCIVILITY AND IMPOLITENESS Preprint on SSRN, 22.11.2019 Anke Stoll HHU Düsseldorf anke.stoll@hhu.de Marc Ziegele HHU Düsseldorf marc.ziegele@hhu.de Oliver Quiring JGU Main quiring@uni-mainz.de ABSTRACT …


A de Bruijn, V Muhonen, T Albinonistraat, W Fokkink… - 2019

Page 1. Detecting offensive language using transfer learning Alissa de Bruijn September 2019 Page 2. Master Thesis Business Analytics Detecting offensive language using transfer learning Author: Alissa de Bruijn …


Detecting Relational States in Online Social Networks

J Zhang, L Tan, X Tao, T Pham, X Zhu, H Li, L Chang - 2018 5th International …, 2019

… Since the social network we obtain from the repositories of common crawl contains missing links and partial information, stochastic estimations are used to measure the accuracy and reliability of our experimental MVVA results [19] …


Detecting Topic-Oriented Speaker Stance in Conversational Speech}}

C Lai, B Alex, JD Moore, L Tian, T Hori, G Francesca - Proc. Interspeech 2019, 2019

… 3.3. Lexical Features We build lexical representations over turns and topic segments using 300 dimensional GloVe word embeddings (Common Crawl, 840B tokens) [26]. We perform basic tokenization to map between the CallHome transcripts and word embeddings …


Page 1. BACHELOR THESIS Yuliya Yamalutdinova Detection of contradictions in pairs of texts in Kazakh Institute of Formal and Applied Linguistics Supervisor of the bachelor thesis: Mgr. Rudolf Rosa, Ph.D. Study programme: Computer Science …


Determining How Citations Are Used in Citation Contexts

M Färber, A Sampath - International Conference on Theory and Practice of …, 2019

… 2. See https://fasttext.cc/. The pretrained vectors were trained on Common Crawl and Wikipedia using the CBOW model of fastText. fastText operates at the character level, and therefore can generate vectors for words not seen in the training corpus …


Development of a Song Lyric Corpus for the English Language

MAG Rodrigues, A de Paiva Oliveira, A Moreira - International Conference on …, 2019

… languages. According to the authors, the texts that compose the corpus were extracted from CommonCrawl (commoncrawl.org), the largest publicly available general Web crawl to date with about 2 billion crawled URLs. The …


Page 1. Development of an End-to-End Deep Learning Pipeline Matthias Nitsche, Stephan Halbritter {matthias.nitsche, stephan.halbritter}@hawhamburg.de Hamburg University of Applied Sciences, Department of …


Digital audio track suggestions for moods identified using analysis of objects in images from video content

N Brochu - US Patent App. 15/392,705, 2019

… the response from the natural language model 240. Exemplary training corpuses of words can include, for example, Common Crawl (eg, 840B tokens, 2.2M vocabulary terms). The natural language model 240 can be improved …


G Bella, F McNeill, D Leoni, FJQ Real, F Giunchiglia - Journal on Data Semantics, 2019

Page 1. Journal on Data Semantics https://doi.org/10.1007/s13740-019-00107-1 ORIGINAL ARTICLE Diversicon: Pluggable Lexical Domain Knowledge Gábor Bella1 · Fiona McNeill2 · David Leoni1 · Francisco José Quesada Real3 · Fausto Giunchiglia1 …


Do It Like a Syntactician: Using Binary Gramaticality Judgements to Train Sentence Encoders and Assess Their Sensitivity to Syntactic Structure

Page 1. City University of New York (CUNY) CUNY Academic Works Dissertations, Theses, and Capstone Projects Graduate Center 9-2019 Do It Like a Syntactician: Using Binary Gramaticality Judgements to Train …


Do It Like a Syntactician: Using Binary Grammaticality Judgements to Train Sentence Encoders and Assess Their Sensitivity to Syntactic Structure

Page 1. Do it like a syntactician: using binary grammaticality judgments to train sentence encoders and assess their sensitivity to syntactic structure by Pablo González Mart´ınez A dissertation submitted to the Graduate Faculty in Linguistics in partial fulfillment of the …


Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?

I Vulić, G Glavaš, R Reichart, A Korhonen - arXiv preprint arXiv:1909.01638, 2019

… Monolingual Embeddings. We use the 300-dim vectors of Grave et al. (2018) for all 15 languages, pretrained on Common Crawl and Wikipedia with fastText (Bojanowski et al., 2017).7 We trim all 5While BLI is an intrinsic task, as discussed by Glavaš et al …


Document Embedding Models on Environmental Legal Documents

… Instead of having aa large vocabulary of pre-computed word embeddings trained on Wikipedia and Common Crawl, this newly trained model is trained on documents from a more specific domain - resulting in a vocabulary …


Document Summarization Using Sentence-Level Semantic Based on Word Embeddings

K Al-Sabahi, Z Zuping - International Journal of Software Engineering and …, 2019

… Training is performed on aggregated global word-word co-occurrence statistics from a corpus. In this work, we use the one trained on Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB download): glove.840B. 300d.zip …


Domain adaptation for part-of-speech tagging of noisy user-generated text

L März, D Trautmann, B Roth - arXiv preprint arXiv:1905.08920, 2019

… The pretrained vectors for German are based on Wikipedia articles and data from Common Crawl3. We obtain 97.988 different embeddings for the tokens in TIGER and the Twitter corpus of which 75.819 were already contained …


Domain-specific word embeddings for patent classification

J Risch, R Krestel - Data Technologies and Applications, 2019

… used to train word embeddings. It contains more than twice the number of tokens of the English Wikipedia (16bn) and is only exceeded by the Common Crawl data set, which consists of 600bn tokens. We assume that the embeddings …


Dual Monolingual Cross-Entropy Delta Filtering of Noisy Parallel Data

A Axelrod, A Kumar, S Sloto - Proceedings of the Fourth Conference on Machine …, 2019

… gual English corpus comparable in size and content to the Sinhala one by randomly selecting 150k lines from Wikipedia and 6M lines from Common Crawl … Each SentencePiece model was trained on 1M lines of monolin …


Dynamic Packed Compact Tries Revisited

K Tsuruta, D Köppl, S Kanda, Y Nakashima, S Inenaga… - arXiv preprint arXiv …, 2019

… name column. • commoncrawl is a web crawl containing the ASCII-encoded content (without HTML tags) of random web pages extracted from Common Crawl. • vital is the main text extracted from the most vital Wikipedia articles …


Page 1. Dynamically Route Hierarchical Structure Representation to Attentive Capsule for Text Classification Wanshan Zheng1,2 , Zibin Zheng1,2 , Hai Wan1 , Chuan Chen1,2 1School of Data and Computer Science, Sun Yat …


EasyChair Preprint

… Fancellu et al. (2016) show in their work that Page 4. Crawl data set with 840B tokens. Additionally we will try 300-dimensional pre-trained fastText2 that were also trained on Common Crawl but on a subset of 600B tokens. This differs from Fancellu et al …


EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks

JW Wei, K Zou - arXiv preprint arXiv:1901.11196, 2019

… We suspect that EDA will work with any thesaurus. Word embeddings. We use 300-dimensional Common-Crawl word embeddings trained using GloVe (Pennington et al., 2014). We suspect that EDA will work with any pre-trained word embeddings. CNN …


Edge Computing for User-Centric Secure Search on Cloud-Based Encrypted Big Data

S Ahmad, SM Zobaed, R Gottumukkala, MA Salehi - arXiv preprint arXiv:1908.03668, 2019

… We used two datasets, namely Amazon Common Crawl Corpus (ACCC) [35] and Request For Comments (RFC) [36], that have distinct characteristics and volumes. ACCC is ≈ 150 terabytes, contains web contents, and is not domain-specific …


EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction

D Bouchacourt, L Denoyer - arXiv preprint arXiv:1905.11852, 2019

… We test on the full test dataset composed of 7, 600 samples. We use pre-trained word vectors trained on Common Crawl [8], and keep them fixed … We use pre-trained word vectors trained on Common Crawl [8], and keep them fixed …


Page 1. MASTER'S THESIS Efficient Classification and Unsupervised Keyphrase Extraction for Web Pages Tim Haarman s2404184 Department of Artificial Intelligence University of Groningen, The Netherlands Primary …


Efficient Contextual Representation Learning With Continuous Outputs

LH Li, PH Chen, CJ Hsieh, KW Chang - Transactions of the Association for …, 2019

Create a new account. Email. Returning user. Can't sign in? Forgot your password? Enter your email address below and we will send you the reset instructions. Email. Cancel. If the address matches an existing account you will …


Efficient Contextual Representation Learning Without Softmax Layer

LH Li, PH Chen, CJ Hsieh, KW Chang - arXiv preprint arXiv:1902.11269, 2019

… The output layer is a sampled softmax with 8192 negative samples per batch. This model is provided in AllenNLP by Peters et al. (2018a). • ELMO-S: The input layer is the FastText embedding trained on Common Crawl (Mikolovetal …


Efficient Sentence Embedding using Discrete Cosine Transform

N Almarwani, H Aldarmaki, M Diab - arXiv preprint arXiv:1909.03104, 2019

… CoordInv Coordination Inversion Table 1: Probing Tasks 3.2 Experimental setup For the word embeddings, we use pre-trained FastText embeddings of size 300 (Mikolov et al., 2018) trained on Common-Crawl. We generate DCT …


Embedding Imputation with Grounded Language Information

Z Yang, C Zhu, V Sachidananda, E Darve - arXiv preprint arXiv:1906.03753, 2019

… KG2Vec 0.02% 7% 0.04% 12% 58.6 56.9 60.1 54.3 GloVe Common Crawl 1% 29% 2% 44% 44.0 33.0 45.1 27.3 … We test on two types of pre-trained word vectors GloVe (Common crawl, cased 300d) and ConceptNet Numberbatch (300d) …


EmbNum+: Effective, Efficient, and Robust Semantic Labeling for Numerical Values

P Nguyen, K Nguyen, R Ichise, H Takeda - New Generation Computing, 2019

… Data portals. For example, 233 million tables were extracted from the July 2015 version of the Common Crawl [10].1 Additionally, 200,000 tables from 232 Open Data portals were analyzed by Mitlohner et al. [12]. These resources …


Emerging Cross-lingual Structure in Pretrained Language Models

A Conneau, S Wu, H Li, L Zettlemoyer, V Stoyanov - … of the 58th Annual Meeting of …, 2020

S Wu, A Conneau, H Li, L Zettlemoyer, V Stoyanov - arXiv preprint arXiv:1911.01464, 2019

… We consider domain difference by training on Wikipedia for English and a random subset of Common Crawl of the same size for the other languages (Wiki-CC). We also consider a model trained with Wikipedia only, the same as XLM (Default) for comparison …


Emoji Powered Capsule Network to Detect Type and Target of Offensive Posts in Social Media

… Also character embeddings handle in- frequent words better than word2vec embedding as later one suffers from lack of enough training opportunity for those rare words. We used fasttext embeddings pre trained on Common Crawl (Mikolov et al., 2018) …


EmoLabel: Semi-Automatic Methodology for Emotion Annotation of Social Media Text

L Canales, W Daelemans, E Boldrini, P Martínez-Barco - IEEE Transactions on …, 2019

Page 1. 1949-3045 (c) 2019 IEEE. Personal use is permitted, but republication/ redistribution requires IEEE permission. See http://www.ieee.org/ publications_standards/publications/rights/index.html for more information. This …


EMOMINER at SemEval-2019 Task 3: A Stacked BiLSTM Architecture for Contextual Emotion Detection in Text

N Chakravartula, V Indurthi - Proceedings of the 13th International Workshop on …, 2019

… them with GloVe vectors. As a result, Glove vectors will have syntactic information of words (Rezaeinia et al., 2017). 3.2 Feature Extraction • Word Embeddings: Glove840B - common crawl (Pennington et al., 2014) pre-trained …


Emotional Embeddings: Refining Word Embeddings to Capture Emotional Content of Words

A Seyeditabari, N Tabari, S Gholizadeh, W Zadrozny - arXiv preprint arXiv …, 2019

… vector spaces used here are: • Word2Vec trained full English Wikipedia dump • GloVe from their own website • fastText trained with subword information on Common Crawl • ConceptNet Numberbatch It is clear that each emotionally …


Encoder-Decoder Network with Cross-Match Mechanism for Answer Selection

Z Xie, X Yuan, J Wang, S Ju - China National Conference on Chinese Computational …, 2019

… 4.2 Implementation Details. We initialized word embedding with 300d-GloVe vectors pre-trained from the 840B Common Crawl corpus [8], while the word embeddings for the out-of-vocabulary words were initialized randomly …


End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots

… We used the pretrained word embedding model GloVe, trained on the Common Crawl corpus [5]. The dimension of word embedding is 300, and a zero vector is used for unknown words. A gesture is represented as a sequence of human poses …


End-to-end Neural Information Retrieval

Page 1. End-to-end Neural Information Retrieval by Wei Yang A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Master in Computer Science Waterloo, Ontario, Canada, 2019 c Wei Yang 2019 Page 2 …


End-to-End Speech Recognition

U Kamath, J Liu, J Whitaker - Deep Learning for NLP and Speech Recognition, 2019

… of certain words. Therefore, an n-gram language model was trained using the KenLM [Hea+ 13] toolkit on the Common Crawl Repository, 1 using the 400,000 most frequent words from 250 million lines of text. The decoding …


English-Czech Systems in WMT19: Document-Level Transformer

M Popel, D Macháček, M Auersperger, O Bojar… - arXiv preprint arXiv …, 2019

… brevity. sentence words (k) data set pairs (k) EN CS CzEng 1.7 57 065 618 424 543 184 Europarl v7 647 15 625 13 000 News Commentary v12 211 4 544 4 057 CommonCrawl 162 3 349 2 927 WikiTitles 361 896 840 EN NewsCrawl …


Enhancing AMR-to-Text Generation with Dual Graph Representations

LFR Ribeiro, C Gardent, I Gurevych - arXiv preprint arXiv:1909.00352, 2019

… 5 Experiments and Discussion Implementation Details We extract vocabularies (size of 20,000) from the training sets and initialize the node embeddings from GloVe word em- beddings (Pennington et al., 2014) on Common Crawl …


Enhancing Semantic Word Representations by Embedding Deeper Word Relationships

A Nugaliyadde, KW Wong, F Sohel, H Xie - arXiv preprint arXiv:1901.07176, 2019

… 2 https://commoncrawl.org/ differentiating similarity from association and relatedness which is reflected in table 1. The context to create the word embedding in order to test on SimLex-999 is created based on Common …


Environmental hazards, rigid institutions, and transformative change: How drought affects the consideration of water and climate impacts in infrastructure management

N Ulibarri, TA Scott - Global Environmental Change, 2019


eTranslation's Submissions to the WMT 2019 News Translation Task

C Oravecz, K Bontcheva, A Lardilleux, L Tihanyi… - Proceedings of the Fourth …, 2019

… En→De the reduction in ParaCrawl was from 31M to 18M segments and in CommonCrawl from 2.3M to 1.4M segments with a drop of 0.2 BLEU points compared to us- ing the full sets3. No additional cleaning was ap- plied to …


Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates

J Iranzo-Sánchez, JA Silvestre-Cerdà, J Jorge… - arXiv preprint arXiv …, 2019

… De↔Fr eubookshop, JRC-Acquis, 14.3 TildeModel En↔Es commoncrawl, eubookshop, 21.1 EU-TT2, UN, Wikipedia En↔Fr commoncrawl, giga, 38.2 undoc, news-commentary Es↔Fr DGT, eubookshop, 37.2 JRC-Acquis, UNPC …


Evaluating Commonsense in Pre-trained Language Models

X Zhou, Y Zhang, L Cui, D Huang - arXiv preprint arXiv:1911.11931, 2019

… Note that XLNet-base is trained with the same data as BERT, while XLNet-large is trained with a larger dataset that consists of 32.98B subword pieces coming from Wiki, BookCorpus, Giga5, ClueWeb, and Common Crawl. RoBERTa (Liu et al …


Evaluating KGR10 Polish word embeddings in the recognition of temporal expressions using BiLSTM-CRF.

J Kocoń, M Gawor - arXiv preprint arXiv:1904.04055, 2019

… The second one, called FASTTEXT4, is original FastText word embeddings set, created for 157 languages (including Polish). Authors used Wikipedia and Common Crawl5 as the linguistic data source … C2 Common Crawl …


Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models

C Hokamp, J Glover, D Gholipour - arXiv preprint arXiv:1906.09675, 2019

… We use all available parallel data from the WMT19 news-translation task for training, with the exception of commoncrawl, which we found to be very noisy after manually checking a sample of the data, and paracrawl, which …


Evaluation of basic modules for isolated spelling error correction in Polish texts

S Rutkowski - arXiv preprint arXiv:1905.10810, 2019

… How this representation is constructed is informed by the whole corpus on which the embedder was trained. The pretrained ELMo model that we used (Che et al., 2018) was trained on Wikipedia and Common Crawl corpora of Polish …


Evaluation of Czech Distributional Thesauri

P Rychlý - RASLAN 2019 Recent Advances in Slavonic Natural …, 2019

… The results are summarized in Table 2. The czTenTen12 corpus was evaluated with Sketch Engine thesaurus and also with word vectors compiled by FastText. We have also included prebuild model from Common Crawl. Table …


Evaluation of State Of Art Open-source ASR Engines with Local Inferencing

Page 1. Institute Of Information Systems (iisys) Hof University in exchange for Media Engineering and Technology Faculty German University in Cairo Evaluation of State Of Art Open-source ASR Engines with Local …


Evaluation of vector embedding models in clustering of text documents

… The second group of sources of word2vec models for Polish are web pages of word embedding tools like fastText, ELMo and BERT. They were trained on Polish Common Crawl and Wikipedia. However, the BERT …


Event-Argument Linking in Hindi for Information Extraction in Disaster Domain

SK Sahoo, S Saha, A Ekbal, P Bhattacharyya…

… 3.3 Word Embedding For word embedding (WE) of each word, we use pre-trained fastText [5] word vectors. These embeddings were trained on Hindi Common Crawl and Wikipedia dataset. The size of the word embedding used in our experiments is 300 …


Evolution of the PAN Lab on Digital Text Forensics

P Rosso, M Potthast, B Stein, E Stamatatos, F Rangel… - … Retrieval Evaluation in a …, 2019

… The static web search environment is comprised of the web search engine ChatNoir, which indexes the ClueWeb 2009, the ClueWeb 2012, and (as of 2017) the CommonCrawl, delivering search results in milliseconds while …


Example-Driven Question Answering

Page 1. August 19, 2019 DRAFT Example-Driven Question Answering Di Wang August 2019 School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee: Eric Nyberg (chair) (Carnegie Mellon …


Explicit Discourse Argument Extraction for German

P Bourgonje, M Stede - International Conference on Text, Speech, and …, 2019

… The word embeddings are trained on Common Crawl and Wikipedia [6]. We generated the part-of-speech embeddings from the TIGER corpus [4]. We use a CNN with four fully connected layers. Training this on all classes from Table 1 results in an accuracy of 94.52 …


Exploitation vs. exploration—computational temporal and semantic analysis explains semantic verbal fluency impairment in Alzheimer's disease

J Tröger, N Linz, A König, P Robert, J Alexandersson… - Neuropsychologia, 2019

… For deriving semantic metrics, the semantic distance between produced words was calculated based on a fastText (Joulin et al., 2016) neural word embedding, pre-trained on the French Common Crawl and Wikipedia corpora (Grave et al., 2018; Linz et al., 2017) …


Exploiting EuroVoc's Hierarchical Structure for Classifying Legal Documents

E Filtz, S Kirrane, A Polleres, G Wohlgenannt - … " On the Move to Meaningful Internet …, 2019

G Wohlgenannt - On the Move to Meaningful Internet Systems: OTM …

… First, we tested large-scale pre-trained language models trained with generalpurpose text corpora such as GoogleNews and the CommonCrawl, but as expected both performed badly on the legal dataset, for example the Common …


Exploiting knowledge graphs for entity-centric prediction

Page 1. © 2018 Shan Jiang Page 2. EXPLOITING KNOWLEDGE GRAPHS FOR ENTITY-CENTRIC PREDICTION BY SHAN JIANG DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of …


Exploiting Temporal Relationships in Video Moment Localization with Natural Language

S Zhang, J Su, J Luo - arXiv preprint arXiv:1908.03846, 2019

… extracted from VGG [24] fc7 layer, optical flow features are extracted from the penultimate layer [27] and the 300-d Glove feature [21] pretrained on Common Crawl (42 billion tokens) are used as the word embedding. The segment …


Exploiting the Hierarchical Structure of a Thesaurus for Document Classification

E Filtz, S Kirrane, A Polleres, G Wohlgenannt

… First, we tested large-scale pre-trained language models trained with generalpropose text corpora such as GoogleNews and the CommonCrawl, but as ex- pected both performed badly on the legal dataset, for example the …


Explore FREDDY

M Günther, M Thiele, W Lehner, Z Yanakiev - BTW 2019, 2019

… The configuration of the search function is defined in the sidebar (Figure 3b) just as in the query view. 4 Screencast on our FREDDY website https://wwwdb.inf.tu-dresden.de/research-projects/freddy/ 5 https://dblp.uni …


Explore FREDDY: Fast Word Embeddings in Database Systems

M Günther, Z Yanakiev, M Thiele, W Lehner


Exploring Numeracy in Word Embeddings

A Naik, A Ravichander, C Rose, E Hovy - Proceedings of the 57th Conference of the …, 2019

… FastText (Bojanowski et al., 2017): Extended Skipgram model representing words as character n-grams to incorporate sub-word information. We evaluate Wikipedia and Common Crawl variants. 3.1 Retrained Word Vectors …


Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

C Raffel, N Shazeer, A Roberts, K Lee, S Narang… - arXiv preprint arXiv …, 2019

… unsupervised pre-training for NLP is particularly attractive because unlabeled text data is available en masse thanks to the Internet – for example, the Common Crawl project2 produces about 20TB of text data extracted from web pages each month …


Extending Cross-Domain Knowledge Bases with Long Tail Entities using Web Table Data

… In a second experiment, we apply the system to a large corpus of web tables extracted from the Common Crawl. This experiment allows us to get an overall im- pression of the potential of web tables for augmenting knowledge bases with long tail entities …


Extracting and Analyzing Context Information in User-Support Conversations on Twitter

D Martens, W Maalej - arXiv preprint arXiv:1907.13395, 2019

… As the list of marketing names also includes common words (eg, 'five', 'go', or 'plus'), we used the natural language processing library spaCy [33] to remove words that appear in the vocabulary of the included …


Extracting Novel Facts from Tables for Knowledge Graph Completion (Extended version)

B Kruit, P Boncz, J Urbani - arXiv preprint arXiv:1907.00083, 2019

… The first one is the T2D dataset [23], which contains a subset of the WDC Web Tables Corpus – a set of tables extracted from the CommonCrawl web scrape6. We use the latest available version of this dataset (v2, released 2017/02). In …


Extracting Novel Facts from Tables for Knowledge Graph Completion

B Kruit, P Boncz, J Urbani - International Semantic Web Conference, 2019

… The first one is the T2D dataset [25], which contains a subset of the WDC Web Tables Corpus – a set of tables extracted from the CommonCrawl web scrape 2 . We use the latest available version of this dataset (v2, released 2017/02) …


Facebook AI's WAT19 Myanmar-English Translation Task Submission

PJ Chen, J Shen, M Le, V Chaudhary, A El-Kishky… - arXiv preprint arXiv …, 2019

… For Myanmar language, we take five snapshots of the Commoncrawl dataset and combine them with the raw data from Buck et al. (2014) … The Myanmar monolingual data we collect from Commoncrawl contains text in both Unicode and Zawgyi encodings …


Facebook FAIR's WMT19 News Translation Task Submission

N Ng, K Yee, A Baevski, M Ott, M Auli, S Edunov - arXiv preprint arXiv:1907.06616, 2019

… We train two language models LI and LN on Newscrawl and Commoncrawl respectively, then score every sentence s in Commoncrawl by HI(s)−HN (s). We select a cu- toff of 0.01, and use all sentences that score higher than …


Facilitating access to health web pages with different language complexity levels

M Alfano, B Lenzitti, D Taibi, M Helfert - 2019

… The Web Data Commons (WDC) (Meusel, 2014) contains all Microformat, Microdata and RDFa data extracted from the open repository of web crawl data named Common Crawl (CC)16 … 15 http://webdatacommons.org/ 16 http://commoncrawl.org …


Fast and Accurate Network Embeddings via Very Sparse Random Projection

H Chen, SF Sultan, Y Tian, M Chen, S Skiena - arXiv preprint arXiv:1908.11512, 2019

… WWW-200K and WWW-10K [11]: these graphs are derived from the Web graph provided by Common Crawl, where the nodes are hostnames and the edges are the hyperlinks between these websites. For simplicity, we treat this graph as an undirected graph …


Faster Neural Network Training with Data Echoing

D Choi, A Passos, CJ Shallue, GE Dahl - arXiv preprint arXiv:1907.05550, 2019

… 2http://commoncrawl.org/2017/07/june-2017-crawl-archive-now-available/ 3Each time a training example is read from disk, it counts as a fresh example. 420k steps for LM1B, 60k for Common Crawl, 110k for ImageNet, 150k for CIFAR-10, and 30k for COCO …


Y Zhang, A Azad, Z Hu - arXiv preprint arXiv:1910.05971, 2019

Page 1. FastSV: A Distributed-Memory Connected Component Algorithm with Fast Convergence Yongzhe Zhang ∗ Ariful Azad † Zhenjiang Hu ‡ Abstract This paper presents a new distributed-memory algorithm called FastSV …


FastText-Based Intent Detection for Inflected Languages

… For the word embeddings released by Facebook, we used the ones trained on Wikipedia (https: //fasttext.cc/docs/en/pretrained-vectors.html) because the ones trained on Common Crawl (https: //fasttext.cc/docs/en/crawl-vectors.html) showed inferior results in our tests …


Feature Engineering for Text Representation

D Sarkar - Text Analytics with Python, 2019

In the previous chapters, we saw how to understand, process, and wrangle text data. However, all machine learning or deep learning models are limited because they cannot understand text data directly...


Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labels

L Lange, MA Hedderich, D Klakow - arXiv preprint arXiv:1910.06061, 2019

… clustering. While the Brown clustering was trained on the relatively small Europarl corpus, k- Means clustering seems to benefit from the word embeddings trained on documents from the much larger common crawl. 7 Analysis …


Feature2Vec: Distributional semantic modelling of human property knowledge

S Derby, P Miller, B Devereux - arXiv preprint arXiv:1908.11439, 2019

… For our experiments, we make use of the pretrained GloVe embeddings (Pennington et al., 2014) provided in the Spacy1 package trained on the Common Crawl2. The GloVe model includes 685,000 tokens … 1https://spacy …


Feeling Anxious? Perceiving Anxiety in Tweets using Machine Learning

D Gruda, S Hasan - Computers in Human Behavior, 2019

… tweets. Words-to-vectors mapping is based on the deep neural network learning GloVe (Pennington, Socher, & Manning, 2014) embedding space built from the Common Crawl Web Data (42 Billion tokens, 1.9M vocab). The …


FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms

HB Moss, A Moore, DS Leslie, P Rayson - arXiv preprint arXiv:1906.12230, 2019

… optimiser settings and the same regularisation. All words are lower cased and we use the same Glove common crawl 840B token 300 dimension word embedding (Pennington et al., 2014). We use variational (Gal and Ghahramani …


Figurative Usage Detection of Symptom Words to Improve Personal Health Mention Detection

A Iyer, A Joshi, S Karimi, R Sparks, C Paris - arXiv preprint arXiv:1906.05466, 2019

… The first four are a random initialisation, and three pre-trained embeddings. The pretrained embeddings are: (a) word2vec (Mikolov et al., 2013); (b) GloVe (trained on Common Crawl) (Pennington et al., 2014); and, (c) Numberbatch (Speer et al., 2017) …


Finding Generalizable Evidence by Learning to Convince Q&A Models

E Perez, S Karamcheti, R Fergus, J Weston, D Kiela… - arXiv preprint arXiv …, 2019

… fastText We define a function BoWFT that computes the average bag-of-words representation of some text using fastText embeddings (Joulin et al., 2017). We use 300-dimensional fastText word vectors pretrained on Common Crawl …


Findings of the First Shared Task on Machine Translation Robustness

X Li, P Michel, A Anastasopoulos, Y Belinkov… - arXiv preprint arXiv …, 2019

… To explore effective approaches to leverage abundant out-of-domain parallel data. • To explore novel approaches to leverage abundant monolingual data on the Web (eg, tweets, Reddit comments, commoncrawl, etc.). • To …


Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions

P Koehn, F Guzmán, V Chaudhary, J Pino - Proceedings of the Fourth Conference on …, 2019


FlauBERT: Unsupervised Language Model Pre-training for French

H Le, L Vial, J Frej, V Segonne, M Coavoux… - arXiv preprint arXiv …, 2019

… Common Crawl).11 The data were collected from three main sources: (1) monolingual data for French provided in WMT19 shared tasks (Li et al., 2019, 4 sub-corpora); (2) French text corpora offered in the OPUS collection …


Frame Augmented Alternating Attention Network for Video Question Answering

W Zhang, S Tang, Y Cao, S Pu, F Wu, Y Zhuang - IEEE Transactions on Multimedia, 2019

Page 1. 1520-9210 (c) 2019 IEEE. Personal use is permitted, but republication/ redistribution requires IEEE permission. See http://www.ieee.org/ publications_standards/publications/rights/index.html for more information. This …


Page 1. Frequency, acceptability, and selection: A case study of clause-embedding Aaron Steven White University of Rochester aaron.white@rochester.edu Kyle Rawlins Johns Hopkins University kgr@jhu.edu Abstract We investigate …


From Legal to Technical Concept: Towards an Automated Classification of German Political Twitter Postings as Criminal Offenses

… We use a bi-directional LSTM (Hochreiter and Schmidhuber, 1997) for classification.30 We use the 300-dimensional German pre-trained word embeddings provided by Grave et al. (2018), which are trained on the German common crawl …


Frontiersinpatternrecognitionandartificialintelligence


Frowning Frodo, Wincing Leia, and a Seriously Great Friendship: Learning to Classify Emotional Relationships of Fictional Characters

E Kim, R Klinger - arXiv preprint arXiv:1903.12453, 2019

… We obtain word vectors for the embedding layer from GloVe (pre-trained on Common Crawl, d = 300, Pennington et al., 2014) and initialize out- of-vocabulary terms with zeros (including the po- sition indicators). 4 Experiments Experimental Setting …


Fusing Vector Space Models for Domain-Specific Applications

L Rettig, J Audiffren, P Cudré-Mauroux - arXiv preprint arXiv:1909.02307, 2019

… Despite the convenience they bring, using such readilyavailable, pre-trained models is often suboptimal in vertical applications [2], [3]; as these models are pre-trained on large, non-specific sources (eg, Wikipedia and the Common …


JA Balazs, Y Matsuo - arXiv preprint arXiv:1904.05584, 2019

Page 1. Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study Jorge A. Balazs and Yutaka Matsuo Graduate School of Engineering The University of Tokyo {jorge, matsuo}@weblab.tu-tokyo.ac.jp Abstract …


General Purpose Vector Representation for Swedish Documents: An application of Neural Language Models

Page 1. General Purpose Vector Representation for Swedish Documents An application of Neural Language Models Simon Hedström Master's Thesis in Engineering Physics, Department of Physics, Umeå University, 2019 Page …


Generalizable prediction of academic performance from short texts on social media

I Smirnov - arXiv preprint arXiv:1912.00463, 2019

… We obtained significantly better results with a model that used word-embeddings (see Methods). We also find that embeddings trained on the VK corpus outperform models trained on the Wikipedia and Common Crawl corpora (Table 1). 3 Page 4 …