Source:

Massih-Reza Amini
Université Joseph Fourier
Laboratoire d'Informatique de Grenoble
Email : Massih-Reza.Amini '@' imag.fr

Cyril Goutte
National Research Council Canada
Interactive Language Technology group
Email : Cyril.Goutte '@' nrc.ca

Data Set Information:

Uncompressing rcv1rcv2aminigoutte.tar.bz2 will create a directory that contains 5 subdirectories EN, FR, GR, IT and SP, corresponding to the 5 languages. Each subdirectory in {EN, FR, GR, IT, SP} contains 5 files, each containing indexes of the documents written or translated in that language. For example, EN contains files:
- Index_EN-EN : Original English documents
- Index_FR-EN : French documents translated to English
- Index_GR-EN : German documents translated to English
- Index_IT-EN : Italian documents translated to English
- Index_SP-EN : Spanish documents translated to English

And similarly for the 4 other languages.

Each file contains one indexed document per line, in a format similar to SVM_light. Each line is of the form: : : ... where is the category label, ie one of C15, CCAT, E21, ECAT, GCAT or M11. : is the feature, value pair, in ascending order of feature index.

The order of documents is maintained in corresponding files, for example, FR/Index_EN-FR and EN/Index_EN-EN have the same number of documents (and therefore the same number of lines), in the same order.

Attribute Information:

We focused on six relatively populous categories: C15, CCAT, E21, ECAT, GCAT, M11. For each language and each class, we sampled up to 5000 documents from the RCV1 (for English) or RCV2 (for other languages). Documents belonging to more than one of our 6 classes were assigned the label of their smallest class. This resulted in 12-30K documents per language, and 11-34K documents per class. The distribution of documents over languages and classes are:

Number of Vocabulary
Language documents percentage size
************ ********** ************ ************
English 18,758 16.78 21,531
French 26,648 23.45 24,893
German 29,953 26.80 34,279
Italian 24,039 21.51 15,506
Spanish 12,342 11.46 11,547
-------
Total 111,740

The distribution of classes in the whole collection is
Number of
Class documents percentage
********* ********** ************
C15 18,816 16.84
CCAT 21,426 19.17
E21 13,701 12.26
ECAT 19,198 17.18
GCAT 19,178 17.16
M11 19,421 17.39


In experiments that we conducted in cite{AUG09}, we considered each document available in a given language as the observed view for an example and all translated documents were used as the other views for that example, generated using Machine Translation. Results shown in this study were averaged over 10 random samples of 10 labeled examples per view for training, and 20% of the collection for testing.

Relevant Papers:

Massih-Reza Amini, Nicolas Usunier and Cyril Goutte. Learning from Multiple Partially Observed Views - an Application to Multilingual Text Categorization. Advances in Neural Information Processing Systems 22, pp. 28-36, 2009

Massih-Reza Amini and Cyril Goutte. A Co-classification Approach to Learning from Multilingual Corpora. Machine Learning Journal Springer, 79(1-2):105-121, 2010

Abhishek Kumar, Hal Daumé III. A co-training approach for multi-view spectral clustering. International Conference on Machine Learning, pp. 393-400. 2011

Citation Request:

If you publish results based on this data set, please acknowledge its use, by referring to:

M.-R. Amini, N. Usunier, C. Goutte. Learning from Multiple Partially Observed Views - an Application to Multilingual Text Categorization. Advances in Neural Information Processing Systems 22, p. 28-36, 2009

Supported By:

 In Collaboration With: