DELIVERABLE
Project Acronym: FLAVIUS
Grant Agreement number: ICT-PSP-250528
Project Title: Foreign LAnguage Versions of Internet and User generated Sites
D3.5 Translation quality report based on
grammar analysis
Revision: 1.0
Authors:
Sonia Collada Pérez (Daedalus)
Julio Villena Román (Daedalus)
1
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
Revision History
Revision
Date
Author
Organization
Description
0.1
October,
25th, 2012
Sonia Collada Pérez,
Julio Villena Román
Daedalus
First version
Statement of originality:
This deliverable contains original unpublished work except where clearly
indicated otherwise. Acknowledgement of previously published material
and of the work of others has been made through appropriate citation,
quotation or both.
2
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
Contents
Contents .................................................................................................................................................. 3
1.
Introduction..................................................................................................................................... 4
2.
Procedure ........................................................................................................................................ 4
3.
Corpus.............................................................................................................................................. 4
4.
Measures ......................................................................................................................................... 5
5.
Actionability..................................................................................................................................... 5
6.
Source texts ..................................................................................................................................... 5
7.
Evaluation based on number of mistakes ....................................................................................... 6
Grammar Mistakes per Sentence ........................................................................................................ 6
Mistakes per sentence ........................................................................................................................ 7
Improvement achieved ....................................................................................................................... 7
8.
Evaluation based on Translation Quality......................................................................................... 8
Expert evaluation ................................................................................................................................ 8
Errors detected.................................................................................................................................. 16
9.
Text correction .............................................................................................................................. 18
Grammar errors ................................................................................................................................. 18
Spelling errors.................................................................................................................................... 19
10.
Conclusions................................................................................................................................ 19
3
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
1. Introduction
This deliverable applies the text correction software on the generated translations to assess their
quality, which will be measured by the GMS metric. It will be applied on English, Spanish, Italian and
French translations. It will analyze the impact of source text correction on the quality of the
machine-translated output.
This task produces a report of the translation quality by analyzing the grammar mistakes that the
translated texts contain. The quality report evaluates the number of grammar mistakes and their
types, as well as the impact of the source text correction on the machine translation quality: we will
translate both the raw source text and the corrected source text to see which grammar mistakes
come from the translation engine itself and which ones come from the imperfections of the source
text.
2. Procedure
The number of mistakes per sentence has been extracted automatically with the Spell and Grammar
Checker, which has allowed us to calculate the number of errors obtaining as well information about
the type of error.
In order to evaluate the understandability of the text we will extract 200 sentences from each pair
source-target language, and our experts will assess its quality.
3. Corpus
The corpus used in order to evaluate the translation quality is the same one that was used to
evaluate the Spell and Grammar Checker quality for user generated content. This corpus was
automatically extracted in previous testing phases and every segment contains a maximum of 1000
characters. The corpus has 300 segments from user generated content.
Another corpus was collected so as to analyze the behavior of the Checker when correcting expert
generated corpus. Even when those texts are generally characterized by its accuracy, they might
contain some style errors or even some spelling errors but in general no grammar errors. Then only
user generated content will be considered to evaluate the translation quality.
4
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
4. Measures
The measure used to evaluate the translation quality is the GMS, which is the average number of
grammar mistakes per sentence in translated texts. The lower the GMS, the more confidence we can
have in the translation quality.
GMS =
Number _ of _ grammar _ errors
Number _ of _ sentences
5. Actionability
It is usually very difficult to estimate or evaluate the quality of automatic translations. All along this
project the main objective has been obtaining actionable translations, meaning translations which
are not completely correct but understandable and useful for indexing.
The measure proposed, GMS, allows detecting the number of grammar mistakes per sentence,
however this measurement does not provide information about the quality of the translation.
The traditional way to estimate the translation quality is comparing the machine translation with a
translation made by a professional translator, or having a native speaker checking the final
translation. However, these kinds of methods are too time consuming, so in order to evaluate the
understandability of the text we will extract 200 sentences from each pair source-target language,
and analyze their quality.
6. Source texts
Firstly, it is important to analyze the total number of errors in the source language. As shown in
previous evaluations English is the language with less number of errors in source text, the main
reason is that agreement errors are much more common in language such as Spanish, French or
Italian. Besides Spanish and French contain more errors than Italian because Italian is much more
flexible in the structure of sentences.
Table 1 Number of errors in source texts
Source Language Number of errors
5
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
English
73
Spanish
French
325
264
Italian
141
7. Evaluation based on number of mistakes
This evaluation aims at analyzing the number of errors found on the translated texts both applying
the Spell and Grammar Checker and translating the original texts without performing any correction
on the source text.
Grammar Mistakes per Sentence
Since for each source language three different translations are obtained, firstly it is essential to
compute the number of sentences for each translation.
Table 2 Grammar mistakes per sentence. Checked text
Source
Language
Target
Language
Number
sentences
English
Spanish
French
Italian
English
French
Italian
English
Spanish
Italian
English
Spanish
French
345
341
348
635
642
645
565
559
562
745
746
742
Spanish
French
Italian
of Number of GMS
grammar
errors
34
0.10
29
0.09
20
0.06
4
0.006
51
0.08
13
0.02
5
0.008
21
0.038
17
0.03
10
0.013
69
0.09
86
0.12
The results obtained for each pair of languages show that the requirements for this phase of the
project are satisfied, as the GMS value is much lower than the expected one (GMS<3).
6
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
Mistakes per sentence
Grammar errors do not provide enough information about the translation quality. The text correction
engine is able to detect four different types of errors: style, spelling, typography and grammar; so we
have decided to include a new measure: the number of mistakes per sentence, which includes the
information for the four types of errors mentioned. By considering this new measurement we can
analyze the proportion of errors in the final documents; those errors might be either untranslated
words or errors added during the translation phase.
Table 3 Grammar mistakes per sentence. Checked text
Source
Language
Target
Language
Number
sentences
English
Spanish
French
Italian
English
French
Italian
English
Spanish
Italian
English
Spanish
French
345
341
348
635
642
645
565
559
562
745
746
742
Spanish
French
Italian
of Number
of errors
detected
273
226
179
314
630
444
268
566
360
474
634
827
MS
0.80
0.66
0.51
0.5
0.98
0.69
0.47
1.01
0.64
0.64
0.85
1.12
When considering the total number of errors detected, we can see that grammar errors are in the
worst case the 12% of the number of added errors. This means that in average the structure of the
sentences on translated files is quite accurate; nevertheless there is an important number of style,
typographic and spelling errors.
As the user might not be really concerned about style or typography we will deeply analyze the
grammar and spelling errors and the text quality.
Improvement achieved
In order to have an idea about the improvement achieved by correction we have compared the
results for both checked and unchecked source text.
7
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
Table 4 Grammar mistakes per sentence. Checked text
Source
Language
English
Spanish
French
Italian
Target
Language
Spanish
French
Italian
English
French
Italian
English
Spanish
Italian
English
Spanish
French
Unchecked
source
282
238
196
368
682
476
316
440
410
505
653
854
Checked
source
273
226
179
314
629
444
268
566
360
474
634
827
Improvement
3%
5%
8%
14.7%
7.7%
6.7%
15.2%
-28%
12.2%
6.2%
3%
3.2%
The improvement is not quite significant in most of the cases, only two pairs seem to have quite an
important improvement; those cases are French-English and French-Italian, in which the
improvement is above the 10%. However when translating from French into Spanish we can see that
the number of errors added by the translation is so big that even when checking source text the
number of errors detected is even bigger than in unchecked source text.
8. Evaluation based on Translation Quality
The translation quality can be evaluated either comparing the machine translation with the one done
by a professional translator, or having a native speaker checking the result of the automatic
translation. However, this kind of methods are too time consuming, so in order to evaluate the
understandability of the text we will extract 200 sentences from each pair source-target language,
and analyze their quality.
Expert evaluation
Each of our experts have evaluated the sentences, three evaluators will score each sentence with a
value between 1 and 5, where 1 is a not understandable sentence and 5 is a clear and correct
sentence. The following tables show, for each pair of languages 10 sentences (out of the 200
sentences analyzed) and their score calculated as the mean of the scores provided by the three
evaluators.
8
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
Table 5 Evaluation source language English
Source
language
ENGLISH
Target
language
SPANISH
Sentences
Largo tiempo de espera y bebidas había que re-ordenado
2 veces, ya que olvida.
Lo siento que no ha publicado en los últimos días han sido
muy ocupado.
Texto original en árabe por Idriss Jandary.
Ronan Parke fue, sin duda, la estrella del show Britain's
Got Talent semi-final y final.
Te enseñan cómo comprar vestido de boda en línea.
Ciudadano marroquí medios de comunicación social: un
reto para profesionales de los medios.
Producto única y exclusivamente para uso personal
Cuerpo Rediseñar
Almorzamos en este restaurante, que es el último (en
dirección al norte/Locarno) de los restaurantes del
puerto. Pedimos un Coniglio (conejo) y un plato de
pescado.
Increíble! Había una de las mejores experiencias
gastronómicas aquí.
La bonita y animada ciudad de Avranches es una de las
ciudades más antiguas de Normandía y de sus orígenes se
remontan a la antigüedad.
FRENCH
Score
2
2
5
3
4
2
2
3
3
4
Les fumeurs malades du cancer colorectal plus tôt
Les armes explosives (Woolwich, Londres, par babzi)
Ca del Sol a la beauté d'une galerie d'art avec l'intimité
d'une maison.
Ma femme et moi avons passé quatre nuits dans cet hôtel
en juin 2011 et pense encore à notre merveilleuse
expérience il y a aujourd'hui.
La santé et le conditionnement physique | | exercice
renforcement musculaire.
2
5
5
Et le montant total de la facture - et nous avons tous eu
un dessert et bu beaucoup d'alcool - n'a pas assez de
$300.
Andrew Lansley du Livre blanc a été appelé libérateur du
NHS.
La jolie et animé de la ville d'Avranches est une des villes
les plus anciennes de Normandie et ses origines
remontent au début de l'antiquité.
4
3
4
5
5
9
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
ITALIAN
Syka se tenait sur une scène vide.
Votre meilleur ami serait sourire d'une oreille à l'autre
comme la femme de ménage de l'honneur, et votre
honneur serait belle et rayonnante
NON sono mai stato un posto migliore per avere un
piccolo snack.
I fumatori si ammalano di cancro del colon-retto in
precedenza.
Ti insegnano a comprare abiti da sposa Online
Si affacciava sul traffico e la confusione della città, nonché
la quiete del porto.
Il prezzo era molto ragionevole e dietro l'angolo atteso le
attività e serpeggianti strade dell'antico borgo medievale.
Uno dei modi migliori per scoprire la campagna di La
Hague per camminare lungo uno dei numerosi sentieri
costieri chiamati le sentier des doganieri.
Purtroppo nessuno è stato in grado di spiegare a chi o a
che cosa il NHS deve liberare.
La bella e vivace città di Avranches è una tra le più antiche
città della Normandia e le sue origini risalgono ai primi
dell antichità.
Il vostro migliore amico potrebbe essere veramente felici
da orecchio ad orecchio, come la cameriera di onore, e le
damigelle sarebbe bello e brillante.
Questo messaggio automatico è il primo articolo del tuo
blog. Essa vi aiuterà a iniziare con Overblog.
•
•
•
5
5
2
3
2
3
2
1
3
3
3
3
ENGLISH-SPANISH: The mean result for the English-Spanish pair is 3.2, which means that the
text is generally understandable but contains errors.
ENGLISH-FRENCH: The mean result for the English-French pair is 3.7, which is quite a good
result as the text is considered to be understandable and the errors added by the translation
are not too relevant.
ENGLISH-ITALIAN: The mean result for the English-Italian pair is 2.5, which means that the
text is not understandable enough.
10
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
Table 6 Evaluation source language French
Source
language
FRENCH
Target
language
SPANISH
ENGLISH
Sentences
Score
Es countrymusic y folk como jamás lo(la) había
oído(entendido), a toda velocidad.
Sobre el paseo Jean Naudin en Château-Thierry, esculturas,
pinturas, fotografías, vidriera, joyas, una gama ancha de
talentos propuestos por numerosos artistas se extendía en
miscelánea stands.en este domingo, 3 de julio.
La preparación y la colocación de mis tres expos me
doparon pero tan estresada.
Barnizado(Inauguración de una exposición de arte) de mi
expo " FLORES Y MARIPOSAS " hoy a las 11.
Reservé vía hostelworld sobre el neto y desgraciadamente
debí anular mi reserva.
Hice este briouates a las almendras siempre con ocasión de
mi menú oriental para el postre.
En una ensaladera, mezclar el polvo de almendra, el aroma
de flor de azahar, el azúcar en polvo así como el huevo.
Desembalar las hojas de bricbarcas. Cortar cintas de 6 cm
de anchura. Con la ayuda de un pincel, mojar cada cinta
para ablandarlos(debilitarlos).
Buen ambiente con precios(premios) razonables para
coktails.
Hacer tostar las redes de pollo con una red de aceite de
oliva: sembrar las redes de albahaca y de estragón.
The 16 and 17 July, the Hougue hosts the 10th edition of
the festival of the book
The movie of the week: HARRY POTTER AND THE RELICS OF
DEATH (2nd part)
The end of the adventure approach for Harry Potter ...
Accompanied by Hermione Granger and Ron Weasley, the
famous wizard must defend his life and many more in the
face of his greatest enemy: Lord Voldemort.
Last step before the rest, the latest or the sprinters could
express before Paris.
The preparation and implementation of my three expos
have buoyed me but also anxious.
This is a history that dates back to 2009 already, but it is a
story so incredible, it deserves to be known!
3
3
2
3
3
3
4
3
4
3
4
5
4
2
2
5
11
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
ITALIAN
•
•
•
Although weld the edge of each briouates wetting it with
water.
Excellent camping located to less than 20 km from
Barcelona and opposite the beach.
A fresh salad for beautiful days: i mixed my favorite
ingredients: pancetta, tomatoes, carrots, mozzarella
cheese, pepper and grilled chicken! For a small evening
meal or a picnic ... that of happiness!
So I'll give you my opinion for the 2 missing products: the
mask of unifying dyed and the lotion have a cleansing
cream eyes and neck.
buono ambiente coi prezzi ragionevoli per i coktails. la
decorazione è proprio bene, così come sulle fantasie dei
bicchieri.
Camera un poco piccola per il numero di stelle:) bella
piscina in cima al hotel con visto su Barcellona.
Le docce sono un poco strette ma propri nell'insieme.
La mia giornata del 14 luglio è stato soleggiata...e ricco per
gli occhi ed il cuore.
Anche constatazione che pierceok.
Antioxydante, reminéralisante, rinfrescante, tonificante,
sfogliata, idratante, anti righe, effetto che liscia,: ecco
tutte le virtù del fiore di ibisco
La locanda possiede anche 6 camere e 12 alloggi..Se la
digestione spinge alla siesta...
Le mie piccole cronache: prime giornate in Bretagna
E soprattutto una quantità molto più importante di
hydrolat raccolto.
Un buono luogo per mangiare nella calma. Alcuni piatti vari
e molto buoni.
2
4
4
2
4
4
3
5
2
5
2
5
2
4
FRENCH-SPANISH: The mean result for the French-Spanish pair is 2.8, which means that the
text is almost understandable but contains errors.
FRENCH-ENGLISH: The mean result for the French-English pair is 2.5, which means that the
text is not understandable enough.
FRENCH-ITALIAN: The mean result for the French-Italian pair is 3.4, which is quite a good
result as the text is considered to be understandable and the errors added by the translation
are not too relevant.
12
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
Table 7 Evaluation source language Spanish
Source
language
SPANISH
Target
language
FRENCH
ENGLISH
Sentences
Score
Très bon rapport qualité/prix.
Le roi lion a été libéré en 1994 et dans un très court laps de
temps est devenu le film générant le plus aux ÉTATS-UNIS ,
soulevé uniquement en Amérique du Nord pour 312
millions de dollars.
5
4
4 À 5 cuillères à soupe de crème pour la cuisson.
Que du point de vue géologique n'a pas attractions
spéciales pour un alpiniste, si bien fournit des informations
Ils ont tout, sont rapides et tout est très frais ! Hautement
recommandé !
Si vous voulez découvrir le Mexique authentique de thé de
cette taverne, dans lesquelles l'échantillon l'authentique
saveur mexicaine.
Un menu est composé de deux parties. Une partie en html,
et un autre en CSS.
Döner kebab Talun dans lequel préparé de la nourriture à
transporter et dans lequel vous pouvez appeler pour
commander de la nourriture et aller à ramasser.
Bon nombre des gens qui commencent une activité dans le
secteur de l'emploi autonome est confrontée à deux
problèmes.
Premièrement, ils ont besoin d'un endroit pour recevoir les
clients et de donner une image professionnelle, mais elle
signifie avoir à louer un bureau, avec ce que cela signifie, en
termes de dépenses mensuelles.
A few minutes of wisdom for tod@s
Primtemps i like more than the Galeries Lafayette and in
the terrace i am better.
It is unfortunate that people don't know what is going to a
law firm.
They have everything, are fast and everything is very fresh!
Highly recommended!
Hello, to all and to Rhodes, today, I presented a loaf of
bread superesponjoso, my family loves.
3
3
The place is cozy. The food was very good. Attention was
also very correct. Would definitely return to the restaurant.
Many of the people who start an activity in selfemployment is faced with two problems.
5
4
2
5
2
3
3
3
2
3
5
2
4
13
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
ITALIAN
•
•
•
The simple Indian restaurant in this town, crossroads of the
key British railway, has a secret: their chicken tikka masala
is to make you cry with pleasure!
The properties are what you're inside the { } in the lines of
your style sheet.
I believe that the people who composed is very serious and
dedicated, his passion is to create a warm atmosphere so
that you feel at home and be able to listen to the variety of
products that are and be able to decide which is the
computer that sounds better and the most suitable to your
listening room
Disponibile in uno stampo e mettere su la crema.
4
E' un peccato che le persone non sanno dove andare per
una ditta di legge.
Se volete scoprire il vero Messico sul tè per questa taverna,
in cui il campione autentico sapore Messicano.
Stampo pane spugnoso
Conclusione, il Kebab saporiti, ... Diciamo che ho visto di
meglio.
Molte delle persone che avviare un'attività di lavoro
autonomo si trova di fronte a due problemi.
Il buffet offre un buon servizio ed è possibile scegliere
diversi tipi di insalate e menu " comprende ".
Essi sono i migliori pescateros di Bilbao, e oltre al bastone
del perrechico anche.
Adattare per il grande schermo nei cartoni animati un
racconto popolare francese del xvii secolo era stato un
vecchio progetto di Walt Disney.
Dispone di una terrazza per sfruttare le giornate di sole.
3
4
4
2
2
2
3
3
3
2
3
5
SPANISH-FRENCH: The mean result for the Spanish-French pair is 3.5, which means that the
text is understandable even though it contains some errors.
SPANISH-ENGLISH: The mean result for the Spanish-French pair is 3.5, which means that the
text is understandable even though it contains some errors.
SPANISH-ITALIAN: The mean result for the Spanish-Italian pair is 2.9, which means that the
text is almost understandable but contains errors.
Table 8 Evaluation source language Italian
Source
language
Target
language
Sentences
Score
14
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
ITALIAN
SPANISH
ENGLISH
Excelente servicio son realmente fiable y profesional serio.
Si usted está en Edimburgo y seguidor de Harry Potter, va
a dar un salto este pub/caliente tabla.
Administrado por el pueblo argentino este local además
de ser fascinante por su pequeñez y muy acogedor.
Después de unos cientos de metros el camino se convierte
en una mula con una pendiente abominable, fondo
extremadamente áspero piedras lisas que no sabe si la
lleva arranque.
Sin embargo, ¿se dan cuenta que no es probable que en la
semana me levanté tan temprano? A continuación,
finalmente nos caminar en el día promete espléndida y
tórrido.
Me encantan estos cruces: trasladarse de un lugar, y llegar
a una realidad diferente y caminar por los senderos que
ponen en comunicación los pueblos, países, culturas, a
menudo tan diferentes entre sí.
Hicimos una pequeña parada balisto , con el fin de
permitir que las piernas para recuperarse.
Estamos colaborando con correos internacionales de años
como TNT, DHL ROMA, UPS Y FEDEX.
La Reina del Raval no es para nada malo, como local, de
hecho las noches de verano son muy agradable sentarse
mesas en el exterior.
La primavera, de las que tienes que mirar bien antes fue
revelado.
The new piece has been written and are initiated the first
tests.
The women of the 6° floor: the film of the Guay, having
fun, offers food for thought and analysis of social issues
Managed by the Argentinian people this local in addition
to being fascinating for its smallness and very welcoming.
A beautiful yellow sign indicates to us the starting point
for the trail.
From one year to this part, for reasons of work, i began
attending the Gambarogno, passing briefly also from
fractions placed well above the level of the lake.
In the few sprazi wherein the wood opens we see the lake
Maggiore, also shooting some photos, but you save the
result.
The slope of the Gambarogno, being exposed to the
North, enjoys much less heat and light with respect to the
area of Locarno and Ascona.
2
2
3
2
1
5
2
1
3
1
2
2
2
5
3
2
5
15
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
FRENCH
•
•
•
Finally, past the settlement, the trail enters decided in the
woods. Runs with ascents and descents, silence all around,
we are the only inhabitants of the earth.
And photo to a beautiful waterfall, with the edges,
completely frozen.
After a few hundred meters the path turns into a mule
with a slope abominable, fund extremely rough, smooth
stones that you don't know if the boot takes.
La photo ici, c'est mieux que ce qui était à l'origine, je suis
passé par Picasa.
Un magnifique banc vers la Ghiridone / Gridone /
Limidario, le goflo à Locarno, ses nombreuses espèces
permet des îles, avec vue pratiquement jusqu'à Cannobio.
Juste au-delà, une grave bridge, faite de métal. Atteint la
moitié gauche, un spectacle ravissant.
Jusqu'à maintenant, nous avons perdu un peu du paysage,
la forêt est très dense.
Derrière nous les deux pics du Piz Bird nous rappelle que
nous devons également à la hausse de la partie ...
Nous descendrons, souvent aider avec les mains et/ou à
l'arrière, où l'arrière change de nom ...
Nous avons une bonne heure de retard sur le plan de
l'engrenage que j'avais fait: nous avons obtenu jusqu'à
environ 600 mètres en deux heures, j'ai calculé moins ...
Le ressort du vous avez à regarder bien avant qu'il a été
révélé.
L'idée était de s'arrêter pour prendre un café, mais je
pensais que nous serions arrivés directement. Nous avons
décidé de partir pour rester, et nulle part on ne retrouve à
Meride.
Je suis arrivé ici la première fois avec l'école et m'a frappé
beaucoup.
5
5
1
3
2
2
4
2
2
4
2
2
5
ITALIAN-SPANISH: The mean result for the Italian-Spanish pair is 2.3, which means that the
text is not understandable enough.
ITALIAN-FRENCH: The mean result for the Italian-French pair is 3.1, which means that the
text is understandable even if it contains errors.
ITALIAN-ENGLISH: The mean result for the Italian-English pair is 2.7, which means that the
text is almost understandable enough.
Errors detected
16
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
Most of the errors detected are untranslated or wrongly translated words and the grammar errors
are related to number and gender agreement in the target language; additionally sometimes there
might be grammatically correct sentences, but difficult to understand for the reader.
The main errors are common among the four languages. However, some errors occur more
frequently in some language combinations. This is a summary of the types of errors found among the
tests with representative examples. Errors from the first group are grammatical in nature, while the
second group relates to words or expressions that were not translated or were wrongly translated.
Table 9 Grammar errors
Grammar errors
Subject-verb
agreement
Determinantnoun/adjective
agreement
English
Spanish
French
Italian
- love do not resist - las subvenciones - les chercheurs se fijó
poids
peut
- los precios de los
expliquer
alimentos es
- a inappropriate - la redes sociales
- Un
étrange - la
sua
- a aggression
- una interminable
lecture
sorprendente
- these two case
interrogatorio
e suggestivo
colore
- i lenzuola
- tutti vita
- tutti
assolutamente
geniale
- fue
posteriormente
expuestos
- una
casa
interactivo
Prepositional
- didn't
want - consulta a 68
binding
offend
euros para los
pacientes padecen
- venir testificar
Missing contraction - de el
Verb tense
- may were
- El oro perdido
- have is
terreno y el dólar
cayó
Noun-adjective
agreement
-
- attention très - sincera
professionnel
prestazioni
- votre honneur
serait belle
-
-
- a été étirer
-
Table 10 Translation errors
Translation errors English
Wrong translation -
Spanish
French
- el estreno en cine clips
Italian
17
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
Unstranslated
words
- andariego
- churrería
- blagging
- damned
- images
- spiced
- their
- churrería
- canjearamos
- snacks
- churreros
- coactuamos
- saladier
9. Text correction
The text correction engine aims at detecting and correcting spelling and grammar errors from user
generated content in order to improve the translation results.
The impact of the source text correction on the machine translation quality can be evaluated by
translating the corpus twice; at first the corpus is translated applying text correction and then a new
translation is carried out on raw text.
The previous process provides two different translations that can be easily compared. In order to
evaluate the differences the text correction module is used once more, providing two outputs with
the spelling and grammar errors corresponding to each translation.
Grammar errors
The main grammar errors detected on both translations are related to agreement (see Table 9).
The results show that in some cases the translation engine is quite independent from those types of
grammar errors. However this does not mean that the translation is completely correct, though they
are in general understandable.
•
Incorrect gender agreement :
Déposer un boulette à l'extrémité de la bande […]
o Remove the meatballs to the end of the strip […]
o Quitar las albóndigas para el final de la banda […]
o Depositare una pallottola all'estremità della banda […]
The translation into Italian is the most accurate among the three translations, in the
translation into English the verb is not the most appropriate and the Spanish one is
understandable but the use of the verb and the preposition “para” is not correct.
•
Incorrect number agreement:
Algunos decretos han sido modificado parcialmente […]
o Some decrees have been partially modified […]
o Certains décrets ont été partiellement modifiées […]
18
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services
o
Alcuni decreti sono state parzialmente modificate […]
The translation engine in this case corrects the number agreement in French but adds an error of
gender agreement. In English there is no agreement between noun and adjective so the translation is
correct. In Italian the translation includes the agreement error between noun and adjective and
includes an error into the verb.
Spelling errors
The main spelling errors detected are related to untranslated words (see Table 10), however when
applying the text correction module to the translation process some misspelled words are corrected
and as a result the translation is more accurate and the number of untranslated words decreases. For
instance when translating from French to Italian we can find some misspelled words such as:
•
•
Noêl: when using the spell checker this word is corrected as “Noël” so the translation is
correctly performed as “Padre Natale”.
Acceuil: this word is corrected by the spell checker as “Accueil” and then translated as
“Accoglienza”.
If the translation is performed without using the correction engine then those words remain
untranslated, and the sentences containing those words lose their integrity and the general meaning.
10.
Conclusions
The requirements of linguistic correctness are not only related to a perfect use of grammar, but also
to the use of an accurate and consistent terminology, similarity to the source text, cultural
correctness, and readability for the users.
This final assessment shows that in average, the translations performed with Flavius platform are
actionable as the users are able to understand the general meaning of the translated texts.
Additionally, the percentage of mistakes per sentence complies with the project requirements, and
even if the grammar, typography and style of the translations are not perfect and in some cases it
makes more difficult to understand some sentences, the users feel quite pleased about the overall
result.
Concerning the significance of the correction onto the final result, we can observe that even if in
some cases the translation is quite insensitive to the precision of the source text, in some pairs of
languages the number of errors in the translated text can be reduced in a 15%, which is an important
enhancement.
19
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
C
Confidential, only for members of the consortium and the Commission Services