Using back-translation for
quality control in a
multilingual thesaurus
Lorna Balkan
CESSDA Thesaurus Co-ordination Officer
UK Data Archive
[email protected]
Taxonomy Bootcamp, London 2016
18 October 2016
Overview
•
•
•
•
background and aims of work
methodology
results
conclusions and future directions
Aims
• to evaluate translation quality of multilingual thesaurus,
ELSST, using back-translation method
• develop methodology for how to do it
• results to feed in to ELSST translation guidelines
Funding
• work funded by EU-funded SERISS project - Synergies
for Europe's Research Infrastructures in the Social
Sciences
• 4-year project (2015-2019)
• topics include survey design, management and curation,
including translation quality of survey questions and
metadata
Why ELSST?
• CESSDA (Consortium for European Social Science Data
Archives) is one of the partners on the SERISS project
• ELSST used for multilingual search within CESSDA
data portal
About ELSST
•
•
•
•
•
European Language Social Science Thesaurus
12 languages, including English (source language)
developed by international team
team chaired by UK Data Archive
each country responsible for own language version
UK Data Archive
• based at University of Essex
• curator of UK’s largest collection of social sciences and
humanities digital data (over 7,000 data collections) –
distributes via UK Data Service
• over 40 years thesaurus development
• manages two related social science thesauri:
•
HASSET - monolingual
•
ELSST - multilingual
ELSST structure
• contains usual thesaural terms and relationships:
- Preferred terms (PTs) - Broader Terms/Narrower Terms (BTs/NTs)
- Use For terms (UFs) - Related terms (RTs)
• also scope notes
Why translation quality?
• thesaurus is a controlled language
• unlike normal usage:
• thesaurus terms must be unambiguous – promotes
interoperability
• Target Language PTs must denote same concept as SL PT –
enhances multilingual search
• ELSST based on ISO 25964-1 to promote translation
quality
Why back-translation method?
• recommended as way of evaluating translation quality in
other vocabularies and ontologies, e.g.
• IHTSDO (2012) - SNOMED
• Espinoza et al. (2009)
• one of several evaluation methods within SERISS
Methodology: overview
1.
2.
3.
4.
select target language terms
back-translate terms into English
compare back-translations with original SL terms
if back-translated term ≠ original SL term, then analyse
results
5. develop error classification and translation quality
criteria
Methodology: select terms
• target languages chosen: French and German
• translate Preferred Terms (PTs) only
• 2,000 PTs (1,000 in each language, ≈ 25% of total)
Methodology: back-translate terms
• manual - SL native speaker with good knowledge of
target languages and ELSST
• terms translated in their hierarchies - scope notes, etc.
available if necessary
• language resources including:
•
•
•
•
online dictionaries
parallel corpora
other thesauri
lexical and terminology databases
Methodology: compare and analyse
• if
back-translation ≠ SL PT in form or meaning
• then
• analyse TL PT and SL PT and their metadata
and
•
derive error classification
Results: error classification
semantic
• ambiguous TL terms
• TL term not clearly distinguished from other TL terms
• difference in meaning between TL and SL PTs (concept
mismatch)
• UK-specific SL PTs
• invalid BT relationship in TL
formal
• spelling, formatting and punctuation errors in TL term
• redundant or missing qualifiers in TL terms
• wrong plurality of TL PT
• difference in scope note of the TL and SL PTs
Results: error classification
For each error type describe:
• what it is
• how to find it
• how to solve it
• how to avoid it
Example 1: ambiguous TL term
• Example:
• ambiguity common in normal discourse
• not allowed in thesaurus
Example 1: ambiguous TL term
continuum:
• ambiguity (1 word, 2 unrelated meanings)
• bank = financial institution/riverside
• polysemy (1 word, 2 related meanings)
• newspaper = printed object/organisation
• vagueness (1 word, many meanings hard to distinguish)
• teacher – unspecified for gender
• special problem for social science terms
Ambiguous TL term: how to find
• TL term has more than one non-synonymous
back-translations, e.g.
• check metadata (place in hierarchy, scope note)
to see if ambiguous in thesaurus, e.g.
Ambiguous TL term: how to solve
• relabel PT
• e.g. PROMOTION PROFESSIONNELLE
rather than PROMOTION
• add qualifier
• add scope note
Ambiguous TL term: how to avoid
• ambiguity/polysemy/vagueness not always apparent until
translation sought
e.g. En: TEACHERS -> Ger: LEHRER (masc)
LEHRERINNEN (fem)
• help prevent by providing scope notes in both SL and TL
Example 2: redundant and missing qualifiers
in TL terms
• qualifiers used to disambiguate terms
• ambiguity language-specific, therefore use of qualifiers is
also language-specific
• if more than one homograph in same language, both
need qualifier:
e.g. LABOUR (RESOURCE)
LABOUR (PREGNANCY)
Example 2: redundant and missing qualifiers
in TL terms
Two error types:
• redundant qualifiers: SL qualifier not needed in TL
• missing qualifiers: 1 or more homograph missing
qualifier in TL
Redundant and missing qualifiers in TL: how
to find
• back-translation highlights cases where qualifier in SL but
not TL, or vice versa, but this produces:
– false positives:
– false negatives:
• to find all redundant qualifiers
• generate report of all TL terms (PTs and UFs) that
contain qualifiers, and manually inspect
• to find all missing qualifiers
• generate report of all homographs (PTs and UFs)
with or without qualifiers
Redundant and missing qualifiers in TL: how
to solve
• redundant qualifier:
• delete (qualifiers should be kept to a minimum)
• missing qualifier:
• rephrase if possible
• otherwise, add qualifier to TL terms
From error classification to evaluation
criteria for translation quality
• semantic adequacy:
• How clear/unambiguous is TL term?
• How well does TL PT match SL PT in meaning?
• How valid are the BT relationships in TL?
• formal adequacy:
• How well does TL PT meet translation guidelines?
Conclusion: advantages of back-translation
method
• highlighted problems thesaurus writers not aware of (e.g.
ambiguity)
• suggested enhancements to ELSST
• thesaurus guidelines
• workflow
• management system
• suggested error classification, and related evaluation
criteria
• produced other results, including alternative labels for SL
PTs and UFs
Conclusion: limitations of back-translation
method
• labour-intensive
• says nothing about:
• importance of each error type
• acceptability of the TL terms to the TL user (pragmatic
adequacy)
• effectiveness of terms in operational setting
• needs to be complemented by other evaluation methods
Future directions
• discuss findings with partners and implement agreed
changes
• update translation guidelines
• arrange for experts to give feedback on pragmatic
adequacy
• conduct operational evaluation
References
• Espinoza, Mauricio Mejıa, Montiel-Ponsoda, Elena, and
Gomez-Perez, Asuncion (2009) Ontology Localization, in
Proceedings of K-CAP '09 Fifth International Conference
on Knowledge Capture 2009 Redondo Beach, CA, USA,
September 01 - 04, 2009, ACM New York, NY, USA
• IHTSDO (2012) A methodology and toolkit for evaluating
SNOMED CT® Translation Quality, Version 1.00,
ihtsdo.org/snomed-ct/snomed-ct-worldwide/translationsof-snomed-ct
• ISO 25964 (2011) niso.org/schemas/iso25964/
Links
•
•
•
•
•
UK Data Archive: data-archive.ac.uk/
UK Data Service: ukdataservice.ac.uk/
ELSST thesaurus: elsst.ukdataservice.ac.uk/
SERISS project page: seriss.eu/
CESSDA-ELSST project page:
ukdataservice.ac.uk/about-us/projects/cessdaelsst/details.aspx
Thank you!
Questions
Lorna Balkan:
[email protected]