Consolidated Report ­ TAUS DQF Aggregated Enterprise Solution Survey results and summary of feedback gained in April 2016 from 16 companies that participated in the consultation. Content 1. Introduction 2. Recommendations 2.1
Benchmarking 2.2
Privacy 2.3
Frequency of use 2.4
Productivity vs Quality tracking 2.5
Type of reports 2.6
Level of reporting 2.7
DQF/MQM error typology 2.8
Number of users 3. Recommendations 3.1
Redefining quality levels 3.2
Productivity and quality separated 3.3
Segment level tracking 3.4
Batch processing 3.5
Reliability 3.6
Time measurement 3.7
Additional evaluation types 3.8
Pull method for raw data 3.9
Automation 3.10 Flexibility 3.11 Business intelligence 4. Consolidated agenda and roadmap 1 1. Introduction In the course of the past months, TAUS has received requests from large enterprise users to develop an alternative solution to the current DQF API in which the translated data segments remain on the enterprise servers. DQF in its original design posed some obstacles to adoption based on scale and confidentiality. It seemed impractical for some users to transfer all resources externally for data calculation and it also posed some confidentiality concerns. The initiative is steered by TAUS and Microsoft. As a company Microsoft has always been looking for ways to understand quality and continually improve it. In that context, Microsoft expressed its willingness to share data through DQF for internal and external benchmarking. In order to arrive at a sustainable solution and to serve not only Microsoft's needs but also align with the needs of other translation vendors and buyers, TAUS has undertaken a consultation with 16 companies. Current report is the result of extensive feedback from TAUS members and enterprise participants in an online survey and during 1­on­1 calls. The aim of the consultation was to get consensus on the type of reports, level of granularity and the IT architecture that is required. The following enterprises provided feedback: ​
Microsoft,​
​
Cisco, Dell, eBay, Lionbridge, Moravia, Oracle, PTC, VMware, Welocalize, Amazon, Intel, PayPal, SDL, Alpha CRC, LDS Church. The general opinion is that an aggregated solution is much more attractive for large corporates than the non­aggregate set­up. The consultation reinforced the main benefits of DQF in this new offering. Points for further discussion were also revealed as the ones below: ­ the need to clarify the aggregation level (project vs. segment) ­ a new distribution of the different quality levels (currently only two) ­ the separation of productivity and quality tracking (currently no separation) In what follows we will provide a brief analysis of the survey and the feedback we received during the consultation ​
(section 2)​
; we will list the key­take aways that need reconsideration during the follow­up meetings​
(section 3) ​
and that can serve as an agenda for further discussions; finally we will present the consolidated agenda and roadmap ​
(section 4)​
. 2. Survey results 2.1 Benchmarking The industry benchmarking seems to be one of the most important and most attractive offerings of the TAUS DQF aggregated solution. Some participants asked for a more detailed description 2 of the benchmarking process in the scoping documents and more transparency of the aggregated data on the website. Current scoping documents mainly contain project level reports. 2.2 Privacy In general, participants think that the aggregate solution will resolve the "data privacy dilemma". This is true to the point that even Language Service Providers feel comfortable with this new offering. 2.3 Frequency of use Almost half of the users would use the Quality Dashboard in the new setting once or twice per week. A couple of users would consult the dashboard daily and a larger group would use it incidentally. 3 2.4 Productivity vs Quality tracking A number of companies tracking quality already expressed their need for productivity measurement. They would benefit from productivity and edit distance scores. Most users are interested in both productivity and quality tracking. Some users were solely interested in quality tracking and would like to see a solution where productivity tracking can be turned off. 4 2.5 Type of reports Half of the participants in the consultation find the current plans for the reports satisfactory. Some participants found that more reports are needed and additional datapoints should be collected. All in all, participants agree that current plans for reports are a good starting point and new reports can be added in later releases. 5 2.6 Level of reporting Project level seems to be the most popular level of reporting but some users would also like to see higher levels like program or division level and even more granular levels like segment level. 6 2.7 DQF/MQM error typology One reason some companies were hesitant to give a confirmative 'yes' to this question was that they were not aware of the flexibility provided in selecting error­types, granularity levels and severities. Such flexibility will be provided and will facilitate the customized use of the typology. In addition to that, when it comes to quality metrics, some vendors depend on the requirements of their customers, which makes it difficult for them to be affirmative at this point. Still, almost all companies participating in the consultation answered positively to the question whether they would use the DQF/MQM error­typology. 7 2.8 Number of users The number of users of this future solution within companies vary from "less than 10 users" (30%) to "more than 50 users" (23%) with the midrange of "between 10 and 50 users" leading with 47%. 3. Recommendations 3.1 Redefining quality levels Participants expressed the need for more quality levels than only the currently supported two (i.e. good enough and similar/close to human translation) and a redefinition of quality levels in general in order to align with their current practice. Some companies use a tiered model of three levels. Others use four or five levels and connect these to different processes (e.g. 1=MT, 2=MT+light PE, 3=MT+full PE, 4=MT+full PE+Review1, 5=MT+full PE+Review1+Review2). 8 3.2 Productivity and quality separated Possibility to "turn off" productivity and only track quality. Some companies are either not interested in productivity or are restricted by national law. 3.3 Segment level tracking Segment level tracking as opposed to project level tracking. In certain cases it would be useful to dig down to the segment level even in the aggregate solution and see what caused a certain error or a productivity decrease. Also removing outliers would be easier if data is tracked at the segment level instead of the project level. 3.4 Batch processing Accumulating data and then offering the user to submit/distribute afterwards is probably the best approach for various reasons not only related to connectivity. Some companies face contractual issues or consent issues with data sharing, so this should be reflected by the architecture. The choice to submit whether users agree to send such data, and if, what data and when must be captured. This acknowledgement should be visible and transparent for users and quite possibly permit users to view what has been gathered prior to submitting it to any aggregation platform. One disadvantage of this approach is that project creators will no longer be able to track productivity and quality "on the fly" but need to wait till the project is completed and submitted to the platform. 3.5 Reliability How to ensure that integrated technologies provide the valid and comparable numbers? One example is word­counts. The functionality related to the generation of word counts in SDL products is a proprietary solution. Word­counts can also depend on specific settings in the CAT tool making it difficult to exactly compare data across tools. Another example is the calculation of edit distance. Currently character­based edit distance is applied in DQF while some technologies employ word­based edit distance. In order to make results comparable, TAUS should propose standards or an SDK that need to be complied with in order the integration to be successful. Another option is to translate proprietary counts and algorithms to the ones used in DQF through mappings which would not be a trivial task. 3.6 Time measurement Sometimes total time of the translation project can be captured in addition to the active time per segment. In certain cases the total time is the only indicator available. It might be interesting to manage both the active time a user spends in each segment and the total time of the project. 9 3.7 Additional evaluation types Current set­up contains productivity measurement and error­review. Users would like to see additional evaluation types that are still available in the old DQF tools including rank comparison and adequacy/fluency evaluation. These additional evaluation types should be available for the reviewer not only at the segment level but also at the paragraph, file or project level to offer faster and a more holistic type of evaluation of translated content. 3.8 Pull method for raw data Many participants would like to see a download button that would allow them to download their own raw data for a selection of projects. Users should be able to process their own data and create their own reports if needed. 3.9 Automation When it comes to selecting error types, setting penalties and thresholds, profiling content or sampling for review, participants in the consultation would like to see more automation and less manual work. One question to discuss in upcoming meetings is where the automation should occur: internally or externally to the aggregate solution. An example for external automation is the current DQF auto action in WorldServer that requires manual steps but can be automated adding customized components to the workflow. 3.10 Flexibility An added value to the enterprise solution would be more flexibility in terms of the metadata collected and the filters provided to tweak the reports. An example is the need to use additional content types in a hierarchical order. This kind of requests should be made possible as long as a mapping is available on the high level to the current list of possible values in DQF. 3.11 Business intelligence While the main aim of the Quality Dashboard is collecting, aggregating and visualizing data, an additional benefit in the enterprise solution could be data analysis and automatically generated predictions based on the aggregated data. Suggestions could be provided on the dashboard as to how to improve weak points offering links to best practices or providing information in pop­ups. Links could point to related articles in the knowledge base or to courseware on the TAUS website. Correlation between different metrics could be indicated (e.g. between productivity, edit distance and quality). Outliers or contradictory results could be flagged automatically. This way the Dashboard would help analyze and interpret the data, would be 10 more interactive and would truly become a business intelligence platform for the translation industry. 4. Consolidated agenda and roadmap May 6 Consolidated report based on consultation with 16 companies shared with all participants. May 9 (9:00 AM PT) Conference call​
with participants to discuss the scoping document and agree on the agenda for the face­to­face meeting on June 8. May 24 Proposal document, scoping out the Quality Dashboard reports and the IT architecture, will be shared with all participants. June 8 Presentation and discussion of aggregated enterprise solution at QE Summit at Microsoft in Dublin followed by a ​
1­hour breakout session​
to kick off the development and implementation. June 15 Scoping of the Quality Dashboard and IT architecture completed. Document describing the standard aggregated enterprise solution of DQF distributed to all participants. July 1 Start of development and implementation August 31​
Project completion 11