Associate Member
RATERS TRAINING AS A SAFETY CONCERN
ICAO LPR Implementation Workshop
Russia, St-Petersburg, May 24-26, 2011
Sergey MELNICHENKO
CompLang Aviation Training Centre
(Moscow),
Deputy Director
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
- RATER AS A DECISION MAKER
- SELECTION OF PERSONNEL
- AB INITIO TRAINING
- RATERS CERTIFICATION
- INTER-RATER RELIABILITY
- INTRA-RATER RELIABILITY
- RASCH ANALYSIS
- RECURRENT TRAINING
- EVALUATION OF RATER’S
ADEQUACY
RATERS TRAINING AS A SAFETY CONCERN
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
• RATER AS A DECISION MAKER
•
•
•
•
•
•
•
•
SELECTION OF PERSONNEL
AB INITIO TRAINING
RATERS CERTIFICATION
INTER-RATER RELIABILITY
INTRA-RATER RELIABILITY
RASCH ANALYSIS
RECURRENT TRAINING
EVALUATION OF RATER’S ADEQUACY
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Aviation English test results are felt
around the globe
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Inadequacy in Level Evaluation
Level 3 instead of Level 4:
Loss of job and/or occupation
Financial losses for AO or ANSP
Market losses
Redistribution of air transportation
services
Lack of confidence for test-takers
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Inadequacy in Level Evaluation
Level 4 instead of Level 3:
Aviation safety is compromised
Degradation to Level 2
Overconfidence for test-takers
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
•
•
•
•
•
•
•
AB INITIO TRAINING
RATERS CERTIFICATION
INTER-RATER RELIABILITY
INTRA-RATER RELIABILITY
RASCH ANALYSIS
RECURRENT TRAINING
EVALUATION OF RATER’S ADEQUACY
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
ICAO recommendations on interlocutors and raters
qualifications
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
ICAO recommendations on interlocutors and raters
qualifications (Doc 9835)
Familiarity with ICAO documentation
Annex 1;
Doc 9835;
ICAO Rated Speech Samples CD;
ICAO Circ 318
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
ICAO recommendations on interlocutors and raters
qualifications (Doc 9835)
Familiarity with ICAO documentation
Operational and/or language specialist
expertise
Level 5 or Level 6 proficiency
Familiarity with Aviation English
RATERS CODE OF ETHICS
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
•
•
•
•
•
•
RATERS CERTIFICATION
INTER-RATER RELIABILITY
INTRA-RATER RELIABILITY
RASCH ANALYSIS
RECURRENT TRAINING
EVALUATION OF RATER’S ADEQUACY
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
• Theory of language testing
• ICAO documents: Annex 1, Annex 10, Doc 9432,
Doс 9835, Circ 318
• The ICAO Proficiency Rating Scale and holistic
descriptors
• Correlating speech samples with the ICAO LPR
Scale
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Raters training course should be
aimed at experienced non-native and
native English language teaching
professionals and subject-matter
experts who are currently teaching
Aviation English and who want to be
able to rate according to the ICAO
Language Proficiency Scale
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
•
•
•
•
•
INTER-RATER RELIABILITY
INTRA-RATER RELIABILITY
RASCH ANALYSIS
RECURRENT TRAINING
EVALUATION OF RATER’S ADEQUACY
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
• Certification by national Civil Aviation
Authority or by international aviation
organization is mandatory
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
• Is geographical principle necessary?
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
• Are quotas necessary?
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
•
•
•
•
RATER AS A DECISION MAKER
SELECTION OF PERSONNEL
AB INITIO TRAINING
RATERS CERTIFICATION
• INTER-RATER RELIABILITY
•
•
•
•
INTRA-RATER RELIABILITY
RASCH ANALYSIS
RECURRENT TRAINING
EVALUATION OF RATER’S ADEQUACY
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Different raters but same results
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
•
•
•
•
•
RATER AS A DECISION MAKER
SELECTION OF PERSONNEL
AB INITIO TRAINING
RATERS CERTIFICATION
INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATER’S ADEQUACY
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Different times but same results
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
•
•
•
•
•
•
RATER AS A DECISION MAKER
SELECTION OF PERSONNEL
AB INITIO TRAINING
RATERS CERTIFICATION
INTER-RATER RELIABILITY
INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATER’S ADEQUACY
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Evidence of the validity of the
responses
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Evidence of the validity of the
responses
Correctional refresher course
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Evidence of the validity of the
responses
Correctional refresher course
Administrative measures to
marginal raters
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
•
•
•
•
•
•
•
RATER AS A DECISION MAKER
SELECTION OF PERSONNEL
AB INITIO TRAINING
RATERS CERTIFICATION
INTER-RATER RELIABILITY
INTRA-RATER RELIABILITY
RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATER’S ADEQUACY
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Annual refresher course for raters
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Annual refresher course for raters
New materials
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Annual refresher course for raters
New materials
Practical training
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
•
•
•
•
•
•
•
•
RATER AS A DECISION MAKER
SELECTION OF PERSONNEL
AB INITIO TRAINING
RATERS CERTIFICATION
INTER-RATER RELIABILITY
INTRA-RATER RELIABILITY
RASCH ANALYSIS
RECURRENT TRAINING
• EVALUATION OF RATER’S
ADEQUACY
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Rater is not a lifetime profession
RATER TRAINING IS A SAFETY
CONCERN
Raters Training as a Safety Concern
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Associate Member
Thank you
Raters Training as a Safety Concern