Kia Karavas
TRAINING THE ORAL EXAMINER 

Within the context of the KPG exams, the systematic training of oral and script raters is regarded as one of the most significant factors contributing to the validity and reliability of the speaking  and writing tests (i.e., Modules 4 and 2 respectively) and to the sustainability of the examination system as a whole. This presentation will focus on the main principles and characteristics of the oral examiner training programme.

The oral test and the role of Examiners

Speaking is assessed in Module 4 of the KPG exams. The test paper contains three activities involving candidates in interaction with the Examiner or with each other. From the B1 level onwards, one of the activities requires candidates perform mediation tasks which require relaying information from Greek to English. 

The development of the oral test paper is based on clearly laid out specifications (which all languages certified through the KPG examination system follow and on detailed descriptors of oral performance (articulated as can-do statements) for each level[1]. Tasks and items designed take seriously into account candidates’ age as well as their social, cultural and linguistic background. Once designed, they undergo rigorous and systematic pilot testing on the basis of which changes and improvements are made before the final fine tuning just before the exams.

After each oral test administration, task analysis is undertaken while systematic input from Examiners is an object of investigation at the RCeL to determine the construct validity and effectiveness of the test as a whole.

However, all the work going into designing a quality assessment instrument would be in vain if the system did not make sure that Examiners assign the ‘right’ mark to candidates (Alderson, Clapham and Wall 1995). In other words, ultimately, the validity and reliability of the oral test and fairness of the oral test procedure depends on the Examiners; that is, the extent to which Examiners (a) share a common understanding of the tasks and what type of output these require, (b) follow instructions regarding test procedures in a uniform manner, (c) keep to the strict guidelines when in the role of Examiner-Interlocutor, and (d) apply the oral assessment criteria consistently and use the rating scales as instructed.

In order to achieve the aforementioned goals, systematic and intensive Examiner training is considered a crucial and indispensable component of the testing programme. For the KPG English test development team, Examiner training has been a major concern from the very beginning and its goal has been to create a pull of Examiners  who are fully and systematically trained to assess oral production at all levels offered by the KPG exams – a team of Examiners whose performance has been evaluated on the job as satisfactory.  

The aims and main features of the KPG oral examiner training programme

The aims of the Examiner training programme include:

• helping Examiners develop a common understanding of the type of communicative performance required at each level of proficiency

• training them to consistently follow test procedures and carry out the oral tasks in ways instructed so as to get candidates to perform to the best of their ability

• facilitating them to assess and rate candidates performance in English accurately and consistently by applying the assessment criteria and the respective rating scales

The training of Examiners has been systematic and seminars are carried out all over Greece just before each exam administration, but also in between periods. By December 2008 more than 3000 Examiners were trained in assessing candidates’ oral performance at A1/A2, B1, B2, C1 levels.

Each seminar takes place with groups of 18-20 people maximum to enable dialogue, discussion and group work. For each round of training seminars a new, tailor made oral examiner training pack and accompanying videos with simulations of the oral test are developed by a group of experienced teacher educators from the Faculty of English Language and Literature who are also involved in the design of the oral test. The oral examiner training seminars are conducted by this group and a body of 50 ‘Multipliers’ throughout Greece. WE do not use the term Teacher-Trainer, given that the term Multiplier describes best what these highly qualified and experienced professionals do: they ‘multiply and dispense uniform information’ to all Examiners, helping them develop the skills they need to have as Examiners.

Before each round of training seminars begins, Multipliers are invited to a one day seminar in Athens or Thessaloniki during which they are informed of the aims, content and structure of the training seminar and are trained in using the seminar materials. Depending on their area of residence and their mobility they are assigned groups of oral examiners and have the responsibility of setting a date and place for the seminar, contacting the oral examiners and photocopying the training material; after each seminar they are requested to provide an evaluation report of the seminar noting cases of absent examiners, and providing details of problems/issues raised in the seminar.

The structure of the KPG oral examiner training programme

The training programme has a series of stages presented below which all oral examiners need to go through:

• Recruitment (applications), initial screening, inclusion in database

• Initial evaluation of oral performance and induction (applies once to an applicant oral examiner)

• Training seminars (recurrent and cyclical since the outcomes of monitoring and evaluation feed into training)

• Monitoring of the oral test and evaluation of Examiners’ performance by multipliers through specially designed observation schemes

In designing and developing the training programme for KPG Examiners, various constraints and factors have been taken into account, making this programme unique in many ways. These are:

1. Dealing with a huge number of applications from EFL teachers to become oral examiners (to date, more than 5,500 teachers have applied). Since there is no closing date for applications, every 6-12 months new applications need to be processed, included in our database, and new cohorts of potential oral examiners need to be screened and trained from scratch.

2. Dealing with applications by individuals with varying qualifications and varying degrees of experience in teaching and testing; all are eligible to apply and must be considered as potential oral examiners and undergo the training programme. If individuals are rejected, they are rejected because a) their oral performance has been assessed as unsatisfactory during the screening process which takes place during the initial training seminars, or b) because they do not systematically take part in training sessions they have been invited to attend, or c) because they have personally expressed their unwillingness to continue with their role as an oral examiner, or d) because their performance on the job on the basis of the observation was consistently rated as unsatisfactory.

3. Coordinating a Nationwide training programme. Our seminars do not take place in big cities or designated exam centres. They take place throughout Greece and we try to accommodate examiners who live even in remote areas (islands for instance). Seminars are organised in places which are easily accessible by national road, making sure that oral examiners do not need an overnight stay in order to attend and that the seminar is within reasonable driving distance from their homes.

4. Constantly updating our database with changes in contact details, the training record of each oral examiner and the results of the observation of oral examiners.

The continuing and developing success of such a large scale training programme is certainly due to the personal commitment and hard work of the Director of the English Team, Prof. Bessie Dendrinos, but it is also due to the commitment of the English Team and all our associates – many of whom are School Advisors for English and state school or language centres teachers of English. Having started with a group of 300 oral examiners and having at our disposal few resources and limited funding for a considerable period of time, we are now proud to have developed a team of over 3000 trained and evaluated Examiners.