Elizabeth Apostolou & Bessie Dendrinos
INVESTIGATING DIFFICULTY OF THE LISTENING TEST

KPG’s concern to be a ‘fair’ testing system led the university of Athens team preparing the exams in English to embark on a large-scale research project regarding listening comprehension, the assessment of which is a rather neglected area of investigation. Meanwhile, the listening comprehension component of all well-known exam systems is what candidates complain about the most. They often whine that the activities were either too difficult or that they couldn’t hear the speakers well, that the speaker didn’t speak clearly enough or that there was too much noise in the exam room which prevented them from making out what was being said, and so on.  Whether or not, though, these complaints correspond to reality, the fact remains that the listening test is often the most difficult section for many candidates. This is the case with our exams in English (as our systematic analysis reveals) and, thus, we have set out to investigate the most important factors involved in making our listening items ‘easy’ or ‘difficult’.

There are several studies which attempt listening comprehension task analysis and investigate the linguistic, pragmatic and cognitive factors which contribute to task difficulty (e.g., Ur 1984, Anderson and Lynch 1988, Rost 1990, Conrad 1985, Buck 2001). These studies often point to factors concerning the learner and his/her lack of language skills being the main cause for listening comprehension difficulty. They also point out that it is the very nature of the spoken language which is usually considered much more difficult to understand due to such characteristics as the use of elision, speech rate, accent variation, stress and intonation, hesitation, redundancy, etc.

Furthermore, these and other studies point to the cognitive factors involved in listening comprehension and attempt to show that understanding is invariably linked to the listener’s prior knowledge, experiences and expectations.  Anderson & Lynch (1998), for example, argue that understanding is not something that happens because of what the speaker says, but that the listener has a crucial part to play in the process. S/he activates various forms of knowledge and by applying what s/he knows to what s/he hears, s/he ultimately understands the message conveyed. Buck (2001) agrees that the cognitive aspect of listening comprehension is very significant, and views listening comprehension as an inferential process which moves beyond the knowledge of discrete elements of language, such as phonology, vocabulary and syntax. According to him, “meaning is not something in the text that the listener has to extract, but is constructed by the listener in an active process of inferencing and hypothesis building” (ibid: 29).

Though we are in full agreement that linguistic and cognitive factors as well as learner skills are all responsible for successful or unsuccessful listening comprehension, in testing situations, however, there are additional factors which may cause comprehension prevention. One of these factors is the testing environment itself: the acoustics in the exam room, the quality of sound in the recordings (especially, when speech is not studio recorded), technical problems with the audio equipment, intentional or unintentional background sounds and noise inside or outside the exam room may seriously affect comprehension. Of course, test performance is contingent upon skills and characteristics of individual candidates and these are not always related to their language ability and knowledge. They have to do with how well different candidates have learnt to retain information, how anxious they get during an exam situation and whether they have developed test-taking skills such as speed in responding, self-evaluation and self-corrections, etc. But candidate’s individual characteristics are just as important as group characteristics. From the research we are carrying out, it seems that there are rather significant differences between how different ethnic, social and age groups perform when assessed for listening comprehension, due to a series of factors. Older candidates respond differently to the same listening texts and tasks than younger candidates and so do males and females, highly literate and less highly literate candidates, and so on. The knowledge and experiences of these groups play a crucial role in how/what they understand and what responses they select. 

Naturally, the candidate is responsible for the success or failure to understand an oral message, yet s/he is not the only one to blame. The end result has a lot to do with the language of the text, how the text is delivered (spoken), and the nature of the task. Now, choices of texts and tasks are directly related to the approach to language and to the language testing aims of each exam battery. This means that research which is not candidate, oral text and task specific has very little to tell us about listening comprehension difficulties in testing situations.

We have seen very few studies in the literature drawing upon actual data and reporting findings related to one particular testing system. However, the KPG English team is aspiring to complete the project it began in 2007. In order to investigate candidate response to particular texts and tasks, a variety of tools have been used (questionnaires, interviews and verbal protocols). Systematic task analysis is being carried out as well as post administration item analysis is being used, not only to investigate difficulty but also to assess the listening comprehension test and ultimately take any measures necessary to secure the reliability of the exam and the candidates’ scores.

Using Item Response Analysis Research (Bachman 2004, McNamara 1996), listening comprehension item analysis is conducted for both the listening as well as the reading comprehension test papers after each administration, and they provide the English test development team with useful information regarding a) internal consistency or reliability of the exam, b) item difficulty (i.e., the proportion of candidates that get an item right or wrong), c) distractor analysis (i.e., the frequency with which each option of a particular test question is chosen) and d) discrimination efficiency (i.e., how well an item succeeds in distinguishing highly competent from less competent candidates).  Any test item that item analysis shows to have an index of difficulty above 0,80 or below 0,50 is considered to be too easy or too difficult respectively for the exam level (since the normal values of difficulty for a test item should range between 0,50 and 0,80) and this is then analyzed further so that conclusions can be drawn as to what features make the item difficult or easy for the specific group of candidates.

This investigation is then complemented with a systematic examination of the texts from which the tasks originate, in an attempt to find the relationship between text variables and item difficulty. The analysis concerns (a) linguistic features of the text and especially lexical appropriacy to exam level, information structure, information density, (b) paralinguistic features (i.e., accent, speech rate, background noise, visual support, number of speakers involved). All these can, of course, have a serious impact on the level of difficulty of the relevant test items.

Although this part of the research is still at an initial stage, some first conclusions have been drawn as to what task and text-related factors can be associated with item difficulty. For example, it has been determined that even a single unfamiliar word or phrase either in the stem or in the distractors of a multiple-choice item may throw some candidates completely off, while, on the other hand, some candidates seem to do poorly when they must interpret or make an inference rather than get straightforward information from the text. The role of distractors has proven to be an important factor in item difficulty too, since, in a large number of items examined, candidates’ failure to select the correct response can be explained by or attributed to how ‘easy’ or ‘difficult’ the distractors are.  

Distractors can also play a significant role in making items too easy.  This is mostly the case when the distractors themselves are irrelevant to the content of the aural message; thus, making the correct response far too obvious. An additional factor that makes some items too easy has to do with the way the right option is articulated. For example, an item that uses some of the wording of the text is easier than if synonyms are supplied.

The research project is also yielding interesting information about what makes a listening comprehension text difficult or easy but the limited space does not allow us to report findings in detail.[1] As the project develops, reports and papers will be published through the RCeL publications. The outcomes of the particular research are bound to be extremely useful to KPG item writers and test developers. Most importantly, however, they will be a valuable source of information for candidates and teachers preparing them. Once we determine what is difficult for whom, it is possible to teach different groups of learners/candidates how to overcome these difficulties by providing test-taking strategies which will prove helpful for the particular listening comprehension testing situations. 

References

Anderson, A., & Lynch, T. (1988). Listening. Oxford: Oxford University Press.

Bachman, L. F. (2004) Statistical Analyses for Language Assessment. Cambridge: Cambridge University Press.

Buck, G. (2001). Assessing Listening. Cambridge: Cambridge University Press.

Conrad, L. (1985). Semantic versus syntactic cues in listening comprehension. Studies in Second Language Acquisition, 7(1), 59-72.

McNamara, T. (1996). Second Language Performance Measuring. London and New York: Longman.

Rost, M. (1990). Listening in Language Learning. London and New York: Longman.

Ur, P. (1984). Teaching Listening Comprehension. Cambridge: Cambridge University Press.

 [Back]