Jenny Liontou
FACTORS AFFECTING THE COMPREHENSION DIFFICULTY OF TEXTS: THE KPG TEST-TAKERS’ BELIEFS AND READING EXPERIENCES[1] 

Abstract

This paper[2] examines test-takers’ attitudes to the texts and tasks of the reading comprehension module of the KPG English language exams. As such, it constitutes part of ongoing doctoral research on the effect specific text and reader variables such as text organisation, genre, content, readers’ background knowledge, topic preference and familiarity, sex, age, exam preparation, place of residence, etc. have on text comprehensibility. Findings from a longitudinal survey conducted at the RCeL[3] on a national scale in the form of questionnaires administered to EFL learners sitting for the KPG English language exams are presented in this paper which discusses text and task difficulty from the test-takers’ perspective i.e., their attitudes towards and opinions about the lexical complexity of the texts, the difficulty of the tasks and their familiarity with or preference for specific text topics.

Keywords: reading comprehension, reading attitudes, text difficulty, readability

1. Introduction

Over the last twenty years there has been an increasing interest in exploring and better defining the interaction between the reader and the text, rather than text features alone, acknowledging that reading is a multifaceted phenomenon, reflecting properties of both the text and the reader and the interaction between them (Keshavarz et al., 2007: 20; Rupp et al., 2006: 445; Drucker, 2003: 25; Bachman, 2002: 464; Parker et al., 2001: 308; Alderson, 2000: 33). Especially in the field of language testing, a number of researchers have highlighted the significant effect of reader factors on comprehension and further supported the view that a satisfactory understanding of the reading process depends on clearly describing the impact readers’ personal characteristics such as sex, age and years of instruction in a foreign language as well as prior knowledge of and interest in the topic, can have on overall exam performance (Pomplun & Omar, 2001: 171; Stricker et al., 2001: 205; Sullivan, 2000: 373; Wightman, 1998: 255; Sunderland, 1993: 47). As Sullivan has stressed (2000: 373-4) it is necessary for test-designers to better define any learner variables that may affect exam performance in order to be able to detect and eliminate any test features that can be sources of measurement error and thus distort their attempts to achieve an accurate assessment of learners’ language proficiency. To this end, a number of researchers have called for more research in order to enhance our knowledge about the factors contributing to text difficulty while at the same time examining the interaction between text and reader variables in order to determine how, when and why one mediates the other under high-stakes reading test conditions (c.f. Shokrpour, 2004: 15-16; Phakiti, 2003: 651; Ungerleider, 2003: 122; Bailin & Grafstein, 2001: 298; Chalhoub-Deville & Turner, 2000: 528; Saito et al., 1999: 216; 212; Purpura, 1997: 290; Hock, 1990: 215; Nevo, 1989). Given this need, the current research has been designed to add to our present state of knowledge on reading comprehension by exploring text difficulty from the test-takers’ point of view and examining the effect a number of reader variables can have on perceived task and reading module difficulty.

2. Literature review

The extensive research conducted in the field of second language acquisition with specific reference to ways of reading and text-processing strategies has shown that several reader and text variables can have an impact on either the reading process or reading product and, as such, need to be taken into account during test design and validation (c.f. Carr, 2006: 282; Ben-Anath, 2005: 1; Brantmeier, 2005: 37; Mesmer, 2005: 2; Phakiti, 2003: 651; Leong et al., 2002: 130; Lin et al., 2002: 187; Read, 2000: 191). In fact, as pointed out by Alderson (2000: 81) “If the reading process or product varies according to such influences, and if such influences occur in the test or assessment procedures, then this is a risk to the validity of test scores, to the generalizability of results or to the interpretation of performances”. While differences in the test performance of different groups of test-takers has been a long-standing interest among language testers, the past decade has seen increased research in this area, both in terms of test-takers’ personal characteristics such as sex, age, native language and years of instruction in a foreign language as well as psychological factors such as feelings of anxiety under testing conditions.

Especially in relation to sex, as Brown & McNamara (2004: 525) acknowledged, the issue of sex-related effects in educational tests has been a concern of psychometrics in order for test developers to ensure that test items are not biased against particular subsets of the testing population due to their sex, ethnicity or age inter alia. Sunderland (1995: 32) and Brown & McNamara (2004: 524) further emphasized that, since test results can affect people’s lives, detecting sex bias, it is not only a professional issue, but also an ethical, even political one. Despite the importance of detecting sex-based bias in foreign language reading performance, the relevant literature is rather limited and its findings far from consistent (Phakiti, 2003: 652; Coles & Hall, 2002: 97; Garner & Engelhard, 1999: 29; Moreno & Mayer, 1999: 355; Pomplun & Sundbye, 1999: 95). For example, Wen and Johnson (1997: 27) reported that females outperformed males on a standardized national proficiency test. On the other hand, Boyle (1987: 273) found that although female Chinese university students of English as a foreign language scored higher overall in a language proficiency test, males scored higher than females on listening vocabulary, which might be related to reading comprehension. Scarcella and Zimmerman (1998: 27) also found that males performed significantly better than females on specific types of vocabulary tests.

Although a lot of research has focused exclusively on the role of prior knowledge in reading comprehension, Bügel and Buunk (1996: 15) investigated the impact of both reading interests and prior knowledge on the performance of Dutch secondary students on English reading tests. Their data analysis revealed significant differences in test results, which, according to the researchers, appeared to be largely due to sex-based differences in reading habits that consequently had an impact on students’ prior knowledge and topic interests (ibid: 16). In a more recent study, Brantmeier (2003: 11) also investigated the effect of sex-based differences and topic familiarity on second language reading comprehension performance of intermediate learners of Spanish and, similarly with Bügel and Buunk, reported significant interactions between readers' sex and passage content with reading performance.

Much earlier, Allen et al. (1988: 164) investigated personal factors by asking learners of French, German and Spanish to read four passages and report on their thoughts while reading. Statistical analysis of the protocols showed that years of instruction had a significant effect on subjects’ text comprehension (ibid: 169). In contrast with Bernhardt (1986: 26) who argued that comprehension is text-dependent, Allen et al. found that text-based factors such as text genre did little to explain the reading abilities of second language learners (ibid: 170). In other words, when statistical analysis of this study was applied exclusively to the texts no effects were traced, whereas, when readers’ characteristics were evaluated in conjunction with the texts, significant effects did emerge (ibid: 170). In another study, Ilyin et al. (1987: 158) investigated the effect of two specific learner variables i.e., age and high-school graduation, in the test performance of adult learners of English and age was found to have a significant impact on their scores in the reading and listening modules of the exam.

The effect of topic familiarity on test-takers’ performance has always been considered a fundamental issue in validating reading comprehension test scores (c.f. Nassaji, 2007: 83-85; Alptekin, 2006: 494; Calvo et al., 2006: 68; Ketchum, 2006: 22; Krekeler, 2006: 99; Pulido, 2004: 503; Wylie & McGuinness, 2004: 497; Ellis, 2001: 34; Barry & Lazarte, 1998: 190-1; Bensoussan, 1998: 213). In 2004, Pulido examined the effect of topic familiarity on second language reading comprehension and retention of new lexical items from printed texts. On average, the participants recalled more semantic propositions from the more familiar stories in comparison to the less familiar ones, which, according to Pulido, shows that their text comprehension increased due to topic familiarity (ibid: 489). However, in another study carried out by Jennings et al. (1999) regarding the Canadian Academic English Language test battery, it was found that factors such as test-takers’ interest in the topic, prior knowledge or perceived relevance of the topic had no significant effect on their overall performance, and as a result were not a potential threat to the validity of the test (ibid: 448). Despite these findings, according to the authors, the fact that test-takers expressed strong concerns about the topic of the test indicated that it was a salient factor to them (ibid: 449). Finally, Hong (1999: 431) investigated the relationship between students’ perception of test difficulty and actual performance by analyzing questionnaire and test data collected from 208 undergraduate students and found that although perceived test difficulty did not have a direct effect on students’ performance, it had an indirect effect on the arousal of anxiety i.e., students who perceived the test as more difficult showed higher levels of worry and anxiety, which in turn influenced their performance as measured by their final scores (ibid: 432). Hong concluded by pointing out the importance of understanding students’ perception of test difficulty because its impact on the arousal of anxiety can further lead to impaired performance (ibid: 432).

Undoubtedly, performance in language tests can be influenced by a wide range of factors, which can interact in an unpredictable way with the characteristics of individual test-takers (Lumley & O’Sullivan, 2005: 415, Woodfield et al., 2005: 45; Horiba, 2000: 223). Having said that, does not, however, mean that test developers should remain passive observers of such interactions but rather try to eliminate their effect and control for any sources of bias and error in their tests. In fact, as Brown & McNamara (2004: 524) and Shohamy (2001: 383-4) artfully pointed out, a responsible professional practice in language testing involves care for the rights and interests of particular social groups who may be at risk from biased language assessments. Test designers’ knowledge of the variables that can influence the reading process and product, which is the main objective of our ongoing research project, is, thus, in many respects linked to the validity of the reading tests i.e., test designers need to know whether and to what extent the instruments they use to measure reading ability are, amongst other things, affected by the readability[4] of the texts, the difficulty of the tasks, or the readers’ background knowledge.

3. Aim and methodology

The aim of the research presented in this paper has been twofold: a) to provide a description of KPG test-takers’ personal characteristics for all interested parties such as test-developers and material writers to have a clearer picture of their intended audience and b) to examine whether and to what extent specific reader variables such as sex, age, years of instruction, background knowledge and topic preference influence KPG test-takers’ impressions of text and task difficulty. Taking into consideration test-takers’ feedback and in a way treating them as “judges” of reading texts is a crucial part of the present research, since they were the actual readers who had to interact with the texts in order to perform a set of tasks and demonstrate a successful performance in the specific exams. So, their feelings and opinions about the texts are believed to be of great importance since they can give us more in-depth information about text difficulty from the reader’s point of view. At this point it should also be mentioned that most studies pertinent to EFL comprehension processes and reading performance involved a small number of EFL learners taking part not in real, high-stakes exams but in experiments designed to explore a limited number of reader or text variables in isolation and which had in many cases produced rather contradictory results. With particular reference to the KPG English language exams, no empirical evidence was available on what features make a text easier or more difficult for Greek learners of English to comprehend or which topics are more familiar to them and could better suit their interests and reading preferences. Furthermore, the idea of examining the effect not only of text characteristics but also of personal characteristics on test-takers’ perceptions of reading module difficulty comes to question a rather common belief that since nothing can be done to control these factors there is no need to look into their impact; to the contrary, it is hoped that such an investigation will make us aware of the impact less explored factors may have on overall exam performance. In order to explore these issues, the following research questions were formed:

1. Which is KPG test-takers’ profile in terms of personal characteristics such as age, sex, years of learning English and place of residence? Are there any significant differences between B2 and C1 test-takers with regard to these characteristics?

2. To what extent do specific reader variables contribute to KPG test-takers’ perceptions of reading texts and tasks difficulty?

3. Is there a relationship between text characteristics and test-takers’ perceived level of text difficulty?

In order to collect valuable information on the KPG test-takers’ profile and reading preferences as well as their perceptions of text and task difficulty, a national survey was conducted by the RCeL with the support of the Greek Ministry of Education & Religious Affairs in the form of questionnaires administered to all KPG test-takers sitting for the B2 and C1 English language exams in the May and November 2006 and 2007 and May 2008 examination periods. The questionnaire was administered on the day of the exam, in exam centres all over Greece. More specifically, once test-takers had finished their exam and before leaving the examination room, they were kindly requested to provide some personal information regarding their age, sex, place of residence, years of learning English and preparation for the KPG exams. They also rated on a Likert scale of 1-5 or 1-3 a variety of statements about module, task, text and instructions difficulty, topic and genre familiarity, topic preference, text lexical complexity and text processing strategies (Appendix A). The design of the questionnaire was rather time consuming and required in depth research on relevant literature, careful piloting and refining for its final version to contain a comprehensive range of questions. However, this effort proved its worth since the collected data has yielded valuable information from the test-takers’ point of view that would not have been possible to otherwise obtain. The actual process of designing the questionnaire involved the clarification of the research purpose and the formulation of specific questions that corresponded to this purpose. More specifically, since the aim of the study was to investigate the effect specific text and reader variables have on test-takers’ perceptions of reading comprehension difficulty, the questions were designed to elicit information related to these variables, i.e., topic familiarity, topic preference, text lexical complexity and employed reading strategies. Bearing in mind the fact that the larger the sample size (n), the more reliable the findings, the present questionnaire was administered to a rather large sample (i.e., all test-takers sitting for the B2 and C1 English language exams), and thus had to include structured, closed questions which test-takers could answer by selecting a numerical value. These kinds of questions are, according to Cohen et al. (2000: 251), very useful in large-scale surveys since they enable collection of data amenable to thorough statistical analysis and allow comparisons to be made across groups, facilitating the observation of patterns occurring in the same sample. To avoid dichotomous questions, a Likert scale was employed, since such a scaling can provide a range of responses with a degree of sensitivity while still generating numerical values (ibid: 251). As can be seen in Table 1, to date 7.250 questionnaires from five examination periods have been analyzed, 4.750 in relation to the B2 level and 2.500 in relation to the C1 level.

Moreover, in order to get a general idea of the readability level of reading texts and to check whether specific readability formulas could be of any practical usefulness within the context of the KPG exams, the readability level of all reading texts used in past KPG exams was electronically estimated using four readability formulas[5], namely the Dale-Chall Readability Formula, the Flesch Reading Ease Index, the Gunning's Fog Index and the Flesch-Kincaid Grade Level Index, and two computer programs, i.e., the AceReader Pro Deluxe and the TextAnalyser Version 2 software (for more information on readability formulas see Lane, 2004; Leong et al., 2002; Bailin & Grafstein, 2001; Dreher & Singer, 2001; Fulcher, 1997; Oakland & Meade & Smith, 1991; Pichert & Elam, 1985; Fry, 1989; Klare, 1984). To date, 24 B2 level texts and 24 C1 level texts have been processed, resulting in a total of 192 analyses. Along with readability indices, additional text features that may contribute to text difficulty at word and sentence level have been measured, such as word length, number of sentences, average characters per word, average words per sentence, average number of syllables per 100 words, average number of sentences per 100 words, and lexical density. The latter, as defined by Halliday (1996: 347), refers to the proportion of content (lexical) words – basically nouns, full verbs, adjectives, and adverbs derived from adjectives – over the total number of words in a text.

Most importantly, an attempt was made to investigate whether text lexical diversity (D) could provide a consistent measurement tool of text difficulty. Lexical diversity refers to ‘the range of vocabulary displayed in the discourses,’ as roughly defined by Durán et al. in 2004, and has been extensively mentioned in literature related to learners’ written or spoken production. Actually, as Jarvis (2002: 57) explains, it is considered an essential indicator of learners’ quality of output, but has not yet been extensively applied to reading texts. Measuring lexical diversity is a rather complex task that has troubled researchers across several disciplines and has triggered the development of different formulas, amongst which the most widely used has been the type-token ratio (TTR) i.e., the ratio of different words to total words used in a text. However, the traditional TTR has been considered sensitive to sample size and, according to Richards (1987: 203), renders itself incapable of assessing lexical diversity in a satisfactory way if the sample sizes vary a great deal. More recently, another transformation of the traditional TTR was developed by Malvern and Richards (2002: 85) and incorporated into the vocd programme, which is included in the CLAN suite of programmes in the Child Language Data Exchange System (CHILDES). A minimum sample size of 50 words-tokens is required to compute a valid D. In this process, data first has to be transcribed in a standard format (CHAT), following a specific coding system (which in fact makes the whole procedure rather time consuming), but once the text has been checked and the coding accepted, the software calculates the average of Ds three times and reports a final optimum D value for a text. According to its designers, D presents methodological advantages over other measures of lexical diversity since it is independent of sample size and makes it possible to perform valid comparisons between varying quantities of linguistic data (Malvern & Richards, 2002: 85).

As previously stated, up until now measurement of lexical diversity has been mainly confined to spoken data produced by learners. However, given the fact that lexical diversity relates to vocabulary and that vocabulary itself is an important element of reading texts, the idea of investigating the relation between lexical diversity (D) and text complexity could not remain unexplored. After personal communication with Professor Malvern (in 19/03/2008), such an application was considered to be viable so in the following section, some preliminary findings of one such investigation are presented.

4. Results and discussion

4.1 KPG test-takers’ profile

Descriptive statistics have shown that the KPG B2 level test-takers’ age in all five exam periods ranges from 12 to 62 years. Although age values are scattered, indicating lack of normal distribution, most values are found within the range of 15-20 years, with 15 being the most frequent age for test-takers sitting for the B2 exam. As far as their sex is concerned, women outnumber men in all five exam periods. Regarding the number of years of learning English, the majority of test-takers have stated that they had been learning English for about six to eight years before sitting for the exam, and also that they have been especially prepared for it (which means that they were at least partly familiar with the format and type of activities included in the exam). With regard to the KPG C1 level test-takers’ profile, age values in all five exam periods range from 12 to 48 years. Although age values follow no normal distribution, most of them occur within the range of 16-24 years, as was the case for B2 test-takers, with 16 being the most frequent figure. As far as sex is concerned, once again women taking part in the C1 exam outnumber men in all five exam periods. In relation to years of learning English, the majority of C1 test-takers seem to have been learning English for about seven to nine years before sitting for the exam, a bit longer than the B2 ones. However, as opposed to the B2 findings, a lower percentage of test-takers sitting for the C1 exam have stated that they have undergone some kind of special preparation before sitting for the specific exam.

4.2 KPG test-takers’ personal characteristics & reading module difficulty

Statistical analysis has shown that age as a predictor variable can have an effect on test-takers’ perception of reading module difficulty, topic familiarity, topic preference, and text lexical complexity. More specifically, ordinal and logistic regression analyses revealed that at a 0.05 significance level, the older the test-takers, the higher the probability of finding the reading module more difficult and the text language more complex to comprehend (Table 2-Appendix B). However, at the same time, older test-takers seem to be more familiar with and to enjoy reading about specific topics, such as environmental issues, than their younger counterparts.

Years of learning English as an independent variable can also be used to a limited extent to predict test-takers’ responses. Regression analysis has shown that as the years of learning English increase, test-takers are more likely to be familiar with certain topics and to find the reading module less difficult and the text language less complex to comprehend. Surprisingly enough, a significant effect of sex on test-takers’ perception of module difficulty has also been observed since in most B2 and C1 examination periods, it has been found that women are more likely than men to find the vocabulary and the rubrics less difficult to comprehend and the items less difficult to answer, but also to be less familiar with certain topics i.e., environmental issues like climate change and recycling (Table 3-Appendix B).

Although no significant effect of place of residence on overall module difficulty has been observed, it should be pointed out that test-takers from urban areas seem to have been more familiar with and liked reading about topics of rather special social interest (i.e., multilingualism in the United States, Parkour, Common History Books in Europe) than those living in rural regions. In May 2007, a question regarding preparation for the KPG exams was added in the questionnaire to test the hypothesis that test-takers’ familiarization with the format and content of the specific exam has a positive impact on their perception of module difficulty. Data analysis of three consequent examination periods showed no significant effect of exam preparation on test-takers’ perception of text and task difficulty. However, in May 2007, a significant effect of exam preparation on test-takers’ perception of text difficulty was observed i.e., test-takers specially prepared for the KPG exam found the text language less complex to comprehend and the items less difficult to answer.

4.3 KPG test-takers’ perspective on module difficulty

Data analysis of KPG B2 test-takers’ perspective on overall difficulty of the reading comprehension module has shown that in all five exam periods the majority of responses ranged from ‘slightly’ to ‘fairly difficult,’ and only a small percentage of test-takers found the module ‘very’ or ‘extremely difficult’ (12% in May 2006, 9% in November 2006, 16% in May 2007, 8,9% in November 2007 and 10.7% in May 2008). Task instructions were mainly treated as ‘not difficult’ or ‘slightly difficult.’ However, as regards expected level of text and item difficulty, a noticeable difference has be traced between May and November 2007, since in May 2007 a much lower percentage of test-takers found the texts and the items as difficult as expected, whereas in November 2007 the majority of participants (almost 80%) stated that texts and tasks were as difficult as expected. Such a discrepancy calls for further investigation in order for specific patterns of difference to be identified.

As far as C1 test-takers are concerned, their perceptions of reading module and instructions difficulty are quite similar to those of the B2 level test-takers, i.e., in all five exam periods, the majority of participants found the reading module ‘slightly’ to ‘fairly difficult,’ and the task instructions ‘not difficult’ to ‘slightly difficult.’ However, when looking at the expected level of text and item difficulty, a discrepancy can once again be observed between exam periods: in November 2006, a much lower percentage of participants found the texts and the items as difficult as expected, in comparison with May 2006, November 2007 and May 2008.

4.4   KPG test-takers’ perspective versus textual features

The comparative examination of test-takers’ questionnaires and textual features has provided preliminary insight into text comprehension constraints on the part of the reader. In Table 4, data regarding B2 test-takers’ perspectives on text lexical complexity for the B2 level May and November 2006 exams are presented (Column 1) and cross-related to four readability formulas (Columns 2-4), and estimates of the relationship between the lexical diversity of a text and its readability (Column 5). All four readability formulas seem to fail to agree with test-takers’ perspective on text lexical complexity, since texts are mainly rated as ‘very difficult’ by readability formulas and appropriate for advanced readers (i.e., above average readers). On the other hand, if we compare data in the first and last columns, there seems to be a better correspondence between participants’ responses and lexical diversity (D), since, in most cases, as the percentage of participants who found the text ‘very’ to ‘extremely difficult’ increases, so does the lexical diversity of the text.