Abstract

Examiner judgements are an essential part of the assessment process. In this session there will be two presentations looking at how examiners in different contexts arrive at their judgements, and what variables have an influence on the judgements that are made

Cognitive cherry picking: the patchwork process of examining A level essays (Victoria Elliott)

The assumption of the A level system is that examiners make their decisions in a rule-based, logical way. However, given the time constraints under which examiners work, and the amount of information which must be considered, it is unlikely that the cognitive process is as rational as it is conceived to be. Essay subjects, such as history and English (two subjects which are associated with difficulty of marking), are likely to form a particular challenge in terms of the limits of working memory. The fact remains that most examiners mark within accepted margins of reliability. How, then, do they make these decisions, given the amount of information and the limited time available?

This paper considers some of the data gathered during an in-depth study of examiners’ training and decision-making, utilising Verbal Protocol Analyses of live marking and recordings of training meetings. A wide range of cognitive processes are demonstrated in the data, which are used to varying degrees by different examiners, at different times within and between scripts, according to the most useful, and potentially the most economical method at any one time.

Influences on rater judgements of second language speech (Dr Talia Isaacs)

The overall goal of this presentation is to examine rater perceptions of second language (L2) speech to identify possible sources of rater bias and shed light on the focal construct. First, we will examine the effects of individual differences in rater cognitive variables (musical ability, phonological memory, and attention control) on raters’ judgements of L2 comprehensibility (ease of understanding), accentedness (degree of foreign accent), and fluency (speed of delivery and smoothness). If these cognitive variables are found to exert a measurable influence on the scores that raters assign, then this could pose a threat to the validity of the assessments. Next, we will examine the way that ‘comprehensibility’ is operationalised in current L2 speaking scales (e.g., Cambridge ESOL Common Scale for Speaking, IELTS pronunciation scale). We will then examine how a greater understanding of the linguistic dimensions that underlie listeners’ L2 comprehensibility ratings can elucidate our understanding of the construct at different levels of ability. Implications for rating scale validation and for rater training in high-stakes assessment settings will be discussed.