BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:How do examiners reach judgements? - Victoria Elliott (Oxford Univ
 ersity Centre for Educational Assessment and Warwick Institute of Educatio
 n) and Dr Talia Isaacs (University of Bristol)
DTSTART:20120906T153000Z
DTEND:20120906T173000Z
UID:TALK34862@talks.cam.ac.uk
CONTACT:14709
DESCRIPTION:Examiner judgements are an essential part of the assessment pr
 ocess. In this session there will be two presentations looking at how exam
 iners in different contexts arrive at their judgements\, and what variable
 s have an influence on the judgements that are made\n\nCognitive cherry pi
 cking: the patchwork process of examining A level essays (Victoria Elliott
 )\n\nThe assumption of the A level system is that examiners make their dec
 isions in a rule-based\, logical way. However\, given the time constraints
  under which examiners work\, and the amount of information which must be 
 considered\, it is unlikely that the cognitive process is as rational as i
 t is conceived to be. Essay subjects\, such as history and English (two su
 bjects which are associated with difficulty of marking)\, are likely to fo
 rm a particular challenge in terms of the limits of working memory. The fa
 ct remains that most examiners mark within accepted margins of reliability
 . How\, then\, do they make these decisions\, given the amount of informat
 ion and the limited time available?\n\nThis paper considers some of the da
 ta gathered during an in-depth study of examiners’ training and decision
 -making\, utilising Verbal Protocol Analyses of live marking and recording
 s of training meetings. A wide range of cognitive processes are demonstrat
 ed in the data\, which are used to varying degrees by different examiners\
 , at different times within and between scripts\, according to the most us
 eful\, and potentially the most economical method at any one time.\n\nInfl
 uences on rater judgements of second language speech (Dr Talia Isaacs)\n\n
 The overall goal of this presentation is to examine rater perceptions of s
 econd language (L2) speech to identify possible sources of rater bias and 
 shed light on the focal construct. First\, we will examine the effects of 
 individual differences in rater cognitive variables (musical ability\, pho
 nological memory\, and attention control) on raters’ judgements of L2 co
 mprehensibility (ease of understanding)\, accentedness (degree of foreign 
 accent)\, and fluency (speed of delivery and smoothness). If these cogniti
 ve variables are found to exert a measurable influence on the scores that 
 raters assign\, then this could pose a threat to the validity of the asses
 sments. Next\, we will examine the way that ‘comprehensibility’ is ope
 rationalised in current L2 speaking scales (e.g.\, Cambridge ESOL Common S
 cale for Speaking\, IELTS pronunciation scale). We will then examine how a
  greater understanding of the linguistic dimensions that underlie listener
 s’ L2 comprehensibility ratings can elucidate our understanding of the c
 onstruct at different levels of ability. Implications for rating scale val
 idation and for rater training in high-stakes assessment settings will be 
 discussed.\n
LOCATION:Downing College\, Regent Street\, Cambridge\, UK
END:VEVENT
END:VCALENDAR