Bachman, L. and Palmer, A. (2010) Language Assessment in Practice. Oxford: Oxford University Press.
This is a detailed, exhaustive guide to developing and using language assessments. I learned a lot from it about all stages of the language assessment process, including a lot of useful theoretical and practical knowledge about issues surrounding development, administration, and scoring of language assessments. There are numerous examples in the book, and a website of sample materials available for further study. Each chapter also has exercises at the end. I skimmed a lot of it, but read several sections with greater attention. There are several things I got out of this book that I want to explore in more depth and attempt to apply to assessing interpreting performance. To facilitate reading, I’m going to divide it up into multiple post.
Notice: All the information about language use and assessment in this post is from the book cited above, and is noted as such throughout. All portions of this post related to interpreting are my own work and protected by my copyright.
Chapter 3: Describing language use and language ability
The authors define target language use (TLU) domain as “a specific setting outside the test itself that requires the test taker to perform language use tasks” (p.60). In their view, language assessments should, insofar as possible, require those assessed to engage in language use activities that are as close as possible to those that the assessor will need to do in the real-life setting under consideration. For example, if the assessment is going to be used in hiring a person for a job, the actual types of language use involved in that job should be analyzed and used as the basis for the assessment. If the assessment to place students in different levels of language classes, again, the relevant tasks or abilities that make the difference between the levels must be identified at the outset.
In order to know what one wants to test, one needs to define it precisely. This precise statement of the thing to be tested is the construct. Constructs are hard to get one’s mind around, but I found this book very helpful in clarifying the concept for me. I’ll write more about constructs in another post..
The authors note that thinking of language use in terms of reading/writing/listening/speaking abilities is reductive (differentiating only between mode and channel of use) and not the best basis for thinking about language use abilities. They question whether one can define a construct by simply naming a skill, as well. (p. 55)
They identify attributes of language users (ie, people) that are used in language-based tasks: topical knowledge, language knowledge, personal attributes, affective schemata. As a person uses language, all of these attributes plus the person’s strategic competence interact to affect the person’s performance in the language task. “Language ability” is not a broad, static umbrella; ability to use language in different ways and settings varies within the same individual across tasks. The authors use the term “internally interactive” to refer to individual attributes which have an effect on language use, and “externally interactive” to refer to refer to aspects of the language situation itself with which the language user must engage. In their example (p. 35) of person searching for a place to live by reading the classified section of the newspaper, language use in the activity is externally interactive insofar as the reader has to actively engage with the newspaper, its layout, and the language used to describe apartments for rent. The activity is internally interactive because the reader must employ strategic competence (ie metacognitive skills), topical knowledge, and personal attributes to plan the activity (get a newspaper, decide on an approach), decide on criteria for choosing possible apartments (based on personal preferences with regard to features and location, and economic constraints), and so forth.
In this chapter, the authors also differentiate between reciprocal and non-reciprocal language use. Non-reciprocal language use involves the language user and some type of linguistic input, but no second language user (they example mentioned above is of non-reciprocal use); reciprocal language use involves another speaker of the language. On the surface, interpreting seems to be reciprocal; by definition there are at least two parties, and at least one has a message to communicate to the other. In dialogue interpreting, there are by definition at least two parties wishing to engage in back-and-forth interaction. On the other hand, is the interpreter herself engaged in a reciprocal language task? Laying aside for the moment all considerations of clarification of speaker meaning, consider this: the interpreter is generally NOT negotiating meaning with the other participant–the interpreter listens and understands the SL message, retains and reformulates it, and then produces in the TL. It might be possible to understand the cognitive portion of the interpreting task as a non-reciprocal use of language–the interpreter must use her own world and linguistic knowledge (including contextual knowledge of the setting & of the specific encounter, understanding of nonverbal communication, understanding of discourse conventions). Except in the specific instance of dialogue or consecutive interpreters engaging in clarification of meaning, the interpreter does not actively engage in negotiation of meaning with the SL speaker. When reformulating into the TL, the interpreter must again rely on her own cognitive and linguistic knowledge and skills. Instances of clarification from the TL speaker to the interpreter are possible in dialogue settings, but except in specific cases (use of a word that isn’t understood in the TL speaker’s dialect, for example), the interpreter will usually refer these clarifications back to the original speaker rather than initiating a clarification side sequence on her own. Thus, while dialogue interpreting clearly makes possible a reciprocal interaction, it might be useful to think about interpreting as a mix of reciprocal and non-reciprocal language use when defining the constructs we are assessing (whether in practice sessions or exams).
Another useful portion of this chapter is the detailed framework for describing language ability and strategic competence (pp. 44-55). When trying to pin down the constructs one wants to assess in a particular assessment, it would be good to refer to this framework in order to go about the process systematically. The main sections of the framework are organizational knowledge (including grammatical and textual knowledge) and pragmatic knowledge (including functional and sociolinguistic knowledge); these sections are broken down further by the authors. In thinking about interpreters’ linguistic knowledge, we would need to consider the whole framework for each working language. For conference interpreters, one could assess reception only for C languages, reception and production for B, and reception and production (with an emphasis on production) for A languages. For dialogue interpreters, one needs to assess both reception and production for all working languages.
One last concept from this chapter that I want to explore is strategic competence. The authors define this as “higher-order metacognitive strategies that provide a management function in language use, as well as in other cognitive activities” (p. 48). They list three categories of strategic competence or metacognitive skills: goal setting, appraising, and planning. The chapter includes concrete examples of these strategies in action. These levels of metacognition are employed in language use; the dialogue interpreting task, which of its nature consists of language use in two directions, plus other tasks such as situational management and dual-tasking (in the case of simultaneous or note-taking),
Chapter 4 of the book (Describing characteristics of language use) explains the authors’ concept of TLU domains in detail, and gives examples. The two principle types of language use domains they identify are language-teaching-domains (ie, school) and real-life domains. A test developer must have a clear idea of the domains and tasks that will be expected of the language user in order to develop an assessment that will test the abilities that will be needed–and not some other set of abilities. They give an example (p.62) of the domain “English Business Communication,” which includes producing written reports and answering phones (in the office) and demonstrating products and troubleshooting problems (in the field). Thus, if one was wishing to assess language ability before hiring a bilingual person to fill this role, the assessment would need to reliably elicit the person’s ability to handle the specific tasks identified for this area.
In thinking about interpreting as a TLU domain, we have both language use tasks and interpreting tasks. What would a framework of interpreting abilities look like? How would we define the constructs we are assessing? These are only preliminary thoughts.
The language use tasks involved in interpreting include listening to & fully understanding the meaning of a message in the SL (which implies analysis–a cognitive task) and producing a grammatically-correct, pleasant-to-listen-to rendition in the TL. The middle portion of this process involves retention of the information and reformulation of the meaning so that it can be produced intelligibly in the other language. These processes are certainly affected by language abilities, even though they may happen a-lingually (depending on which theoretical stance you’re working from); this would be an example of an interactive process. The ability to analyze and retain information is affected by one’s base level of listening ability AND one’s analysis skills AND concentration skills AND topical knowledge. Production involves the ability to produce grammatically-correct utterances at high speed, AND public speaking skills AND knowledge of discourse structure and pragmatics, and so forth. Metacognitive skills are also involved; we know from IS research that the ability to self-monitor for mistakes and re-allocate cognitive resources based on current needs plays a role. Situational management skills are also key, especially for dialogue interpreters; interpreters need to learn to juggle all the cognitive tasks inherent in interpreting PLUS manage their own personal space and feelings (fatigue, emotional reactions to difficult situations), others’ behaviors (interruptions, multiple speakers, turn taking, etc), and environmental factors (ambient noise, physical discomfort), as well as being aware of and sometimes actively managing the flow of discourse and culture-based miscommunication.
The interpreting assessment tool I currently use looks at Accuracy & Completeness, Code of Ethics/Standards of Practice, Linguistic Skills, and Delivery. Based on the ideas discussed above, plus the chapters of this book on constructs & scoring assessments, I am hoping to refine my conceptualization of what a target interpreting domain might look like, and what tasks it might involve. With that, I will be able to better define the constructs I am assessing, and improve the scoring tools that I use.
More to follow….