“Understand that there is always one clearly best answer. My goal is not to trick students or require them to make difficult judgments about two options that are nearly equally correct. My goal is to design questions that students who understand will answer correctly and students who do not understand will answer incorrectly.”
John A. Johnson (in Clay, 2001)
In a report completed for the Canadian government, Makosky (2008) indicated that, at the time of writing, exit test results from LINC programs across the country were “deemed to be subjective/situational and not comparable to any common standard,” with the result that “exit rating and feedback to newcomers may be inconsistent and the results not as portable as newcomers, instructors and program managers would like” (p. 2). Because of the lack of transparent, reliable outcomes, LINC’s funder, Citizenship and Immigration Canada (now Immigration, Refugees and Citizenship Canada), determined that a consistent standard of exit assessment needed to be developed for the national language program. Thus in 2014, after years of development and piloting, Portfolio-Based Language Assessment, or PBLA, was introduced to LINC. Since that time, Regional Coaches have certified four cohorts of Lead Instructors to train and guide over 2,000 classroom instructors in PBLA implementation.1
There is a lot to learn with PBLA, especially for teachers with no formal instruction in assessment. While PBLA training is comprehensive so that teachers understand the methodology and practices to apply in the classroom, there are some specific skills for designing assessments that PBLA implementation workshops cannot address in depth. One of them is how to write effective CLB-aligned comprehension question or other assessment items to assess listening and reading.
It’s not hard to go wrong in writing items to check for understanding of text. Through various roles guiding and supporting teachers in LINC, patterns with writing ineffective comprehension items have emerged. These are common mistakes:
- Items do not check for CLB criteria.
- All or most items check for details only.
- There are assumptions of understanding (e.g. if students can find a detail, they must understand layout).
- Criteria that are harder to write items for, such as understanding of implied meaning, are often not checked for in assessment, even when key to a task or level.
- Items do not focus on comprehension of text.
- Items ask for background knowledge of the topic, not what is in the text.
- Items don’t demonstrate what students know (e.g. true/false questions where the answer is false and a correct answer is not required to be given).
- Items focus on understanding of grammatical elements such as vocabulary, at the expense of important textual and pragmatic elements.
- Items give points for productive skills (skills in writing a response) which are irrelevant to checking for comprehension.
- Items do not check for appropriate levels of comprehension (levels identified in Holmes, Habke, & Schmuck, 2017).
- The wording of the items is confusing or exceeds the learners’ CLB level.
Note that if the wording of the question is too complex, a teacher won’t know if students couldn’t answer a question because they didn’t understand the text they were listening to or reading, or because they didn’t understand the comprehension question.
To avoid these problems and write more effective comprehension items that clearly check for CLB criteria, teachers need to keep in mind what is to be accomplished with classroom-based assessment. As illustrated by the opening quote, PBLA assessment is NOT trying to
- Challenge or test students’ limits,
- Trick students or trip them up,
- Divide students into those passing and failing, or
- Measure abilities against others in the class.
Such outcomes may be part of testing in academic courses or professional/trades training, but they are not the goal of Portfolio-Based Language Assessment. PBLA is simply trying to gauge if learners can accomplish necessary tasks in the real world, at the proficiency indicated in the CLB document for the level. No more, but also no less. Teachers need to check for the key indicators associated with the task at the level, in order to reliably say that learners meet the CLB standard. To achieve this, comprehension items should clearly reflect the indicators and be simple and direct. Higher CLB levels do not need more complicated items or task types. Higher CLB levels are measured by the wider and deeper expectations in the Indicators of Ability, and demonstration of understanding of longer and more complex text, not the difficulty of the items. For example, if the criterion is to “Identify purpose,” the question should be “What is the purpose of the reading/listening text (i.e. the brochure/advertising commercial)?” Remember learners at all levels should have been skill-building on identifying and understanding the components of the task well before encountering criteria in an assessment, so such a question should not be new.
Choosing Question or Task Types to Check for Comprehension
It helps to be aware of different types of question or task types so that an appropriate type can be chosen for each objective and cognitive level being assessed (Jacobs, 2004; Writing Effective Questions, n.d.) with the Indicators of Ability.
Type 1: Demonstrate/show me/respond appropriately
For example:
- Respond appropriately in conversation to show recognition of greetings, introductions, requests, etc.
- Draw a line on a map to demonstrate following instructions
- Point to a product in a flyer to show understanding of details
This type of task, which mostly demonstrates literal comprehension, is good for indicators requiring physical or verbal responses. Remember just responding appropriately in a conversation is not necessarily enough to ensure learners are meeting all the relevant CLB criteria for the task and level. For example, just responding to a co-worker’s message to say that the learner will switch a shift does not indicate what specifically has been understood about any implied meanings or factual details in the message. Additional comprehension items or tasks are generally needed.
Type 2: Matching or sequencing
For assessment, matching must reflect only what is in the text. Matching topic words to pictures or definitions becomes a vocabulary test, which is not part of assessing ability to complete a real world task. Even good matching items take learners’ focus away from comprehending the text to instead sorting the correct answers provided. For that reason, matching is best used for skill-building rather than assessment.
For sequencing items such as ordering the steps of instructions (Interpretive comprehension), be careful to count one whole sequence as one question only, to reflect that identifying or interpreting sequence markers is only one Indicator of Ability. Giving points for individual steps puts emphasis on the score rather than on the criteria where it should be. If learners get an early step out of order, the score could be very low, but one mistake does not necessarily mean the learners cannot sequence.
Type 3: True/False
True/false questions are very difficult to write in a way that both gives clear information about what a learner knows and are not too easy or too difficult to answer. For example, if “false” is the correct answer, the teacher won’t know if students know the information, only that they know what is not the information. As well, teachers often use negatives to make statements false, such as “The Airporter doesn’t always go to the Convention Centre.” (Question based on CLB Support Kit Reading Document G). This can be confusing wording as students then need to unravel the negatives: is the negative statement the positive/true answer, or is the negative statement the negative/false answer? That said, written well, true/false questions can be a good check for details or simple concepts.
Tips for writing true/false questions (Clay, 2001; Jacobs, 2004; Writing Effective Questions n.d.).
- Write a series of true statements, then change slightly more than half to false as students usually answer “true” if guessing (Jacobs, 2004, p. 13).
- Use common mistakes, such as confusing main idea with purpose, or identifying a literal instead of an implied meaning, as statements where the answer is false.
- Only have one idea in each statement.
- Don’t use qualifiers that create ambiguity such as all, none, never, sometimes, generally or often.
- The statement should be absolutely true or absolutely false, not arguable.
- If the answer is false, have learners correct it to be sure of what they know.
- Have students circle T or F instead of writing so there is no question about handwriting.
Type 4: Fill-in-the-blank/completion/fill in a chart
Completion items or filling in a chart are great for checking facts or details, when learners need to listen and recall or scan and copy (Literal comprehension). A bank of answers/options to select from should be avoided because it can reduce the task to a matching exercise rather than keeping the focus on the reading or listening text.
Type 5: Short response
Short-response items are good for checking understanding of single ideas and clear right or wrong responses demonstrating interpretive comprehension. Usually a short response is a word, phrase, or one or two sentences only. Short-response items can be used for interpretive comprehension items such as identifying purpose, main idea, etc. in higher CLB levels where learners can express themselves easily in a few words or a single sentence. Remember, short-response items should have clearly right or wrong responses. If a factual response is required, don’t use opinion stems such as Do you think because an opinion cannot be wrong!
Type 6: Long response (“essay question”)
Long-response items are very good for checking upper levels of comprehension (Applied) when learners need to reference the text in their response with explanations, examples, an opinion, or an analysis.
For both short and long response, as well as completion items, be sure to not mark production skills: grammar, mechanics, etc. Remember we are only checking comprehension with a listening or reading assessment, not how well learners express ideas or spell. Writing elements should only be given points or commented upon in writing assessments.
Type 7: Multiple choice
Multiple choice questions are good for concepts when learners will likely have trouble articulating a reply but know the answer and can recognize it if they see it. Multiple choice is great for all the Interpretive comprehension criteria such as identify purpose, layout, main idea, register, or implied meanings, especially at lower levels.
Tips for Writing Multiple Choice Questions (Clay, 2001; How to Write Good Test Questions 2015; Jacobs, 2004; Piontek, 2008; Writing Effective Questions. n.d.).
- Provide one correct option and two or three good distractors. Don’t include throw-away choices as this focuses learners on test-taking strategies instead of demonstrating comprehension.
- Use common mistakes that students make, such as identifying key information instead of the main idea, as distractors.
- Avoid “all of the above” and “none of the above” as they often do not demonstrate what students know. When included, they are usually the correct choice which learners can guess! (Jacobs, 2004)
- Avoid negatives in the stem! If you MUST use a negative, capitalize or underline it.
- Avoid qualifiers that can make questions ambiguous like usually, always, often, etc..
- Write all the options to be the same length, and use similar phrasal structure in the options to minimize guessing.
- State items so there can be only one interpretation of meaning. The correct option should not depend on how one interprets the question and should not be arguable.
Applying to Assessment
Most LINC teachers have limited time to find appropriate listening and reading texts for their class level and to create assessment tasks. Gaining more familiarity with the types of comprehension items and tasks, how they can be best utilized to reveal ability for different indicators, and tips to make each type most efficient when applied should make the process of creating receptive assessments flow more smoothly. Most importantly, learners should have less trouble demonstrating their ability to meet the CLB standard for the task and level.
References
Clay, B. (2001). Is this a trick question? Topeka, Kansas: Kansas Curriculum Centre. Retrieved from https://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf
Hajer, A. (2012). CLB support kit. Ottawa, ON: Centre for Canadian Language Benchmarks.
Holmes, T., Habke, A, Schmuck, S. (2017). Integrating CLB assessment into your ESL classroom. Ottawa, ON: Centre for Canadian Language Benchmarks. Retrieved from http://iclba.language.ca/
How to Write Good Test Questions. (2015). Help Teaching. Retrieved from http://www.helpteaching.com/about/how_to_write_good_test_questions/
Jacobs, L. (2004). How to write better tests. Bloomington: Indiana University. Retrieved from https://www.uky.edu/Ag/CLD/CETL/files/f09workshop/IU%20How%20to%20write%20better%20tests.pdf
Makosky, L. (2008). The feasibility and way forward for a standardized exit assessment and test for newcomers in LINC training. Ottawa: Citizenship and Immigration Canada.
Pointek, M. (2008). Best practices for designing and grading exams. Ann Arbor: The University of Michigan. http://www.crlt.umich.edu/publinks/CRLT_no24.pdf
Writing effective questions. (n.d.). The Learning Management Corporation. Retrieved from https://web.archive.org/web/20170215030536/http://www.thelearningmanager.com/pubdownloads/writing_effective_questions.pdf
Appendix
Here are some examples of questions checking for ability on indicators for CLB 2,4, 6 and 7 learners using material from the CLB Support Kit.
Level 1-Literal comprehension
CLB Support Kit Listening Exemplar Video Q: CLB 2-Interacting With Others
CLB 2 greeting (listening) – Identify expressions
Q: Valerie shows she is happy. She says ______________________.– fill-in-the-blank
CLB 2 greeting (listening) – Identify common courtesy phrases
Q: Valerie invites Ramona inside. She says _______________________.– fill-in-the-blank
(Note the CLB 2 questions could be posed orally, to remove the interference of reading and writing for listening comprehension questions.)
CLB Support Kit Listening Exemplar K: CLB 6- Interacting With Others
CLB 6 message (listening) – Identify details
Q: Where is Riyad going on the weekend? – short response
Q: What evening is best to get together? – short response
L2-Interpretive comprehension
CLB Support Kit Reading Document G, page 143: CLB 4-Getting Things Done
CLB 4 schedule (reading) – Identify layout
Q: What part of the schedule shows you shuttle stops? – short response
Q: Describe how you find the time to catch the shuttle. Write 3 or 4 steps. – short response
CLB 4 schedule (reading) – Identify purpose
Q: What is the purpose of this schedule? – multiple choice
A. To give information about domestic and international terminals
B. To provide names of hotels downtown where you can stay
C. To give information about times and places to catch a shuttle
D. To show the airport shuttle service is very busy
CLB Support Kit Listening Exemplar K: CLB 6- Interacting With Others
CLB 6 message (listening) – Identify formal and casual style and register
Q: What style is the message? – multiple choice
A. Static – Speech
B. Formal – Businesslike
C. Casual – Friendly
D. Intimate – Close
CLB 6 message (listening) – Identify situation and relationship between speakers
Q: What is the relationship between Jabar and Riyad? – multiple choice
A. Family
B. Friends
C. Co-workers
D. Acquaintances
CLB Support Kit Reading Text H, page 164: CLB 7-Getting Things Done
CLB 7 memo (reading) – Identify implied meaning
Q: The letter says: “Hertzogg Building management has made it clear that there will be no exceptions to these regulations under any circumstances.” What is implied? – multiple choice
A. There will be no reasons accepted for violating a regulation.
B. Management told staff about the regulations before.
C. There will be a consequence if a staff member violates a regulation.
D. Management allowed some exceptions to the regulations before.
CLB 7 memo (reading) – Find and integrate information
Q: Which building are staff required to sign in and sign out of? – short response
Q: When can staff use priority pass keys and authorized security codes to enter Hertzogg Building? – short response
Level 3-Applied comprehension
(Italics in the indicator indicate additions to extend from L2-Interpretive to L3-Applied)
CLB Support Kit Reading Document G, page 143: CLB 4-Getting Things Done
CLB 4 schedule (reading) – Identify information to make a choice
Your flight is leaving at 7 pm from Terminal 2. You need to be at the airport at least 1 hour before the flight. You are staying at Garden Inn. What time will you catch the shuttle? – short response
CLB Support Kit Listening Exemplar K: CLB 6- Interacting With Others
CLB 6 message (listening) – Identify feelings and apply to own situation
How does Riyad feel about missing the hockey game. How do you know? If you got this message from Riyad, how would you reply? – long response
CLB Support Kit Reading Text H, page 164: CLB 7-Getting Things Done
CLB 7 memo (reading) – Identify mood and attitude and give opinion
Q: What is the mood and attitude of this letter? How do you know? Do you think it is an appropriate tone for a letter from a Manager to employees? Why do you think that? – long response
1 This article is based on a workshop at the 2017 TESL Canada Conference.
This informative and practical article has been forwarded by our coordinator to all of the ESL instructors in our organization. The break-down of levels and types of questions is very helpful. Unfortunately for us, in our organization we have been instructed that we are neither able to use T/F nor multiple choice questions and this, as you can imagine, has really increased the amount of time required to both design and mark task assessments. I will, nevertheless, benefit from the tools provided herein.
Very helpful and clear.
Thank you Lisa for this very clearly written and informative article.
A very useful article! In particular I think it effectively highlights what question types do/don’t lend themselves to assessing a given category of skill, which is extremely important. My one caveat is that, although it is surely true that the wording of test items is problematic if it “exceeds the learners’ CLB level”, such advice assumes that teachers have sufficient facility with the CLB to control for items’ language level, which may not always be the case.
Please have a session on Tutela. This is valuable information.