Abstract
The nature and effects of PBLA were investigated. I examined LINC program evaluations, government-solicited assessment reports, PBLA research, and other PBLA-related documents. I discuss the features of PBLA and its reported effects on language outcomes and teacher and student attitudes. I found that the government did not provide a rationale for PBLA and that the results of research did not support the introduction of PBLA. I also found that PBLA is neither standardized nor portfolio-based as claimed. It is costlier, more time-consuming, and appears to have more teacher pushback than the approach it replaced. Regardless, there is no evidence that the LINC program has improved students’ language skills before or after the implementation of PBLA
In 2010, Citizenship and Immigration Canada (CIC) introduced Portfolio-Based Language Assessment (PBLA) to improve the Language Instruction for Newcomers to Canada program (LINC) (CIC, 2010). With more than $200 million (FY 2015/16) in funding, it is the largest language program in Canada and it continues to grow (Immigration, Refugees and Citizenship Canada [IRCC], 2017). LINC is important to students because the achievement of certain LINC benchmarks can improve high-stakes citizenship and employment outcomes (CIC, 2010).
PBLA was a large national intervention intended to deliver a new approach to language instruction and assessment to more than fifty thousand students (CIC, 2010). What was the rationale underpinning PBLA? What characterizes PBLA? What was its impact? To answer these questions, I examined PBLA research, program evaluations, reports, and other PBLA related documents (CIC, 2010; Fox, 2014; Fox & Fraser, 2012; Holmes, 2015; Immigration, Refugees and Citizenship Canada [IRCC], 2017; Pettis, 2014). I will begin by discussing LINC assessment reports and the PBLA pilot study. I will then look at claims that PBLA is standardized, authentic, collaborative, and portfolio-based. I will finish by describing PBLA’s impact on teachers’ attitudes, program costs, and student proficiency.
Concepts of formative, summative, and standardized assessment vary in PBLA literature, so I will define these terms for clarity. Summative and formative assessments have different purposes. Both assess current performance in relation to the outcomes, but formative assessments are ad-hoc, unstandardized, and used to ‘form’ or guide instruction and learning. A summative assessment, however, is planned and often standardized to some extent but has no formative role because it marks the end of an instructional unit. Scores on summative assessments are permanently recorded and represent value to stakeholders after the course is finished. Formative assessment scores are not recorded and have no value once the target outcome has been achieved.
Standardized assessments have been characterized as externally developed, large-scale, and norm-referenced (Henning, 1987; Pettis, 2014). However, the defining feature of a standardized test is its reliability rather than its size or the type of inferences drawn from its scores. A standardized test requires “all test takers to answer the same questions, or a selection of questions from a common bank of questions, in the same way, and … is scored in a ‘standard’ or consistent manner” (Abbot, 2014). A standardized assessment can be large- or small-scale, and inferences drawn from its scores can be either norm- or criterion-referenced.
The Introduction of PBLA
Pre-PBLA
Just prior to introducing PBLA, CIC completed a comprehensive LINC program evaluation, which included the results of a survey and quasi-experimental study (CIC, 2010). In the program evaluation, CIC reported that the existing LINC program was cost-effective and that the teachers, materials, curriculum, and assessments were superior (pp. 14, 41, 42). The results of the survey and quasi-experimental study indicate otherwise. When surveyed, teachers reported that determining levels was confusing, that there were no progress and exit tests, that the program’s objectives were unclear, and that there was no standard curriculum (p. 62). In the quasi-experimental study, CIC (2010) concluded that there were no significant differences in the “four skill[s]” between those who had taken LINC classes and those who had not (pp. 34, 35). The $100 million (FY 2008/09) LINC program in place when PBLA was introduced was neither pedagogically effective nor cost effective as claimed earlier in the same document (CIC, 2010).
Rationale for PBLA
It is unclear why the government chose PBLA. Prior to the PBLA pilot study, external experts hired by the government to evaluate assessment models did not support assessments like PBLA (Makosky, 2008; Nagy & Stewart, 2009). In a literature review of alternative assessments Fox (2008), the same researcher who later conducted the pilot study, reported on the lack of reliability, high costs, and high labour demands that were associated with portfolios. These drawbacks had also been discussed by Nagy and Stewart in their report to CIC on LINC assessment (2009) prior to PBLA. The government-hired experts, including a pilot study researcher, had clearly described inherent problems with PBLA-like assessments (Fox, 2008; Nagy & Stewart, 2009), yet there was no evidence that these particularly relevant issues and documents were ever considered, which suggests that the decision to implement PBLA in 2010 (CIC) was foregone.
Four years after PBLA was introduced, experts (Pettis, 2014; Holmes, 2015) claimed that the decision to implement PBLA was based on the recommendations of other experts Makosky (2008) and Nagy and Stewart (2009). As mentioned above, these three experts did not support the use of portfolios as standardized assessments. Makosky did not even discuss the use of portfolios other than in a parenthetical comment about them being formative and controversial tools used by some school boards. Nagy and Stewart (2009) specifically warned against using portfolios as standardized assessments because of their low reliability, lack of validity, and high costs – problems acknowledged by the pilot study researcher prior to PBLA (Fox, 2008) and evident in the researcher’s subsequent PBLA studies (Fox & Fraser, 2012; Fox, 2014).
The Pilot Study
CIC’s first step in implementing PBLA was to pilot a “standardized portfolio-based assessment” that would “produce reports on student progress and the immediate outcomes of language training” (2010, p. x) to improve achievement and its measurement. The pilot study, however, focused on student and teacher behaviours and perceptions rather than achievement (Fox & Fraser, 2012). More than half of all the teachers surveyed reported that PBLA had no effect or a negative effect on their planning, teaching, and assessment (Fox & Fraser, 2012, p. 11).
Likewise, there was little or no evidence that the implementation of PBLA improved students’ perceptions of how often they reviewed their work (Fox & Fraser, 2012, p. 20), their attendance (p. 23), their own fluency (p. 26), the effectiveness of the class materials (p. 26), and the organization of their work (p. 27). Insufficient rater-reliability, low teacher participation, and teacher complaints about PBLA were also reported. Teacher and student opposition to PBLA is convincing evidence that PBLA was ineffective, unpopular, and time-consuming, yet it was deemed a success by the government, the pilot study researchers, and an expert (CIC, 2013; Fox & Fraser; 2012, Pettis; 2012).
The Nature of PBLA
Formative and Summative Roles
At the time it was introduced, the government and experts established PBLA as a formative, summative, and standardized assessment (CIC, 2010; Fox & Fraser, 2012; Pettis, 2012). Two years later, however, one of the experts, the pilot study researcher, stated that she and others had “got it wrong”, arguing that PBLA was not working as intended because it was used as a summative rather than a formative tool (Fox, 2014, p. 68). Other experts made similar arguments. Citing Black and Wiliam (1998a), Pettis (2012, 2014) and Holmes (2015) claimed that all assessment must be formative, but Holmes also stressed the importance of good summative assessments. Teachers and other PBLA experts stated that summative testing would be unnecessary once PBLA was implemented (Elsa Net, 2015). These claims are inconsistent with the purpose of PBLA and the definitions of formative and summative assessments. Unlike formative assessments, summative assessments are necessary and their scores represent value outside the classroom. Furthermore, Black and Wiliam did not argue that all assessment should be formative, but they did state that summative assessments are required (1998b).
PBLA as a Portfolio-based assessment
Whether used formatively or summatively, PBLA lacks the characteristics of a true portfolio-based assessment. The PBLA “portfolio” is organized and scored according to the outcomes (the CLBs) but scores are not assigned to the portfolio itself. In a true portfolio-based system, the student, not the teacher, selects artefacts for assessment. Ultimately, the PBLA portfolio serves the same purpose as a typical binder — to improve the summative assessment outcome by organizing prescribed course work, including formative assessments, for review and study.
PBLA as systematic, authentic, collaborative, and learner-centred
PBLA has been defined as “…systematic, authentic, and collaborative” and learner-centred (Pettis, 2014, p.7; 2012, p. 18). PBLA is based on standardized outcomes but its assessment tasks are unstandardized because they are individually created or chosen (Pettis, 2014) by each teacher. In these respects, PBLA resembles a typical course with multiple sections and a shared exam. These teacher-chosen assessment tasks are authentic for the teacher but not the student. Furthermore, the involvement of individual teachers for artefact selection and the use of students’ peers for summative assessment tasks (Pettis, 2014) is unsystematic and raises bias and authorship issues (Gearhart & Herman, 1998).
PBLA has been called collaborative and learner-centred (Pettis, 2012, 2014) but compared to students in a true portfolio or traditional system, PBLA students have less control. They have no role in the selection of artefacts for assessment (Ripley, 2012) nor do they want one (Fox, 2014). According to the PBLA guide (Pettis, 2014), teachers must choose artifacts because students choices were “unhelpful” (p.38). PBLA also prescribes much of the content and the use of student binders (Hajer, n.d.). Each binder must contain personal information, all class work for an entire benchmark, formative and summative assessment tasks for an entire benchmark, and 128 pages of reference material.
Much of the reference material, which includes information on the House of Commons, how to buy a house, and the provincial and federal income tax brackets, is widely available online, rarely accessed, and not tested. Therefore, it is not surprising that students and teachers reported that the PBLA binder was unnecessary, cumbersome, and stressful to use and repeatedly carry home and to class (ELSA Net, 2015; Fox, 2014). Students in traditional and true portfolio systems have more control over what goes in their binder or portfolio and how it is organized.
The Effects of PBLA
Effect on Proficiency
The goal of the LINC program is to help students settle in Canada by improving their language skills (IRCC, 2017), but the implementation of PBLA has had a negative overall effect. In the pilot study, students reported that PBLA did not improve their English (Fox & Fraser, 2012). PBLA students also needed 50% more instructional time to complete a benchmark, and they used English outside of the home 20% less frequently than newcomers not in the program (IRCC, 2017; 2018). Most importantly, students who had been in the LINC program, pre- or post-PBLA, had not improved their English any more than newcomers not in the program (CIC, 2010, p. 35; IRCC, 2018, p. 9). Other negative effects associated with PBLA were high teacher pushback, higher labour requirements, and higher costs.
Teacher Pushback
Teacher complaints remain a prominent feature of PBLA seven years after it was introduced in 2010. Teacher pushback was clearly evident in early research (Fox, 2014; Ripley, 2012) and more recently in a petition signed by a group of more than 650 people who are appealing to the minister responsible for the IRCC to abolish PBLA (Lachini, 2018). Based on the petition statement and the comments, it appears many teachers are unhappy with PBLA for similar reasons given by the teachers surveyed six years earlier in the pilot study (Fox, 2012). Teachers have opposed PBLA since its introduction but the current opposition to it is striking because it is widespread and strong.
Labour Requirements
Another important problem, albeit not directly pedagogical, is the number of teachers and instructional hours PBLA requires. The use of portfolios increases labour requirements and costs, which can be significant (Catteral & Winters, 1994; Hargreaves, Earl, & Schmidt; 2002; Nagy & Stewart, 2009), and PBLA is no exception. Language instruction costs, which were highly correlated with the number of teachers (CIC, 2010), had changed very little in the five years prior to PBLA. However, four years after PBLA was introduced, assessment costs rose 220% while the number of students rose only 75% (CIC, 2010; IRCC, 2017). Teacher complaints about the additional time demands and unpaid work under PBLA (Fox, 2014; Lachini, 2018; Ripley, 2012) suggest that the large increase in funding was insufficient and that its labour requirements are still unmet.
Conclusions
First, the decision to implement PBLA was neither evidence-based nor clear. The results of the pilot study and the two government reports on assessment (Fox & Fraser, 2012; Makosky, 2008; Nagy and Stewart, 2009) did not support the implementation of a system like PBLA. Second, conceptions of portfolios and standardized, summative, and formative assessments were inconsistent with their definitions and the purpose of PBLA. Third, PBLA is neither portfolio-based nor standardized. It is an assessment based on shared outcomes, which are standards, but it uses unstandardized tasks. Fourth, teachers and students view PBLA as onerous and ineffective. Fifth, PBLA and the previous approach were equally ineffective in changing language outcomes in Canada’s $200 million (FY 2016) national language program (IRCC, 2017). PBLA, however, was costlier and required many more instructional hours.
References
Abbot, S. (2014). Standardized test. Retrieved May 11, 2018, from The glossary of education reform.: http://edglossary.org/standardized-test/
Black, P. & William, D. (1998a). Inside the black box: Raising standards through classroom assessment. London, UK: King’s College London School of Education.
Black, P. & William, D. (1998b). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 139–144, 146–148. doi:http://dx.doi.org/10.1177/003172171009200119
Catterall, J. & Winters, L. (1994). Economic analysis of testing: Competency, certification, and ‘authentic’ assessments (CSE Report 383). University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST)., Los Angeles. Retrieved from http://cresst.org/wp-content/uploads/TECH383.pdf
Citizenship and Immigration Canada. (2010). Evaluation of the Language Instruction for Newcomers to Canada (LINC) program. Ottawa: Citizenship and Immigration Canada. Retrieved from https://www.canada.ca/content/dam/ircc/migration/ircc/english/resources/evaluation/linc/2010/linc-eval.pdf
Citizenship and Immigration Canada. (2013, March 15). Operational bulletin 510. Ottawa. Retrieved May 11, 2018, from https://www.canada.ca/en/immigration-refugees-citizenship/corporate/publications-manuals/operational-bulletins-manuals/bulletins-2013/510-march-15-2013.html
ELSA Net. (2015). LISTN Learning Event 2015. Language Instruction Support and Training Network.
Fox, J. (2008). Alternative assessment. In E. Shohamy & N. Hornberger (Eds.), Encyclopedia of language and education (2 ed., Vol. 7, pp. 97–109). New York: Springer.
Fox, J. (2014). Portfolio based language assessment (PBLA) in Canadian immigrant language training: Have we got it wrong? Contact, 40(2), 68–83. Retrieved from https://www.teslontario.org/uploads/publications/researchsymposium/ResearchSymposium2014.pdf
Fox, J. & Fraser, W. (2012). Report on the impact of the implementation of PBLA on LINC teachers and students: Ottawa field test, September 2010 to December 2011. Ottawa: Citizenship and Immigration Canada.
Gearhart, M. & Herman, J. (1988). Portfolio assessment: Whose work is it? Educational Assessment, 5(1), 41–55. doi:https://doi.org/10.1207/s15326977ea0501_2
Hajer, A. (n.d.). Language Companion | Stage II: CLB 5–8. Citizenship and Immigration Canada. Retrieved from https://www.7oaks.org/Programs/Settlement/Documents/PBLALanguage%20Companion,%20CLB%205-8.pdf
Hargreaves, A., Earl, L., & Schmidt, M. (2002). Perspectives on Alternative Assessment Reform. American Educational Research Journal, 69–95. doi:https://doi.org/10.3102/00028312039001069
Henning, G. (1987). A guide to language testing: Development, evaluation, research. Heinle & Heinle.
Holmes, T. (2015). PBLA: Moving toward wustainability. TESL Canada Journal, 32(9), 113–123. doi:https://doi.org/10.18806/tesl.v32i0.1220
Immigration, Refugees and Citizenship Canada. (2017). Evaluation of the settlement program. Immigration, Refugees and Citizenship Canada. Retrieved from https://www.canada.ca/content/dam/ircc/documents/pdf/english/evaluation/e2-2016-settlement-en.pdf
Immigration, Refugees, and Citizenship Canada. (2018). IRCC evaluation of the settlement program & upcoming language training evaluation. Edmonton. Retrieved May 11, 2018, from https://www.norquest.ca/NorquestCollege/media/pdf/media-centre/events/2018/WCLTL/Day-1/IRCC_Settlement_Eval_LTLE.pdf
Lachini, K. (2018, March 16). Stop PBLA for continuous intake and academic programs. Retrieved from change.org: https://www.change.org/p/ministry-of-immigration-and-refugees-stop-pbla-for-continuous-intake-and-academic-programs
Makosky, L. (2008). The feasibility and way forward for a standardized exit assessment and test for newcomers in LINC training. Ottawa: Canada, Citizenship and Immigration.
Nagy, P. & Stewart, G. (2009). Research study on potential approaches to second language assessment. Ottawa: Citizenship and Immigration Canada.
Pettis, J. (2012, Spring). Portfolio based language assessment (PBLA). (INSCAN Special Issue). Retrieved from http://www3.carleton.ca/cimss/inscan-e/v24_se.pdf
Pettis, J. (2014). Portfolio-based language assessment (PBLA): Guide for teachers and programs. Ottawa. Retrieved from https://listn.tutela.ca/wp-content/uploads/PBLA_Guide_2014.pdf
Ripley, D. (2012). Implementing portfolio-based language assessment in LINC programs: Benefits and challenges. TESL Canada Journal, 30(1), 69–86. doi:https://doi.org/10.18806/tesl.v30i1.1126
Bio:
Terry Vanderveen is an English instructor and psychometrician. He holds a master’s degree in applied linguistics and has published or presented papers on student attitudes, L2 motivation, self-monitoring, instructional design, and assessment in education.
This article is a breath of fresh air! Congratulations for finally talking about this “hellish experiment” in your magazine. I agree with many of the points made in the article. Among my colleagues, we feel that PBLA is great in theory but not in practice. Some have even gone further to suggest it is nothing more than a make work project. It would have been better to take the existing LINC binders, revised and updated that than torment many of us with PBLA. I hope many more in our industry continue to raise this as not just a bad idea but taxing on teachers’ prep time which for many is not even paid. I also hope PBLA is on the way out!
I agree completely!
Great idea about the LINC Classroom Activities. About time!
This is really an excellent assessment of the PBLA issue…it appeared to be a make-work project for a small group of people who managed to convince the government that it was worthwhile. With funding cuts to the LINC program and its disappearance in some places, I have to wonder why PBLA money is not being redirected to LINC and the far more useful IT2Teach computer moodle.
Thank you for this insightful article. It explains a lot of the conundrum Petis and others who championed PBLA have placed us in.
As a LINC 6 and 7 instructor (BB 7, 8; 7, 8; 6, 7,8 and 6,7,8) I was challenged to deliver a program that moved learners towards their goals, but after 5-6 years I developed a core program with enough flexibility to move within the 3 LINC curricula at these levels according to the learners’ preferences. (I appreciated the LINC activities and standardized assessment tests when they came along., and agree money spent on PBLA would have been better used expanding them.) I did enable learners to go on to enter and succeed in HS credit, HS work- focused programs (Chef, PSW, Hairstyling training), Bridging programs (e.g. U of Waterloo – Optometry) and college upgrading or College programs , as well as to immediate employment. Under the constraints of PBLA, I can’t help them to these goals any more.
Why? 1. There isn’t enough time to learn the complexities of English at these levels with such an emphasis placed on producing artefacts.
2. Learners coming in have less ability each year. They do not have enough control of syntax and grammar to meet the level. In the past, control of core language structures was tied to a teaching level. Not so, under PBLA. Learners can ‘complete’ a task without adequate control of either. Also, learners have a narrower range of vocabulary. In the past, learners in my class read widely; now I rarely open the filing cabinets filled with so many great materials. I expect this is true at lower levels, too.
3. Learners have changed. Learners seem more focused on getting 8 tasks in a binder to progress than on learning. And they want to move quickly. Yet, fewer learners study FT , and fewer do anything outside of the classroom to learn. It seems PBLA has given them the idea they need a collection of papers to succeed VS what is in their brains and what they can do with it on an on- going basis.
is a heart-felt loss for me that I can no longer prepare students for their next steps. There are too many gaps to breach and not enough time to teach, yes, actually teach English.
Within a short period, this has been the second scientific proof of pointlessness, valuelessness, inefficacy, and finally — practical failure of this absurdist assessment system that brings about nothing but damage to the teaching-learning process. (The first one is a UoT doctoral student Yulia Desyatova’s study — see her webinar of October 24 at TESL Ontario). Little choice for the provincial and federal governments but to be less dismissive about this tool and to stop not only its implementation but also inculcation of its non-existent benefit for second language education.
“The great tragedy of science…the slaying of a beautiful hypothesis by ugly fact.” Thomas Henry Huxley
Based on both the research and overwhelming anecdotal information from the field, it appears that it’s time to start a new conversation. What are the best practices to support our adult learners in their English language education in Canada? How can we get this information and implement it in a practical manner given prep time and resource constraints?
Since the PBLA was introduced in 2010, teaching has become a facilitating task instead of a fun filled teaching lesson. Teachers are not sharing their knowledge with their learners through fun activities or field trips anymore, instead teach them for the test. The fun of teaching a LINC classroom disappeared the moment the binders were handed over to learners and the one-size-fits-all progress reports were imposed on the already underpaid and overworked teachers. For the learners, registering into the language program, the PBLA was the new way of demanding that teachers “moved them up”. The implementation of the PBLA across the board made the teaching in the LINC programs across Canada a costly business with questionable results. That is the reason I did not want to be a LINC teacher anymore and left my students.
What does it take to get someone to listen to us and not to the commission driven old timers who got rich “advising” and “training” the teachers.
Congratulations to Terry on such a well documented article. Hopefully others will read this and do something.
Maria D
Hear, hear, this article succinctly and clearly explains the basic flaws of PBLA. A lot of free labor has been invested by countless numbers of teachers across the country to try to make this a workable program, but outcomes remain mediocre at best. As others mentioned, there is a tendency to ‘teach to the test’ despite all the claims by PBLA true believers insisting it is not about ‘tests’ only ‘assessments’. The students are not fooled by this. They know a test when they see one and they react the same to an assessment as with a test. That is, often with fear and resignation. Unfortunately, now in 2023, and a couple of years of online PBLA (which is even worse than in the classroom for effectiveness), it seems very clear that this disastrous system is locked in place. The hacks that created this system are probably still receiving some kind of royalties or compensation for creating this monstrosity. As we see, our governments do not care about the effectiveness nor the costs of most programs, so PBLA will likely be entrenched for a long time to come, or at least until the next ridiculous fad comes along to replace it. I feel sorry for the students (and the teachers) who have to suffer under this poorly constructed and unreliable and invalid system that is no fun for the students nor the teachers. Worst of all, outcomes for the students is poor. It is money certainly wasted.