The nature and impact of portfolio-based language assessment (PBLA)

Download PDF

Abstract

The nature and effects of PBLA were investigated. I examined LINC program evaluations, government-solicited assessment reports, PBLA research, and other PBLA-related documents. I discuss the features of PBLA and its reported effects on language outcomes and teacher and student attitudes. I found that the government did not provide a rationale for PBLA and that the results of research did not support the introduction of PBLA. I also found that PBLA is neither standardized nor portfolio-based as claimed. It is costlier, more time-consuming, and appears to have more teacher pushback than the approach it replaced. Regardless, there is no evidence that the LINC program has improved students’ language skills before or after the implementation of PBLA

In 2010, Citizenship and Immigration Canada (CIC) introduced Portfolio-Based Language Assessment (PBLA) to improve the Language Instruction for Newcomers to Canada program (LINC) (CIC, 2010). With more than $200 million (FY 2015/16) in funding, it is the largest language program in Canada and it continues to grow (Immigration, Refugees and Citizenship Canada [IRCC], 2017). LINC is important to students because the achievement of certain LINC benchmarks can improve high-stakes citizenship and employment outcomes (CIC, 2010).

PBLA was a large national intervention intended to deliver a new approach to language instruction and assessment to more than fifty thousand students (CIC, 2010). What was the rationale underpinning PBLA? What characterizes PBLA? What was its impact? To answer these questions, I examined PBLA research, program evaluations, reports, and other PBLA related documents (CIC, 2010; Fox, 2014; Fox & Fraser, 2012; Holmes, 2015; Immigration, Refugees and Citizenship Canada [IRCC], 2017; Pettis, 2014). I will begin by discussing LINC assessment reports and the PBLA pilot study. I will then look at claims that PBLA is standardized, authentic, collaborative, and portfolio-based. I will finish by describing PBLA’s impact on teachers’ attitudes, program costs, and student proficiency.

Concepts of formative, summative, and standardized assessment vary in PBLA literature, so I will define these terms for clarity. Summative and formative assessments have different purposes. Both assess current performance in relation to the outcomes, but formative assessments are ad-hoc, unstandardized, and used to ‘form’ or guide instruction and learning. A summative assessment, however, is planned and often standardized to some extent but has no formative role because it marks the end of an instructional unit. Scores on summative assessments are permanently recorded and represent value to stakeholders after the course is finished. Formative assessment scores are not recorded and have no value once the target outcome has been achieved.

Standardized assessments have been characterized as externally developed, large-scale, and norm-referenced (Henning, 1987; Pettis, 2014). However, the defining feature of a standardized test is its reliability rather than its size or the type of inferences drawn from its scores. A standardized test requires “all test takers to answer the same questions, or a selection of questions from a common bank of questions, in the same way, and … is scored in a ‘standard’ or consistent manner” (Abbot, 2014). A standardized assessment can be large- or small-scale, and inferences drawn from its scores can be either norm- or criterion-referenced.

The Introduction of PBLA

Pre-PBLA

Just prior to introducing PBLA, CIC completed a comprehensive LINC program evaluation, which included the results of a survey and quasi-experimental study (CIC, 2010). In the program evaluation, CIC reported that the existing LINC program was cost-effective and that the teachers, materials, curriculum, and assessments were superior (pp. 14, 41, 42). The results of the survey and quasi-experimental study indicate otherwise. When surveyed, teachers reported that determining levels was confusing, that there were no progress and exit tests, that the program’s objectives were unclear, and that there was no standard curriculum (p. 62). In the quasi-experimental study, CIC (2010) concluded that there were no significant differences in the “four skill[s]” between those who had taken LINC classes and those who had not (pp. 34, 35). The $100 million (FY 2008/09) LINC program in place when PBLA was introduced was neither pedagogically effective nor cost effective as claimed earlier in the same document (CIC, 2010).

Rationale for PBLA

It is unclear why the government chose PBLA. Prior to the PBLA pilot study, external experts hired by the government to evaluate assessment models did not support assessments like PBLA (Makosky, 2008; Nagy & Stewart, 2009). In a literature review of alternative assessments Fox (2008), the same researcher who later conducted the pilot study, reported on the lack of reliability, high costs, and high labour demands that were associated with portfolios. These drawbacks had also been discussed by Nagy and Stewart in their report to CIC on LINC assessment (2009) prior to PBLA. The government-hired experts, including a pilot study researcher, had clearly described inherent problems with PBLA-like assessments (Fox, 2008; Nagy & Stewart, 2009), yet there was no evidence that these particularly relevant issues and documents were ever considered, which suggests that the decision to implement PBLA in 2010 (CIC) was foregone.

Four years after PBLA was introduced, experts (Pettis, 2014; Holmes, 2015) claimed that the decision to implement PBLA was based on the recommendations of other experts Makosky (2008) and Nagy and Stewart (2009). As mentioned above, these three experts did not support the use of portfolios as standardized assessments. Makosky did not even discuss the use of portfolios other than in a parenthetical comment about them being formative and controversial tools used by some school boards. Nagy and Stewart (2009) specifically warned against using portfolios as standardized assessments because of their low reliability, lack of validity, and high costs – problems acknowledged by the pilot study researcher prior to PBLA (Fox, 2008) and evident in the researcher’s subsequent PBLA studies (Fox & Fraser, 2012; Fox, 2014).

The Pilot Study

CIC’s first step in implementing PBLA was to pilot a “standardized portfolio-based assessment” that would “produce reports on student progress and the immediate outcomes of language training” (2010, p. x) to improve achievement and its measurement. The pilot study, however, focused on student and teacher behaviours and perceptions rather than achievement (Fox & Fraser, 2012). More than half of all the teachers surveyed reported that PBLA had no effect or a negative effect on their planning, teaching, and assessment (Fox & Fraser, 2012, p. 11).

Likewise, there was little or no evidence that the implementation of PBLA improved students’ perceptions of how often they reviewed their work (Fox & Fraser, 2012, p. 20), their attendance (p. 23), their own fluency (p. 26), the effectiveness of the class materials (p. 26), and the organization of their work (p. 27). Insufficient rater-reliability, low teacher participation, and teacher complaints about PBLA were also reported. Teacher and student opposition to PBLA is convincing evidence that PBLA was ineffective, unpopular, and time-consuming, yet it was deemed a success by the government, the pilot study researchers, and an expert (CIC, 2013; Fox & Fraser; 2012, Pettis; 2012).

The Nature of PBLA

Formative and Summative Roles

At the time it was introduced, the government and experts established PBLA as a formative, summative, and standardized assessment (CIC, 2010; Fox & Fraser, 2012; Pettis, 2012). Two years later, however, one of the experts, the pilot study researcher, stated that she and others had “got it wrong”, arguing that PBLA was not working as intended because it was used as a summative rather than a formative tool (Fox, 2014, p. 68). Other experts made similar arguments. Citing Black and Wiliam (1998a), Pettis (2012, 2014) and Holmes (2015) claimed that all assessment must be formative, but Holmes also stressed the importance of good summative assessments. Teachers and other PBLA experts stated that summative testing would be unnecessary once PBLA was implemented (Elsa Net, 2015). These claims are inconsistent with the purpose of PBLA and the definitions of formative and summative assessments. Unlike formative assessments, summative assessments are necessary and their scores represent value outside the classroom. Furthermore, Black and Wiliam did not argue that all assessment should be formative, but they did state that summative assessments are required (1998b).

PBLA as a Portfolio-based assessment

Whether used formatively or summatively, PBLA lacks the characteristics of a true portfolio-based assessment. The PBLA “portfolio” is organized and scored according to the outcomes (the CLBs) but scores are not assigned to the portfolio itself. In a true portfolio-based system, the student, not the teacher, selects artefacts for assessment. Ultimately, the PBLA portfolio serves the same purpose as a typical binder — to improve the summative assessment outcome by organizing prescribed course work, including formative assessments, for review and study.

PBLA as systematic, authentic, collaborative, and learner-centred

PBLA has been defined as “…systematic, authentic, and collaborative” and learner-centred (Pettis, 2014, p.7; 2012, p. 18). PBLA is based on standardized outcomes but its assessment tasks are unstandardized because they are individually created or chosen (Pettis, 2014) by each teacher. In these respects, PBLA resembles a typical course with multiple sections and a shared exam. These teacher-chosen assessment tasks are authentic for the teacher but not the student. Furthermore, the involvement of individual teachers for artefact selection and the use of students’ peers for summative assessment tasks (Pettis, 2014) is unsystematic and raises bias and authorship issues (Gearhart & Herman, 1998).

PBLA has been called collaborative and learner-centred (Pettis, 2012, 2014) but compared to students in a true portfolio or traditional system, PBLA students have less control. They have no role in the selection of artefacts for assessment (Ripley, 2012) nor do they want one (Fox, 2014). According to the PBLA guide (Pettis, 2014), teachers must choose artifacts because students choices were “unhelpful” (p.38). PBLA also prescribes much of the content and the use of student binders (Hajer, n.d.). Each binder must contain personal information, all class work for an entire benchmark, formative and summative assessment tasks for an entire benchmark, and 128 pages of reference material.

Much of the reference material, which includes information on the House of Commons, how to buy a house, and the provincial and federal income tax brackets, is widely available online, rarely accessed, and not tested. Therefore, it is not surprising that students and teachers reported that the PBLA binder was unnecessary, cumbersome, and stressful to use and repeatedly carry home and to class (ELSA Net, 2015; Fox, 2014). Students in traditional and true portfolio systems have more control over what goes in their binder or portfolio and how it is organized.

The Effects of PBLA

Effect on Proficiency

The goal of the LINC program is to help students settle in Canada by improving their language skills (IRCC, 2017), but the implementation of PBLA has had a negative overall effect. In the pilot study, students reported that PBLA did not improve their English (Fox & Fraser, 2012). PBLA students also needed 50% more instructional time to complete a benchmark, and they used English outside of the home 20% less frequently than newcomers not in the program (IRCC, 2017; 2018). Most importantly, students who had been in the LINC program, pre- or post-PBLA, had not improved their English any more than newcomers not in the program (CIC, 2010, p. 35; IRCC, 2018, p. 9). Other negative effects associated with PBLA were high teacher pushback, higher labour requirements, and higher costs.

Teacher Pushback

Teacher complaints remain a prominent feature of PBLA seven years after it was introduced in 2010. Teacher pushback was clearly evident in early research (Fox, 2014; Ripley, 2012) and more recently in a petition signed by a group of more than 650 people who are appealing to the minister responsible for the IRCC to abolish PBLA (Lachini, 2018). Based on the petition statement and the comments, it appears many teachers are unhappy with PBLA for similar reasons given by the teachers surveyed six years earlier in the pilot study (Fox, 2012). Teachers have opposed PBLA since its introduction but the current opposition to it is striking because it is widespread and strong.

Labour Requirements

Another important problem, albeit not directly pedagogical, is the number of teachers and instructional hours PBLA requires. The use of portfolios increases labour requirements and costs, which can be significant (Catteral & Winters, 1994; Hargreaves, Earl, & Schmidt; 2002; Nagy & Stewart, 2009), and PBLA is no exception. Language instruction costs, which were highly correlated with the number of teachers (CIC, 2010), had changed very little in the five years prior to PBLA. However, four years after PBLA was introduced, assessment costs rose 220% while the number of students rose only 75% (CIC, 2010; IRCC, 2017). Teacher complaints about the additional time demands and unpaid work under PBLA (Fox, 2014; Lachini, 2018; Ripley, 2012) suggest that the large increase in funding was insufficient and that its labour requirements are still unmet.

Conclusions

First, the decision to implement PBLA was neither evidence-based nor clear. The results of the pilot study and the two government reports on assessment (Fox & Fraser, 2012; Makosky, 2008; Nagy and Stewart, 2009) did not support the implementation of a system like PBLA. Second, conceptions of portfolios and standardized, summative, and formative assessments were inconsistent with their definitions and the purpose of PBLA. Third, PBLA is neither portfolio-based nor standardized. It is an assessment based on shared outcomes, which are standards, but it uses unstandardized tasks. Fourth, teachers and students view PBLA as onerous and ineffective. Fifth, PBLA and the previous approach were equally ineffective in changing language outcomes in Canada’s $200 million (FY 2016) national language program (IRCC, 2017). PBLA, however, was costlier and required many more instructional hours.

References

Abbot, S. (2014). Standardized test. Retrieved May 11, 2018, from The glossary of education reform.: http://edglossary.org/standardized-test/

Black, P. & William, D. (1998a). Inside the black box: Raising standards through classroom assessment. London, UK: King’s College London School of Education.

Black, P. & William, D. (1998b). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 139–144, 146–148. doi:http://dx.doi.org/10.1177/003172171009200119

Catterall, J. & Winters, L. (1994). Economic analysis of testing: Competency, certification, and ‘authentic’ assessments (CSE Report 383). University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST)., Los Angeles. Retrieved from http://cresst.org/wp-content/uploads/TECH383.pdf

Citizenship and Immigration Canada. (2010). Evaluation of the Language Instruction for Newcomers to Canada (LINC) program. Ottawa: Citizenship and Immigration Canada. Retrieved from https://www.canada.ca/content/dam/ircc/migration/ircc/english/resources/evaluation/linc/2010/linc-eval.pdf

Citizenship and Immigration Canada. (2013, March 15). Operational bulletin 510. Ottawa. Retrieved May 11, 2018, from https://www.canada.ca/en/immigration-refugees-citizenship/corporate/publications-manuals/operational-bulletins-manuals/bulletins-2013/510-march-15-2013.html

ELSA Net. (2015). LISTN Learning Event 2015. Language Instruction Support and Training Network.

Fox, J. (2008). Alternative assessment. In E. Shohamy & N. Hornberger (Eds.), Encyclopedia of language and education (2 ed., Vol. 7, pp. 97–109). New York: Springer.

Fox, J. (2014). Portfolio based language assessment (PBLA) in Canadian immigrant language training: Have we got it wrong? Contact, 40(2), 68–83. Retrieved from https://www.teslontario.org/uploads/publications/researchsymposium/ResearchSymposium2014.pdf

Fox, J. & Fraser, W. (2012). Report on the impact of the implementation of PBLA on LINC teachers and students: Ottawa field test, September 2010 to December 2011. Ottawa: Citizenship and Immigration Canada.

Gearhart, M. & Herman, J. (1988). Portfolio assessment: Whose work is it? Educational Assessment, 5(1), 41–55. doi:https://doi.org/10.1207/s15326977ea0501_2

Hajer, A. (n.d.). Language Companion | Stage II: CLB 5–8. Citizenship and Immigration Canada. Retrieved from https://www.7oaks.org/Programs/Settlement/Documents/PBLALanguage%20Companion,%20CLB%205-8.pdf

Hargreaves, A., Earl, L., & Schmidt, M. (2002). Perspectives on Alternative Assessment Reform. American Educational Research Journal, 69–95. doi:https://doi.org/10.3102/00028312039001069

Henning, G. (1987). A guide to language testing: Development, evaluation, research. Heinle & Heinle.

Holmes, T. (2015). PBLA: Moving toward wustainability. TESL Canada Journal, 32(9), 113–123. doi:https://doi.org/10.18806/tesl.v32i0.1220

Immigration, Refugees and Citizenship Canada. (2017). Evaluation of the settlement program. Immigration, Refugees and Citizenship Canada. Retrieved from https://www.canada.ca/content/dam/ircc/documents/pdf/english/evaluation/e2-2016-settlement-en.pdf

Immigration, Refugees, and Citizenship Canada. (2018). IRCC evaluation of the settlement program & upcoming language training evaluation. Edmonton. Retrieved May 11, 2018, from https://www.norquest.ca/NorquestCollege/media/pdf/media-centre/events/2018/WCLTL/Day-1/IRCC_Settlement_Eval_LTLE.pdf

Lachini, K. (2018, March 16). Stop PBLA for continuous intake and academic programs. Retrieved from change.org: https://www.change.org/p/ministry-of-immigration-and-refugees-stop-pbla-for-continuous-intake-and-academic-programs

Makosky, L. (2008). The feasibility and way forward for a standardized exit assessment and test for newcomers in LINC training. Ottawa: Canada, Citizenship and Immigration.

Nagy, P. & Stewart, G. (2009). Research study on potential approaches to second language assessment. Ottawa: Citizenship and Immigration Canada.

Pettis, J. (2012, Spring). Portfolio based language assessment (PBLA). (INSCAN Special Issue). Retrieved from http://www3.carleton.ca/cimss/inscan-e/v24_se.pdf

Pettis, J. (2014). Portfolio-based language assessment (PBLA): Guide for teachers and programs. Ottawa. Retrieved from https://listn.tutela.ca/wp-content/uploads/PBLA_Guide_2014.pdf

Ripley, D. (2012). Implementing portfolio-based language assessment in LINC programs: Benefits and challenges. TESL Canada Journal, 30(1), 69–86. doi:https://doi.org/10.18806/tesl.v30i1.1126

Bio:

Terry Vanderveen is an English instructor and psychometrician. He holds a master’s degree in applied linguistics and has published or presented papers on student attitudes, L2 motivation, self-monitoring, instructional design, and assessment in education.

POST COMMENT 0

Leave a Reply

Your email address will not be published. Required fields are marked *