Testing and reporting methods do not compute

Alberta Education plans to spend millions developing Computer Adaptive Tests (CATs) for students in Grades 1–12 to support its Grade Level of Achievement initiative.

The Alberta government’s request for proposal (RFP) calls on testing corporations to bid on a three-year $3 million dollar contract with Alberta Education to produce CATs. According to the RFP, the contract could be renewed after the development phase for up to a total of six years. Renewal of the contract would create a market with a potential value of an additional $16 million or more for the successful corporation.

In March, just prior to releasing the RFP, Alberta Education announced the Grade Level of Achievement pilot project. The project requires that teachers report the grade level that students achieved in each of the four core subjects on year-end report cards. While announced initially as a pilot over three years beginning in 2005/06, the Association is concerned that, given the massive investment planned for CATs, the department intends to proceed with both initiatives regardless of its findings.

The RFP for the development of CATs, posted on the Internet on May 3, 2005, is open to bidders from the United States, where large testing corporations such as Educational Testing Services develop and sell computer adaptive, standardized ability tests. British Columbia is already piloting a U.S model. A decision on a successful bidder for Alberta is set for June 17, 2005.

What are CATs?

CATs are designed to adapt automatically to the ability of the student taking the test. For example, a student who answers the first multiple-choice item correctly will be provided by the computer with a second question of greater difficulty. If a student provides a wrong answer, however, the next question will be easier. The question presented to a student at any time depends on how the previous question was answered. Once the computer decides that the student has answered sufficient questions, it provides a final or summative score. In essence, no two students would complete the same test or necessarily the same number of questions.

The summative score on the test indicates a student’s relative standing in relation to a limited set of learning outcomes easily measured by a machine. No information is provided to the classroom teacher that can be used to promote further learning for the student. A teacher is not permitted access to specific questions that the student found difficult. The investment in CATs fails to support Education Minister Zwozdesky’s interest in providing teachers with diagnostic assessment materials, which he expressed when he halted further work on Grade 4 achievement tests in April.

Jim Dueck, assistant deputy minister of System Improvement and Reporting Division, told the ATA that more than 18,000 test items will be needed. In the first year of development, 2005/06, the RFP calls for items to be prepared for students in Grades 1–9 language arts and mathematics. In 2006/07, the project will be extended to science and social studies. In the third and final year of development, 2007/08, items are to be provided for Grades 10–12.

Beginning with the 2008/09 school year, all Grades 1–12 students in public schools in Alberta could be writing achievement tests in the four core subjects and the results reported as grade level of achievement on year end report cards, if the project proceeds as expected. It is not clear if this direction will affect home education and private schools students as well.

In defending increased investment in provincially controlled testing, Dueck said the department is implementing the intent of the Alberta Commission on Learning’s Recommendation 59. Instead of updating the Classroom Assessment Materials Project (CAMP) materials as specifically stated in the report, the department is proceeding with CATs. The ATA does not believe that CATs will help to "ensure that all teachers have access to high quality, performance-based and cutting-edge classroom assessment materials and practices," as recommended by the Commission. The Commission clearly stated that "No one is better able to assess the achievement of students than classroom teachers, who work with students on a daily basis." Rather than enhancing the teacher’s professional role as intended, CATs will remove teacher judgment from the assessment process, as that is a major purpose of their design.

Dueck did not say how the bank of items would be renewed—curriculum-based test items typically become stale after two or three years. While costs for maintenance of the system are worked into the per student fee of $5 plus inflation, development costs for keeping the bank of items current and relevant to the curriculum do not appear in the RFP.

ATA President Frank Bruseker was surprised by the government’s move. "I am astonished that the government would proceed with this RFP and Grade Level of Achievement reporting without consultation and careful consideration of the consequences. The department’s massive infusion of funds into more provincial testing coupled with new grade level reporting requirements will only undermine teachers’ professional role in evaluating student progress. This direction is contrary to everything we know about ensuring good assessment practices and building capacity in the system for improved learning."

Teachers opposed to CATs

At the Association’s Annual Representative Assembly (ARA) meeting, held May 21–23, delegates expressed strong opposition to the department’s work in these areas. Following discussion of the implications of the Grade Level of Achievement initiative, the lack of consultation with the teaching profession, and the need for further review by education partners, delegates unanimously passed a resolution urging the Department of Education to declare a moratorium on implementing these new reporting requirements. The Assembly discussed the inappropriateness of CATs for reporting grade level of achievement, the costs to provincial and local budgets, and potential misuse of yet another provincial testing initiative. Delegates overwhelmingly voted in favour of urging the department to suspend immediately all activities on the CAT project. ATA President Bruseker will follow up ARA’s decisions with Minister Zwozdesky.