Minister responds to "The Gathering Storm"

Gene Zwozdesky"In follow-up to the publication of ‘The Gathering Storm’ in the October 25, 2005, issue of the ATA News regarding GLA and CAA, I find it necessary to clarify some details surrounding these two initiatives. Here are some facts and related information, and an update in this regard, which I hope all readers will find helpful."

 
Gene Zwozdesky, Minister of Education


What is "Grade Level of Achievement" (GLA) Reporting?

Definition of GLA

GLA is not a test. It is simply a way of reporting student progress, primarily as information for parents; but, in the aggregate, also for schools, school jurisdictions, and Alberta Education. However, other than the one-school-per-jurisdiction requirement for June 2006 schools will not be asked to report GLA to parents until the 2006-07 school year. GLA is targeted for Grades
1–9 only.

How Will GLA Work?

The teacher will assess each student’s classroom performance based on a variety of classroom assessments over the school year in Language Arts (English LA and French LA) and Math, and will assign a "grade level of achievement" for each student in these two subjects. (GLA reporting for Social Studies and Science is scheduled for June, 2008.) In other words, GLA is the teacher’s judgement of each student’s grade level of curriculum achievement in a particular subject. GLA will be reported to Alberta Education as a whole number.

What is the History of GLA?

Research and development of GLA as a means for clearly reporting student achievement to parents has been occurring for over a decade in Alberta. Piloting of GLA as an aggregated data source began with phase one in 2003/04 with six school boards, all of whom continue to advise Alberta Education on reasonable requirements and timelines. (A seventh Board joined the pilot in Spring, 2004.) The Alberta Assessment Consortium, and representatives from the ASBA "Students are our Bottom-Line" project, also participated.

Among other things, phase one of the GLA pilot demonstrated that teachers increased their classroom assessment skills through dialogue with colleagues. Work is now underway to improve pre-service training in classroom assessment; and Alberta Education is developing a Guide to GLA Reporting to assist teachers and Principals in implementing this initiative which will be available for professional development sessions planned for the implementation phase. Special reporting options will be provided regarding students with special needs who are not studying a graded curriculum.

Phase two of the pilot is scheduled for June 2006 at which time all Alberta school boards, including charter schools, are expected to report GLA in LA and Math only and from one school only; however, school boards may report more than one school if they wish.

The evolution of the GLA initiative has included: a development Task Force with representatives from four school boards; an extensive review by an Alberta Education Inter-divisional Team; input from the Review Committee on Outcomes, which included extensive stakeholder participation; an implementation Task Force with representatives from six and then seven school boards; an evaluation of the pilot via the final report Beyond Management Information Reporting Schedules (MIRS) Pilot Project Assessment; and an analysis of the validity and reliability of GLA data as part of the Beyond MIRS Data Technical Report.

In 2003, the Review Committee on Outcomes, which included ATA representation, endorsed a student achievement indicator that measured improvement over time of students performing below, at, or above grade level.

Why is GLA Important?

The GLA initiative addresses recommendation 60 in the Alberta Commission on Learning report regarding a more comprehensive process for measuring student achievement, especially for students in special programs, and the overall state of our education system. The Alberta Home and School Councils Association endorses GLA as an important communication piece for parents. Some jurisdictions, such as Edmonton Public Schools for example, have been collecting GLA data for many years. In fact, reporting to parents the grade level at which their child is performing has been an expectation since 1996 in accordance with the Guide to Education manual; however, it has not been implemented by all school jurisdictions. GLA Reporting will reinforce that expectation.

GLA reporting also fulfills an expectation of Alberta’s Auditor General to have more comprehensive achievement information, supplemental to provincial achievement tests. The aforementioned reports are available at http://www.education.gov.ab.ca/ipr.

How Will GLA Data Be Used?

GLA data will not be a required measure within the accountability pillar; however, school jurisdictions may use GLA data, if they wish, as a local measure in their own three-year Education Plan.

GLA data will simply become a small additional component of a data set that schools already submit to Alberta Education for inclusion in the Student Information System. Alberta Education will use aggregated GLA data for program evaluation purposes. GLA data will also inform school boards how well students are achieving at the jurisdiction level, which will help inform local-level program decisions. Providing analyzed GLA reports to school-based staff is also under consideration.

In the end, GLA reports will help to ensure that students have access to consistent curricular standards throughout Alberta, and that parents will have comprehensive, consistent and transparent information regarding the outcomes achieved by their children in school.

How Can I Find Out More About GLA?

To view GLA "Project Documents" and the GLA "Reporting Form", or for more information about GLA in general, please visit http://www.education.gov.ab.ca/ipr/gla/.

What is "Computer Adaptive Assessment" (CAA)?

Definition of CAA

CAA is an optional, computerized assessment tool — a voluntary test, so-to-speak — that immediately ‘adapts’ or tailors the difficulty of each test question to the individual student. It is another form of student assessment for optional use by the teacher in the classroom. CAA will be closely aligned with the Alberta Program of Studies and will offer questions that are developed and reviewed by experienced Alberta teachers.

Who is Developing CAA and at What Cost?

CAA is being developed by Alberta Education, by Alberta teachers, and by an Alberta-based company (Castle Rock Research) at less than $1 million per year over the next three years — not $12 million as stated in the ATA News. Access to the CAA system in the first two years will be offered at no cost to school authorities. In the third year, school authorities may incur a portion of the cost, which would be shared with Alberta Education. However, CAA will also allow school jurisdictions to save money that is currently being spent on commercially produced, norm-referenced tests because CAA will provide more meaningful, criterion-referenced instruments based on Alberta’s curricula.

How Will CAA Work?

Sitting at a computer, a student begins the test by answering a question of average difficulty. If the question is answered correctly, the next question will be of higher difficulty; and if the question is answered incorrectly, the next question will be of lower difficulty.

This same process goes on for several questions and until a consistent level of the student’s performance is automatically determined, at which point a message appears on the screen telling the student that the assessment has been completed. Built-in "stopping rules" will be established for each grade level to ensure that no student will have to answer more than the required number of items.

For example, a student may be required to answer not less than 15 items and not more than 55 items, meaning the duration of a CAA assessment will vary for any given student. It is important to note that CAA is being specifically designed to not create test anxiety, test boredom or test fatigue.

CAA has been shown to involve less student time than some conventional assessment tools.

How Will Students with Special Needs Use CAA?

Testing accommodations, such as offering more time to take the test, larger font sizes for students with visual impairments, and sign language interpreters, for example, may be provided at the discretion of the teacher and/or in co-operation with the school authority; and a "human reader" may not be necessary for CAA, since tools exist that can read digital texts aloud for any particular student. Similarly, work will be done by the researchers, Alberta Education, teachers, and other stakeholders to ensure that the final design of CAA will be able to accommodate as many students with special needs as possible.

Why is CAA Being Introduced?

CAA will provide teachers with on-demand, criterion-referenced testing that can be added to their repertoire of student assessment methods, if they so wish. CAA results can provide the teacher with an immediate diagnosis of a student’s strengths and weaknesses in a certain curricular area, which can help the teacher to develop a more precise education plan to benefit that particular student. The CAA initiative supports the infusion of technology into the classroom for those who choose to use it and there is no requirement to report CAA results provincially.

The CAA initiative also addresses Alberta Commission on Learning recommendations 53 and 54 — "Encourage school improvement, research and innovation," and in particular 59 — "Support ongoing classroom assessment"; and it also supports recommendations 42-45 and 49 (with respect to helping address every child’s special talents), as well as 72 and 73 by helping to expand professional development.

In fact, the contracted company has committed to developing CAA in a manner that will give teachers the most helpful information possible in assessing student achievement. The steering committee guiding the development of this tool is also working on enhancing its diagnostic capacity.

When Will CAA Be Available?

The CAA initiative will be piloted by approximately 40 jurisdictions over the next three years. After it has been piloted, assessed, reviewed and refined, it will be available as a supplementary classroom assessment method.

How Can I Learn More About CAA?

Please visit http://www.castlerockresearch.com/caa/Contact.aspx for more facts and information about CAA.

Minister offers assurances on GLA and CAA, but serious questions remain

Frank Bruseker"The Alberta Teachers’ Association welcomes the Minister’s initiative in submitting the article that appears on page 4. The ATA has consistently demonstrated its willingness to foster communication between the profession and decision-makers. We expect the Department of Education to listen to what we have to say and we, in turn, are willing to listen ourselves.

"Education Minister Gene Zwozdesky has responded to the Association’s critique of the department’s Grade Level of Achievement (GLA) reporting and Computer Adaptive Assessment (CAA) initiatives. The

ATA News provided the Minister with an opportunity to present the government’s case directly to teachers after the Minister took issue with ‘The Gathering Storm,’ published in the October 25, 2005, ATA News. ‘The Gathering Storm’ outlined the Association’s concerns about GLAs and CAAs. ‘The Gathering Storm’ is available online (www.teachers.ab.ca)."

 

Frank Bruseker, President of the Alberta Teachers’ Association


What is "Grade Level of Achievement" (GLA) Reporting?

As the Minister’s article points out, GLA reporting relies entirely upon the individual teacher to make professional judgements about the progress and standing of individual students. Furthermore, in assessing a student’s Grade Level of Achievement (GLA) in a particular subject and grade, the teacher is expected to consider how the student is performing in all aspects of the program of studies over the entire year. In these respects, GLA reporting is preferable to relying upon standardized tests, which provide only a snapshot of student performance on a limited range of objectives that lend themselves to paper and pencil testing. Clearly this Minister regards GLA as a way of meeting expectations for enhanced accountability while still respecting the work, knowledge and professional acumen of teachers.

The Minister has also provided assurances that GLA will not be used to assess individual school board performance within the Accountability Pillar, although the information will be aggregated and analysed by the department. Instead, individual jurisdictions will determine how the data is used to inform local results reporting.

Why Should Teachers Be Concerned?

The dilemma facing teachers who will be required to report GLA is that the Alberta Program of Studies is not structured as a simple linear progression of skills and knowledge. In fact, much of the strength of the Alberta curriculum derives from its sophisticated design, breadth of scope, and capacity to accommodate diverse approaches to teaching and learning.

From a practical perspective, the complexity inherent in the curriculum makes it quite difficult for a conscientious teacher to certify that a student is performing at a specific grade level in a subject when that student has not actually received instruction in or completed the objectives associated with the program for that grade. Furthermore, there is a natural and acceptable range of variation in the level of performance of the students assigned to grades and some overlap in student performance between grades. Assigning a single digit to describe a student’s level of achievement relative to the objectives set out for a grade tends to gloss over this reality.

One way of making GLA more acceptable would be to avoid forcing teachers to make speculative judgements about how a student might perform at other grade levels and instead limit reporting to whether the student is performing below, at, or above the grade in which they are receiving instruction. As the Minister points out, the ATA participated in the Review Committee on Outcomes that recommended this alternative approach to GLA reporting.

Finally, it is unclear exactly what additional insights might be gained by the province collecting and aggregating GLA data. The department’s own analysis of Provincial Achievement Test Data supports the commonsense notion that, at the provincial level of analysis, the vast majority of students are achieving at their assigned grade level. Of the relatively few students who are markedly above or below grade level, most would be clearly identified and have individual program plans that provide much more useful information to teachers and parents than a GLA score. GLA reporting will do little for the students most in need of additional support.

Nor is GLA reporting necessary to enhance accountability. The Auditor General’s recommendation referenced in the article dates back to 1995–96 and simply directs the department to "require school boards to assess and report on student achievement of those provincial learning expectations that are not assessed by provincially administered examinations." This decade-old recommendation is generally phrased and, given that it has not been reiterated in more recent Auditor General’s reports, was presumably addressed by the Department to the satisfaction of the Auditor General. In any event, the recommendation provides scant justification for the introduction of this model of GLA reporting.

As for parents, they receive much more detailed and nuanced reports of their children’s progress through the regular school reporting process and through direct communication with teachers. The province already requires teachers to report students’ grade level of achievement to their parents, but permits the teacher to determine how this should be communicated given the circumstances of instruction and the needs of the student. The new GLA reporting initiative will diminish teachers’ discretion and provide parents with little, if any, additional value.

While the Minister indicates that GLA reporting will be mandatory for students in Grades 1 to 9 beginning in the 2006–07 school year, piloting jurisdictions have expressed concerns about the viability of the Department’s timeline. As well, the Association has been joined by other key stakeholder organizations, including the Alberta School Boards Association, the College of Alberta School Superintendents, the Alberta Home and School Councils Association and the faculties of education in voicing concerns about the design and implementation of the initiative. In response, Deputy Minister Keray Henke has met with stakeholders, including Association representatives, and has committed to continuing to ongoing consultations on the issues that stakeholders believe require clarification or resolution.

What is "Computer Adaptive Assessment" (CAA)?

Teachers will warmly welcome Minister Zwozdesky’s clear and repeated assurances that the use of Computer Adaptive Assessment in the classroom is voluntary and at the discretion of individual teachers.

It is imperative, however, that the Department holds to this commitment and furthermore, ensures that school jurisdictions extend to teachers the same respect for professional judgement and autonomy that the Minister does. If teachers are compelled by their employer jurisdictions to subject students to Computer Adaptive Tests (CAA), either as a jurisdiction initiative or at the behest of the Department, then it makes little difference that the Department of Education has implemented the program on a "voluntary" basis.

Unfortunately, this scenario is not unprecedented. Last June the Department undertook a pilot project requiring jurisdictions to direct teachers in at least one of their schools to mark the multiple choice portion of the Grade 6 and 9 Provincial Achievement Tests and to report the results together with the school awarded marks in the final report sent home in June. Previously, Department policy insured that decisions about the local marking and use of Provincial Achievement Tests in school-based reporting resided exclusively with the individual teacher. If the pilot initiative becomes the basis of a new provincial reporting requirement, teachers’ confidence in the Department’s assurances that teachers will retain professional control over evaluation, including the administration of CAA, will be seriously eroded.

Teachers will also be pleased that the Minister has committed to ensuring that CAA will be closely aligned with the Alberta Program of Studies and that Alberta teachers will develop and validate the questions being used. Concerns remain, however, about the implications of turning over development of CAA content to Castle Rock Research, a private firm, rather than having the Department’s own Learner Assessment Branch develop the assessment. The Branch has a solid reputation for involving teachers in developing high quality examinations; in fact, Castle Rock Research has been granted free use of test materials that were previously developed for use in the Provincial Achievement Tests and Diploma Examinations. Furthermore, the process for developing Provincial Achievement Tests and Diploma Examinations within the Department is still subject to some limited oversight by the profession. This is not the case with the development of CAA as the profession has not been permitted to participate in the steering committee overseeing the Castle Rock Research contract.

Ultimately, developing a CAA tool that can provide a credible or useful assessment of a student’s level of learning will be difficult given the inherent complexity of the curriculum and the limitations imposed by technology and the design of the evaluation. CAAs are typically used as qualification or admission tests in situations where there is considerable variability in the anticipated performance of the population writing the tests. This is why this approach to testing has been employed to assess recruits entering the U.S. military and in applications such as the GRE (Graduate Records Examination) and TOEFL (Test of English as a Foreign Language). It remains to be seen whether this type of test can provide specific, valid and reliable data about an individual student’s performance relative to the full range of objectives set out in the Alberta Program of Studies for a particular subject and grade.

While it is difficult to assess the quality and usefulness of the CAA software before it is delivered, it is clear that the program will be a costly one. The Minister states that the cost will be $3 million over three years. The Association’s estimate of $12 million over nine years includes not only the original development costs identified by the Minister but also the additional post-development cost to the education system to provide access to CAA over an additional six years to all jurisdictions at the rate of $3 per student as set out in the Department’s original Request for Proposal. Of course, the final cost of the program will depend upon the degree to which jurisdictions and individual teachers choose to make use of the technology.

Conclusion

Despite the Association’s continuing concern with the GLA and CAA initiatives, Education Minister Gene Zwozdesky is to be complemented for his willingness to engage teachers in a dialogue about these important initiatives. This Minister’s approach continues to demonstrate his underlying respect for teachers and their professional Association. Teachers continue to hope that the Department will undertake modifications to the initiatives that will address teacher concerns and help to improve the quality of public education.


Also In the News