Community-Based Assessment Makes the Grade

We're fast approaching a point in this country when the promotion or graduation of students will result not from their classroom work or the opinions of the educators who spend each day with them but from their performance on a single standardized test. Because I've spent the last 15 years inside the testing industry -- working for many of the biggest companies on many of the biggest tests -- this trend doesn't seem so smart to me.

In fact, I'd say linking federal education funds to regional standardized test scores (as No Child Left Behind does) or teacher pay to student test results (the probable, but unintended, outcome of President Obama's Race to the Top program) are ideas that should be reconsidered.

My complaint with large-scale assessment does not lie with the multiple-choice tests, because those are scored electronically. The real trouble begins in the realm of open-ended tests, where students answer questions in their own words and are assessed by fallible human beings. The testing industry wants those subjective student responses to be scored as consistently as multiple-choice tests.

To do this, the industry establishes hard-and-fast rules for its short-term "professional scorers" to adhere to. In my experience, these rules -- written for recently hired temporary employees -- ultimately turn the process into a theater of the absurd. I know, because I've sat through the training sessions.

Working on a national assessment test in 2005, I helped establish scoring rules for a test question that asked students performing a hands-on science task to describe what happened when they mixed a liquid and a solid. The rubric, written by classroom teachers, said full credit should be awarded to answers showing "complete understanding."

But everyone had a different idea of "complete understanding." So the test company tried to specify exactly what that meant. I sat in on a lengthy conference call filled with test developers and science teachers as we tried to hammer out the right and wrong student responses, and I was amazed as those earnest educators considered potential responses.

"If we accept 'The liquid bubbled,'" one scientist said, "then I don't see how we can't accept 'It sizzled.'"

"But sizzled isn't the same as bubbled," another argued, and soon everyone on the phone was debating whether boiled meant the same as sizzled, fizzled the same as sizzled, fizzed the same as fizzled.

When people ask how I would reform standardized testing, I point to models that work on a smaller scale. In the current system, temporary employees must adhere to unyielding rules established to deal with tens of thousands of student responses. A reformed system would have a smaller number of scorers assessing the work of a smaller number of students. This means placing assessment back in the hands of the teacher who can make thoughtful decisions about the students he or she knows.

If small-scale assessment sounds like an expensive solution that won't fly in today's economy, consider Washington State's recent achievements. In response to a 2004 ballot initiative, the state rolled out a comprehensive classroom-based-assessment program for social studies, health, and the arts. These CBAs are written and administered on the state level, but student results are assessed by classroom teachers. This makes for a win-win: Administrators and policy makers receive standardized results across the state, and students are spared the obvious downfalls of large-scale test scoring.

Organizations like Boston-based FairTest consider programs such as Washington's to be authentic assessments. This is because the CBAs are based on student performances or portfolios they produce over a period of time. In this scenario, assessment no longer rests on the open-ended answers that students recall on one stressful day.

It is increasingly important to change the testing industry. Race to the Top is based on national assessment criteria, and that is set to become the new gatekeeper for federal education funding. Absent reform, we are placing life-changing assessments about students in the hands of bored temps who give fleeting glances to students' work.

Todd Farley is the author of Making the Grades: My Misadventures in the Standardized Testing Industry.

More on 21st-Century Assessment

Share your thoughts on this topic at Edutopia.org's Assessment group.