Learning, Innovation & Tech

Bombs & Breakthroughs

Why Testing Fails: How Numbers Deceive Us All

CJW, September 8, 2010 7:04 PM


Read This First.


"Tests fail because of an effect I call the "Numerical Outcome Principle". . . . 
 ". . .Our leaders need to look beyond the numbers,
and judge the context in which the numbers arose. . . "

     - Joe Ganem

Why Testing Fails

 By Joseph Ganem Ph.D.

There is broad agreement, that improving education requires assessing student outcomes, and holding those responsible accountable for the results. But, the use of standardized tests, as a tool for assessment and accountability, has resulted in more disillusionment than improvement. The heavy emphasis on testing mandated by the No Child Left Behind Law, has not lead to the sought after gains in educational outcomes.

I believe that the failure of mandated testing is a predictable result. Tests fail because of an effect I call the "Numerical Outcome Principle."  Simply put: Once a number is used to judge outcomes, people will adjust their behavior to maximize that particular number. The actual outcome no longer matters. The field of education is rife with examples of the Numerical Outcome Principle in action; instances in which the attainment of lofty goals is undone by the substitution of numbers in place of sound human judgment.

Consider the Scholastic Aptitude Test (SAT), which is a widely used instrument for college admission decisions. The original purpose of the SAT was to provide a level playing field for admission to college. The intent of the test was to identify the students most likely to succeed in college, without regard to social and economic background. In theory, a student from a poor inner city school district, and a student from an expensive private prep school, who have identical SAT scores, should have equal abilities to succeed in college.

But, the result of the SAT is the appearance of an industry that profits through the sale of expensive test preparation services to students. Private companies, such as Kaplan and Princeton Review, charge hundreds of dollars for SAT prep courses, and thousands of dollars for private tutoring. Both companies boast that the combined SAT scores of their students increase more than 100 points because of their services. As a result, there is no way to know from a raw SAT score, if the number represents innate ability, or intensive prep work. A privileged background can provide an advantage on the SAT test, which defeats its purpose.

In response, many colleges have rethought the reliance on SAT scores for admission decisions. Loyola University Maryland, where I teach, no longer requires that applicants submit SAT scores. The fact is success in college depends not only on scholastic aptitude, but also on many other intangibles, such as desire, focus, motivation, and maturity. These intangibles defy quantification, but a competent admissions officer is usually able to make an accurate judgment.

Rather than learn from decades of experience with the shortcomings of the SAT, politicians and educators continue to push testing as a cure for education ills. Consider the testing mandated by the No Child Left Behind Law. The enormous pressure placed on administrators and teachers to raise test scores, has changed both teaching methods and curricula in unintended ways. Teachers must adhere to rigid schedules, and have little flexibility to accommodate natural variations in student interests and abilities. The tests no longer assess the effectiveness of curricula; instead curricula are determined by the tests.

The result is the phenomena of "teaching-to-the-test."   As a teacher, I have always found that a strange expression. After all, I give tests, and I only test what I teach. To do otherwise would be unfair to the students. Few people argue against tests, as a method to assess students learning, and test scores as a basis for assigning grades. Used appropriately, tests are a useful learning tool, for both teachers and students. Tests motivate students to study, and inform teachers about the effectiveness of their instruction.

But, "teaching-to-the-test" is something different. It is an educational mindset, in which test scores are not measures of learning outcomes; the test scores are the outcomes. While that distinction might be subtle, it has real effects on how classes are taught, and in the messages we communicate to students about the goals of an education. Tests are measurement tools; they should not be the reasons that students come to class.

The aversion, that "teaching-to-the-test" evokes, arises because at its essence, it is a form of manipulation. Children are perceptive, and they can figure out when adults are manipulating them. Teachers are perceptive, and they will adjust their actions to the reward structure in place. The net result is a poisonous class atmosphere, rather than a collaborative atmosphere needed for learning. It is the Numerical Outcome Principle in action. Which is why testing will fail.

Teachers and students are perceptive. What we need are leaders who are perceptive, and willing to use judgment instead of hiding behind meaningless numbers. A school is not a factory turning out widgets with identical product specifications. But, too often, the characterizing of educational outcomes solely in terms of test scores, results in the mindset of manufacturing company, rather than an educational institution. Our leaders need to look beyond the numbers, and judge the context in which the numbers arose. Each child, each teacher, each school is unique. We should celebrate that uniqueness, instead of trying to eliminate it.


Joseph Ganem, Ph.D., with link to home page and blog here,  is a professor of physics at Loyola University Maryland, and author of the award-winning book on personal finance: The Two Headed Quarter: How to See Through Deceptive Numbers and Save Money on Everything You Buy. It shows how numbers fool consumers when they make financial decisions.

Post a Comment

Speed in tennis is a strange mixture of intuition, guesswork, footwork and hair-trigger reflexes. Many of the players famed for quickness on court would finish dead last in a field of schoolgirls in a race over any distance more than ten yards.
Eugene Scott, world-ranked American tennis player and journalist

Passionate about our edu-nation?

RIFF it here

Follow The Daily Riff on Follow TDR on Twitter

find us on facebook


The Professional Lives of Teachers in Singapore

CJW, 09.10.2010

An American Math Teacher Travel Journal to Singaporean Schools

Read Post | Comments

Riffing good stories


Singapore: Five Surprises in Education

CJW, 09.10.2010

An American Math Teacher's Travel Journal

Read Post | Comments

Does Education Need "A New Sheriff"?

CJW, 09.09.2010

Gotta Watch Two Minute "Truth in Humor" Rap Viral Video

Read Post | Comments
waiting for superman.jpg

Waiting for Superman: A "Signal Moment" In Education?

CJW, 09.09.2010

"It's a moment ripe with ferment and possibility, but also rife with conflict, in which the kind of change that fills many hearts with hope fills others with mortal dread- and which gives a movie like 'Superman' a rare chance to move the needle."

Read Post | Comments

"What Makes A School Great": Time Magazine Special Report

CJW, 09.09.2010

Teachers, Charter Schools, Business, Testing, Unions In The Hot Seat on "Morning Joe"Richard Stengel, Time magazine's editor-in-chief, announced next week's cover story, "What Makes A School Great" which, if judged by the teacher-obsessed conversation on today's MSNBC's "Morning Joe", should...

Read Post | Comments

The Illustrated Guide To A Ph.D.

CJW, 09.09.2010

Very cool representation of the process and end result of obtaining a Ph.D. - all in 10 quick slides.  Very clever conclusion - check it out with link below:"Imagine a circle that contains all of human knowledge . ....

Read Post | Comments

Why Testing Fails: How Numbers Deceive Us All

CJW, 09.08.2010

"Tests fail because of an effect I call the "Numerical Outcome Principle". . .Our leaders need to look beyond the numbers..." Guest Post By Joseph Ganem, Ph.D.

Read Post | Comments

Parents Welcome

CJW, 09.08.2010

A Fictional Back-To-School Newsletter

Read Post | Comments

The Expectations Trap

CJW, 09.08.2010

College Choices: The Unintended Parent-Student Disconnect

Read Post | Comments