Learning, Innovation & Tech

Bombs & Breakthroughs

Why Testing Fails: How Numbers Deceive Us All

CJ Westerberg, July 13, 2013 7:04 PM


"Tests fail because of an effect I call the "Numerical Outcome Principle". . . . 
 ". . .Our leaders need to look beyond the numbers,
and judge the context in which the numbers arose. . . "

     - Joe Ganem

Why Testing Fails: How Numbers Deceive Us All

 By Joseph Ganem Ph.D.

There is broad agreement, that improving education requires assessing student outcomes, and holding those responsible accountable for the results. But, the use of standardized tests, as a tool for assessment and accountability, has resulted in more disillusionment than improvement. The heavy emphasis on testing mandated by the No Child Left Behind Law, has not lead to the sought after gains in educational outcomes.

I believe that the failure of mandated testing is a predictable result. Tests fail because of an effect I call the "Numerical Outcome Principle."  Simply put: Once a number is used to judge outcomes, people will adjust their behavior to maximize that particular number. The actual outcome no longer matters. The field of education is rife with examples of the Numerical Outcome Principle in action; instances in which the attainment of lofty goals is undone by the substitution of numbers in place of sound human judgment.

Consider the Scholastic Aptitude Test (SAT), which is a widely used instrument for college admission decisions. The original purpose of the SAT was to provide a level playing field for admission to college. The intent of the test was to identify the students most likely to succeed in college, without regard to social and economic background. In theory, a student from a poor inner city school district, and a student from an expensive private prep school, who have identical SAT scores, should have equal abilities to succeed in college.

But, the result of the SAT is the appearance of an industry that profits through the sale of expensive test preparation services to students. Private companies, such as Kaplan and Princeton Review, charge hundreds of dollars for SAT prep courses, and thousands of dollars for private tutoring. Both companies boast that the combined SAT scores of their students increase more than 100 points because of their services. As a result, there is no way to know from a raw SAT score, if the number represents innate ability, or intensive prep work. A privileged background can provide an advantage on the SAT test, which defeats its purpose.

In response, many colleges have rethought the reliance on SAT scores for admission decisions. Loyola University Maryland, where I teach, no longer requires that applicants submit SAT scores. The fact is success in college depends not only on scholastic aptitude, but also on many other intangibles, such as desire, focus, motivation, and maturity. These intangibles defy quantification, but a competent admissions officer is usually able to make an accurate judgment.

Rather than learn from decades of experience with the shortcomings of the SAT, politicians and educators continue to push testing as a cure for education ills. Consider the testing mandated by the No Child Left Behind Law. The enormous pressure placed on administrators and teachers to raise test scores, has changed both teaching methods and curricula in unintended ways. Teachers must adhere to rigid schedules, and have little flexibility to accommodate natural variations in student interests and abilities. The tests no longer assess the effectiveness of curricula; instead curricula are determined by the tests.

The result is the phenomena of "teaching-to-the-test."   As a teacher, I have always found that a strange expression. After all, I give tests, and I only test what I teach. To do otherwise would be unfair to the students. Few people argue against tests, as a method to assess students learning, and test scores as a basis for assigning grades. Used appropriately, tests are a useful learning tool, for both teachers and students. Tests motivate students to study, and inform teachers about the effectiveness of their instruction.

But, "teaching-to-the-test" is something different. It is an educational mindset, in which test scores are not measures of learning outcomes; the test scores are the outcomes. While that distinction might be subtle, it has real effects on how classes are taught, and in the messages we communicate to students about the goals of an education. Tests are measurement tools; they should not be the reasons that students come to class.

The aversion, that "teaching-to-the-test" evokes, arises because at its essence, it is a form of manipulation. Children are perceptive, and they can figure out when adults are manipulating them. Teachers are perceptive, and they will adjust their actions to the reward structure in place. The net result is a poisonous class atmosphere, rather than a collaborative atmosphere needed for learning. It is the Numerical Outcome Principle in action. Which is why testing will fail.

Teachers and students are perceptive. What we need are leaders who are perceptive, and willing to use judgment instead of hiding behind meaningless numbers. A school is not a factory turning out widgets with identical product specifications. But, too often, the characterizing of educational outcomes solely in terms of test scores, results in the mindset of manufacturing company, rather than an educational institution. Our leaders need to look beyond the numbers, and judge the context in which the numbers arose. Each child, each teacher, each school is unique. We should celebrate that uniqueness, instead of trying to eliminate it.


Originally Published by The Daily Riff February 2010

Joseph Ganem, Ph.D., with link to home page and blog here,  is a professor of physics at Loyola University Maryland, and author of the award-winning book on personal finance: The Two Headed Quarter: How to See Through Deceptive Numbers and Save Money on Everything You Buy. It shows how numbers fool consumers when they make financial decisions.

  • Richard Wills

    I've been Dean of Admissions at three top universities, and been a consultant to the College Board, Educational Testing Service, The Princeton Review, and Kaplan- this article is dead on target about what's wrong with education in America. These tests are meaningless, and the testing services know it. In fact, The College Board legally changed the name of the Scholastic Aptitude Test to just the SAT several years ago in a tacit acknowledgment of this fact.

blog comments powered by Disqus
It had long since come to my attention that people of accomplishment rarely sat back and let things happen to them. They went out and happened to things.
Leonardo da Vinci
Follow The Daily Riff on Follow TDR on Twitter

find us on facebook


Cool Smart Teen: Changing the conversation about building stuff

SMW, 11.04.2013

Not just building an app to play with . . . but a test for cancer

Read Post | Comments

Riffing good stories

Stanford.design-thinking. innovation.jpg

Video: What is the link between "design-thinking" and creativity?

SMW, 11.03.2013

Creative Confidence: Must-Watch Interview on Charlie Rose

Read Post | Comments
early education.preschool.jpg

How Much Do Early Years Lead to Student Success?

CJ Westerberg, 11.03.2013

"The idea being that wasn't necessarily to create little engineers everywhere, though that would be great for us. The idea was to take advantage of the natural learning process and curiosity that children have at that age . . ." Tamika Lang, Boeing

Read Post | Comments

NEW: Weekend Reading

SMW, 11.03.2013

Looking for Intimacy in the Age of Facebook is a higher education course (yes, you heard right). Interestingly, social media may inhibit creativity and innovation

Read Post | Comments

The New Resume

SMW, 10.31.2013

A picture is worth a thousand words. A video game-like format is worth . . .

Read Post | Comments

Noam Chomsky: The Purpose of Education

CJ Westerberg, 10.31.2013

"Education is really aimed at helping students get to the point where they can learn on their own. . . "

Read Post | Comments

Noam Chomsky: In the news again, this time about Twitter and High Schoolers

SMW, 10.31.2013

Noam Chomsky, Professor Emeritus at MIT, "father of linguistics", political commentator.  From bio: "Besides his work in linguistics, Chomsky is internationally recognized as one of the most critically engaged public intellectuals alive today":               ...

Read Post | Comments

Choke: Test-taking - - - a different way to look at test-prep?

CJ Westerberg, 10.28.2013

"Most students will not find a steady diet of test-prep drills and worksheets to be particularly meaningful, and accordingly, they will not put forth optimal learning effort."

Read Post | Comments

Does your school have a beehive?

CJ Westerberg, 10.27.2013

"If we want children to be inventors, we have to give them opportunities to invent." - 5:00 mark, Mission Hill video

Read Post | Comments