Learning, Innovation & Tech

Bombs & Breakthroughs

Why Testing Fails: How Numbers Deceive Us All

CJ Westerberg, July 13, 2013 7:04 PM


"Tests fail because of an effect I call the "Numerical Outcome Principle". . . . 
 ". . .Our leaders need to look beyond the numbers,
and judge the context in which the numbers arose. . . "

     - Joe Ganem

Why Testing Fails: How Numbers Deceive Us All

 By Joseph Ganem Ph.D.

There is broad agreement, that improving education requires assessing student outcomes, and holding those responsible accountable for the results. But, the use of standardized tests, as a tool for assessment and accountability, has resulted in more disillusionment than improvement. The heavy emphasis on testing mandated by the No Child Left Behind Law, has not lead to the sought after gains in educational outcomes.

I believe that the failure of mandated testing is a predictable result. Tests fail because of an effect I call the "Numerical Outcome Principle."  Simply put: Once a number is used to judge outcomes, people will adjust their behavior to maximize that particular number. The actual outcome no longer matters. The field of education is rife with examples of the Numerical Outcome Principle in action; instances in which the attainment of lofty goals is undone by the substitution of numbers in place of sound human judgment.

Consider the Scholastic Aptitude Test (SAT), which is a widely used instrument for college admission decisions. The original purpose of the SAT was to provide a level playing field for admission to college. The intent of the test was to identify the students most likely to succeed in college, without regard to social and economic background. In theory, a student from a poor inner city school district, and a student from an expensive private prep school, who have identical SAT scores, should have equal abilities to succeed in college.

But, the result of the SAT is the appearance of an industry that profits through the sale of expensive test preparation services to students. Private companies, such as Kaplan and Princeton Review, charge hundreds of dollars for SAT prep courses, and thousands of dollars for private tutoring. Both companies boast that the combined SAT scores of their students increase more than 100 points because of their services. As a result, there is no way to know from a raw SAT score, if the number represents innate ability, or intensive prep work. A privileged background can provide an advantage on the SAT test, which defeats its purpose.

In response, many colleges have rethought the reliance on SAT scores for admission decisions. Loyola University Maryland, where I teach, no longer requires that applicants submit SAT scores. The fact is success in college depends not only on scholastic aptitude, but also on many other intangibles, such as desire, focus, motivation, and maturity. These intangibles defy quantification, but a competent admissions officer is usually able to make an accurate judgment.

Rather than learn from decades of experience with the shortcomings of the SAT, politicians and educators continue to push testing as a cure for education ills. Consider the testing mandated by the No Child Left Behind Law. The enormous pressure placed on administrators and teachers to raise test scores, has changed both teaching methods and curricula in unintended ways. Teachers must adhere to rigid schedules, and have little flexibility to accommodate natural variations in student interests and abilities. The tests no longer assess the effectiveness of curricula; instead curricula are determined by the tests.

The result is the phenomena of "teaching-to-the-test."   As a teacher, I have always found that a strange expression. After all, I give tests, and I only test what I teach. To do otherwise would be unfair to the students. Few people argue against tests, as a method to assess students learning, and test scores as a basis for assigning grades. Used appropriately, tests are a useful learning tool, for both teachers and students. Tests motivate students to study, and inform teachers about the effectiveness of their instruction.

But, "teaching-to-the-test" is something different. It is an educational mindset, in which test scores are not measures of learning outcomes; the test scores are the outcomes. While that distinction might be subtle, it has real effects on how classes are taught, and in the messages we communicate to students about the goals of an education. Tests are measurement tools; they should not be the reasons that students come to class.

The aversion, that "teaching-to-the-test" evokes, arises because at its essence, it is a form of manipulation. Children are perceptive, and they can figure out when adults are manipulating them. Teachers are perceptive, and they will adjust their actions to the reward structure in place. The net result is a poisonous class atmosphere, rather than a collaborative atmosphere needed for learning. It is the Numerical Outcome Principle in action. Which is why testing will fail.

Teachers and students are perceptive. What we need are leaders who are perceptive, and willing to use judgment instead of hiding behind meaningless numbers. A school is not a factory turning out widgets with identical product specifications. But, too often, the characterizing of educational outcomes solely in terms of test scores, results in the mindset of manufacturing company, rather than an educational institution. Our leaders need to look beyond the numbers, and judge the context in which the numbers arose. Each child, each teacher, each school is unique. We should celebrate that uniqueness, instead of trying to eliminate it.


Originally Published by The Daily Riff February 2010

Joseph Ganem, Ph.D., with link to home page and blog here,  is a professor of physics at Loyola University Maryland, and author of the award-winning book on personal finance: The Two Headed Quarter: How to See Through Deceptive Numbers and Save Money on Everything You Buy. It shows how numbers fool consumers when they make financial decisions.

  • Richard Wills

    I've been Dean of Admissions at three top universities, and been a consultant to the College Board, Educational Testing Service, The Princeton Review, and Kaplan- this article is dead on target about what's wrong with education in America. These tests are meaningless, and the testing services know it. In fact, The College Board legally changed the name of the Scholastic Aptitude Test to just the SAT several years ago in a tacit acknowledgment of this fact.

blog comments powered by Disqus
It had long since come to my attention that people of accomplishment rarely sat back and let things happen to them. They went out and happened to things.
Leonardo da Vinci
Follow The Daily Riff on Follow TDR on Twitter

find us on facebook


Big Thinking & the College Promise

SMW, 08.06.2014

" . . . these kids, like any others, are hungry for big ideas."

Read Post | Comments

Riffing good stories

visionsof math.jpg

Visions of Mathematics: What Content Should We Teach?

CJ Westerberg, 07.29.2014

Ben Daley of High Tech High Outlines a Revolutionary Approach to Math

Read Post | Comments

Singapore Math Demystified!

CJ Westerberg, 07.29.2014

Guest Post By Scarsdale NY Teacher Bill Jackson
The Daily Riff EXCLUSIVE
Can It Help Solve Our Country's Math-phobia?

Read Post | Comments

Stephen does Common Core Math

SMW, 06.30.2014

Video in case you missed it - Math!

Read Post | Comments

There is No Education Without the Arts

SMW, 06.29.2014

"We're not talking about making sure that everybody has private music lessons,"

Read Post | Comments

College Professors Read Their Student Reviews

SMW, 06.29.2014

 Quick Video Below Gotta love the college profs for sharing -...

Read Post | Comments

Is Learning a Sport?

CJ Westerberg, 06.25.2014

Maybe . . . Why Kids Care More About Achievement than Helping Others

Read Post | Comments

Disrupt or be disrupted: the original article + the response

SMW, 06.16.2014

"His job appears to be to convince a generation of people who want to do good and do well to learn, instead, remorselessness. Forget rules, obligations, your conscience, loyalty, a sense of the commonweal. " - Jill Lepore

Read Post | Comments

Parenting Cats and Dogs

SMW, 06.16.2014

Adolescents, on the other hand, respond more like cats - aloof, unpredictable, hard to locate, not always coming when they are called, rubbing up against you or jumping into your lap on their terms.

Read Post | Comments