This New Yorker article by Malcolm Gladwell dates back to 2001, but it is has timely relevance to the UK Higher ducation debate...
The SAT is now seventy-five years old, and it is in trouble. Earlier this year, the University of California - the nation's largest public-university system - stunned the educational world by proposing a move toward a "holistic" admissions system, which would mean abandoning its heavy reliance on standardized-test scores. The school backed up its proposal with a devastating statistical analysis, arguing that the SAT is virtually useless as a tool for making admissions decisions.
The report focussed on what is called predictive validity, a statistical measure of how well a high-school student's performance in any given test or program predicts his or her performance as a college freshman.
If you wanted to, for instance, you could calculate the predictive validity of prowess at Scrabble, or the number of books a student reads in his senior year, or, more obviously, high-school grades.
What the Educational Testing Service (which creates the SAT) and the College Board (which oversees it) have always argued is that most performance measures are so subjective and unreliable that only by adding aptitude-test scores into the admissions equation can a college be sure it is picking the right students.
This is what the UC study disputed. It compared the predictive validity of three numbers: a student's high-school Grade Point Average (GPA), his or her score on the SAT (or, as it is formally known, the SAT-I), and his or her score on what is known as the SAT-II, which is a so-called achievement test, aimed at gauging mastery of specific areas of the high-school curriculum.
Drawing on the transcripts of seventy-eight thousand University of California freshmen from 1996 through 1999, the report found that, over all, the most useful statistic in predicting freshman grades was the SAT-II, which explained sixteen per cent of the "variance" (which is another measure of predictive validity). The second most useful was high-school GPA, at 15.4 per cent. The SAT was the least useful, at 13.3 per cent.
Combining high-school GPA and the SAT-II explained 22.2 per cent of the variance in freshman grades. Adding in SAT-I scores increased that number by only 0.1 per cent.
Nor was the SAT better at what one would have thought was its strong suit: identifying high-potential students from bad schools.
In fact, the study found that achievement tests were ten times more useful than the S.A.T. in predicting the success of students from similar backgrounds. "Achievement tests are fairer to students because they measure accomplishment rather than promise," Richard Atkinson, the president of the University of California, told a conference on college admissions last month.
"[Achievement tests] can be used to improve performance; they are less vulnerable to charges of cultural or socioeconomic bias; and they are more appropriate for schools because they set clear curricular guidelines and clarify what is important for students to learn. Most important, they tell students that a college education is within the reach of anyone with the talent and determination to succeed."