Tomorrow’s New York Times carries the second installment in the paper’s series “Grading the Digital School.” Like the first installment, this one finds little solid evidence that popular, expensive computer-aided instruction programs actually benefit students. The focus of the new article, written by Trip Gabriel and Matt Richtel, is Cognitive Tutor, a widely esteemed and much coveted software program for teaching math in high schools. The software was developed by Carnegie Learning, a company founded by Carnegie Mellon professors and now owned by Apollo Group, the same company that owns the University of Phoenix.
Carnegie Learning promotes its software as producing “revolutionary results.” It is widely used, and has been applauded by respected thinkers like the Harvard Business School’s Clayton Christensen, who in an article published by the Atlantic two weeks ago used Carnegie Learning as the poster child for the power of software-based education:
Carnegie Learning is the creation of computer and cognitive scientists from Carnegie Mellon University. Their math tutorials draw from cutting-edge research about the way students learn and what motivates them to succeed academically. These scientists have created adaptive computer tutorials that meet students at their individual level of understanding and help them advance via the kinds of exercises they personally find most engaging and effective. The personalization and sophistication is hard for even an expert human tutor to match. It is a powerful, affordable adjunct to classroom instruction, as manifest by Carnegie Learner’s [sic] user base of more than 600,000 secondary students in over 3,000 schools nationwide.
Sounds terrific. But, as the Times story documents, the evidence for Cognitive Tutor’s benefits is weak. In 2010, the U.S. Department of Education analyzed two dozen studies of the software program, and found that it “had no discernible effects” on math test scores for high school students. Another federal study, conducted a year earlier, examined ten leading software programs for teaching math, including Cognitive Tutor, and concluded that they had no “statistically significant effects on test scores.”
Test scores aren’t everything, of course, but it’s fair to say that one of the main reasons cash-strapped schools invest in the computer-aided programs, which can cost three times as much as traditional textbooks, is to boost students’ test scores. And companies like Carnegie Learning use the promise of improved test scores as a prime marketing pitch, sometimes backing it up with cherry-picked case studies or skewed research reports. Certainly, computer-aided instruction programs have a place in schools, but it’s increasingly clear that the benefits of the software have been oversold, and the faith that many educators place in the programs is often unwarranted. As Gabriel and Richtel report: “School officials, confronted with a morass of complicated and sometimes conflicting research, often buy products based on personal impressions, marketing hype or faith in technology for its own sake.”
Back in the early days of the personal computer, the late Steve Jobs, during his original stint at the helm of Apple Computer, played a major role in promoting the use of computers in education. But in a 1996 Wired interview, conducted after he’d left Apple and before he was rehired, he expressed a very different and much more wary view of the role of computer technology in schools:
I used to think that technology could help education. I’ve probably spearheaded giving away more computer equipment to schools than anybody else on the planet. But I’ve had to come to the inevitable conclusion that the problem is not one that technology can hope to solve. What’s wrong with education cannot be fixed with technology. No amount of technology will make a dent … Lincoln did not have a Web site at the log cabin where his parents home-schooled him, and he turned out pretty interesting. Historical precedent shows that we can turn out amazing human beings without technology. Precedent also shows that we can turn out very uninteresting human beings with technology. It’s not as simple as you think when you’re in your 20s – that technology’s going to change the world. In some ways it will, in some ways it won’t.
His view still seems pretty much on the mark.