« Turn the page, turn the channel | Main | Anthologized »

Close down the schools!

August 20, 2009

The headline on Steve Lohr's Bits post sounds pretty definitive: "Study Finds That Online Education Beats the Classroom." And the quote that Lohr gets from the study's lead author, Barbara Means, sounds equally definitive: “The study’s major significance lies in demonstrating that online learning today is not just better than nothing — it actually tends to be better than conventional instruction."

Predictably, Lohr's post is now inspiring even more extreme summaries of the study. "Want to learn better?" screams the headline at the misnamed SmartPlanet.com. "Crack open your laptop (and ditch the classroom)." Chimes in Podcasting News: "Why Go to School? Study Finds Students Do Better Online."

But the study itself, which was conducted by SRI International for the US Department of Education, is considerably less definitive than the coverage would have you believe. Before school boards start firing teachers and shuttering classrooms, they might want to read the actual report.

The SRI study is a "meta-analysis," or a "study of studies." The authors reviewed research, dating back to 1996, of online education. Of the 1,132 studies they found, they focused their analysis on just 51 "experimental and quasi-experimental studies" that provided sufficient data to directly compare the "learning outcomes" of online courses with those of traditional "face-to-face" courses. In many cases, however, the "online courses" also involved face-to-face instruction; in other words, the online instruction supplemented (rather than replaced) classroom instruction. Only about half of the 51 analyzed studies involved purely online courses. Also, 19 of the 51 studies involved courses that lasted less than a month. And, as the authors note, "many of the studies suffered from weaknesses such as small sample sizes; failure to report retention rates for students in the conditions being contrasted; and, in many cases, potential bias stemming from the authors’ dual roles as experimenters and instructors."

With those limitations in mind, the authors' overall conclusion is as follows: "The corpus of 51 [experiments] was sufficient to demonstrate that in recent applications, online learning has been modestly more effective, on average, than the traditional face-to-face instruction with which it has been compared."

But the authors also provide some important additional caveats:

First, while the intent of this meta-analysis is "to provide policy-makers, administrators and educators with research-based guidance about how to implement online learning for K–12 education and teacher preparation," the study doesn't actually look at K-12 education. In fact, of the 51 studies analyzed, only five were drawn from K-12 education - and none of those involved purely online instruction. The bulk of the studies involved instruction in college or graduate school or in professional training courses, particularly instruction in "medicine or health care" but also in "computer science, teacher education, social science, mathematics, languages, science and business." In fact, when the five K-12 studies were examined in isolation, they revealed no statistically meaningful benefit from online learning (see page xvii of the report). As the authors emphasize, it would be rash to conclude that the study results will apply to K-12 schools.

Second, the finding that online or online-supplemented courses have slightly better learning outcomes may not have anything to do with the fact that they were online. It may simply be that they involved more instructional time than the strictly classroom courses. This is a point that the authors make repeatedly in their report. For example: "Despite what appears to be strong support for online learning applications, the studies in this meta-analysis do not demonstrate that online learning is superior as a medium. In many of the studies showing an advantage for online learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages." The findings of the meta-analysis, the authors later reiterate, "should not be construed as demonstrating that online learning is superior as a medium."

Finally, it should be noted that, of the 51 experiments studied, only 11 actually showed a statistically significant advantage to online instruction. (The authors don't specify how many of those 11 involved purely online instruction versus classroom/online hybrids. Nor do they specify how many involved vocational instruction.) Two of the experiments showed a statistically significant advantage to face-to-face instruction. The vast majority of the studies showed no meaningful differences in outcomes.

Sometimes, the caveats in a study speak louder than the findings. I think that's the case here. A suggestion that classroom instruction supplemented by online exercises can lead to more learning than classroom instruction alone would hardly come as a surprise. But that's about the only firm conclusion I draw from this meta-analysis.

Posted by nick at August 20, 2009 01:06 PM

Advertisement: Coming this spring: Nicholas Carr's new book The Shallows: What the Internet Is Doing to Our Brains. Preorder now from Amazon.

Comments

Having taught decades in the classroom and a few years online, I think any real difference is due to the teachers involved. It's more often true that online teachers are excited about what they're doing, and they want to make a difference. So they use the best of the techniques and tools available.

I was somewhat surprised to discover that I got to know more of my online students in a personal way than I did in the classroom. I think it was because online lent itself to more one-on-one interaction.

OTOH, a teacher at one of the colleges where I taught just shoveled his lecture notes online and hardly ever interacted with the students.

Posted by: LaneLester at August 20, 2009 04:07 PM

@LaneLester: good ol' Hawthorne effect: studies involving new technology often come with enthusiastic practitioners. The more serious studies are aware of this factor, and thus don't take their own statistics seriously. Instead, they focus on the causal explanations.
Sure, a motivated and intelligent teacher will make the most of any resources given, and with all other factors stable, will do better with additional technology. That's a no-brainer. The question is, *how* do you use technology better.
The primary danger with such studies, or rather with their interpretation is that they reinforce the illusion that technology can replace the teacher.
Besides, school was were I learnt to date, became politically involved, and made life-long mates. Can I please see the statistics on these dimensions of on-line instruction?

Posted by: yish at August 21, 2009 03:02 AM

"I have never let my schooling interfere with my education."
-Mark Twain

Posted by: Linuxguru1968 at August 21, 2009 06:06 PM

Lessons in the classroom can be just as interesting as the on-line ones. I worked in special needs environment for a while (while studying for an MA) I found a combination of both these methods stimulate a student what really counts though is to maintain focus on individual students

Posted by: Dape at August 27, 2009 04:33 AM

Might be a huge movement if this swine flu keeps going the way it has been. Woke up this morning to a suspected 50 case at CU in Boulder, CO, however, I think maybe half could just be really bad hangovers knowing that school!

Posted by: Steven Schwartz - The SAN Technologist at August 28, 2009 04:03 PM

Close down the schools!

August 20, 2009

Comments

Post a comment

Greatest hits

Other writing

Rough Type is: