"No Effects" Studies

Education Week's got this article out about randomized trials producing "no effects." According to the article, these null findings are raising eyebrows and "prompting researchers, product developers, and other experts to question the design of the studies, whether the methodology they use is suited to the messy real world of education, and whether the projects are worth the cost, which has run as high as $14.4 million in the case of one such study."

Wow, is that ever a disappointing reaction. Here's why:

1. We should be psyched, not upset, that studies with null effects are being released. That is not always the case. Publication bias, anyone? I've often thought that studies demonstrating null effects need to be publicized even more widely than those that find positive or negative impacts. Too many places out there are at the behest of funders, and can't release null findings. Too many assistant professors don't get tenure because they "didn't find anything." Are you kidding me? If it's a current practice and you learn it doesn't produce any effects, either way, it needs to be out there. We should learn as much from null findings and "worst practices" as we do from "statistically significant" impacts and "best practices."

2. Saying that experimentation isn't suited to the "messy real world" is a cop out. It lumps many different kinds of experiments into one category-- the good, the bad, and the ugly. Field experiments, lab settings, cluster-randomized trials with volunteer districts, and student-level randomized experiments with participants selected via administrative data-- these are very different animals. Each approach has a differential potential for generalizable results (external validity) and varying levels of challenges to internal validity as well. I'll grant you, experiments that rely on volunteer samples probably can't help us much in education-- since in real life programs aren't applied to students, families or schools who volunteer--they apply to everyone. This is especially a problem when we try interventions to close achievement gaps-- African-Americans who volunteer for studies are very, very different from those who do not (Tuskegee anyone?).

3. Doing experiments well costs a LOT of money. Putting trials on tight budgets helps to ensure they aren't run well--PIs cannot build the kinds of relationships that promote treatment fidelity, cannot collect high-quality data, and cannot get inside the black box of mechanisms--and instead are stuck simply estimating average treatment effects. No drug works for everyone, and no drug works in the exact same way for everyone-- the medical community knows this, and uses larger samples to make identifying differential and heterogeneous effects possible. When is Education going to catch up?

4. One thing I do agree with this article on. The model IES is using needs some revisions. I heard William T. Grant president Bob Granger give a great talk at SREE recently, where he made the point that the usual 'try small things then scale them up' model isn't going anywhere fast. We need to know how current policies work as currently implemented-- at scale. Go after that, spend what's necessary to conduct experiments with higher internal AND external validity, and support researchers to do this who reject old models and try new things. I promise you, we'll get somewhere.

You have read this article Education Week / Institute for Education Sciences / research with the title "No Effects" Studies. You can bookmark this page URL http://apt3e.blogspot.com/2009/03/effects-studies.html. Thanks!