Experimental Study Finds Vouchers Don't Boost College Attendance

It all started on Twitter, and then on this blog.... now this!

BOULDER, CO (September 13, 2012) – A recent Brookings Institution report that looked at college enrollment rates of students attending voucher schools in New York City acknowledged no overall impacts of the vouchers on college attendance, but its authors trumpeted large, positive impacts for a subgroup of the voucher students: African Americans.

A new review of the report, however, questions the claim of a strong positive impact even for that group.

The Effects of School Vouchers on College Enrollment: Experimental Evidence from New York Citywas written by Matthew Chingos and Paul Peterson and published jointly by Brookings and by the Program on Education Policy and Governance at Harvard.

It was reviewed for the Think Twice think tank review project by professor Sara Goldrick-Rab of the University of Wisconsin-Madison. The review is published by the National Education Policy Center, housed at the University of Colorado Boulder School of Education.

The report examines college enrollment rates of students participating in an experimental voucher program in New York City, which in the spring of 1997 offered 3-year scholarships worth up to $1,400 annually to low-income families.

In her review of the Brookings report, Goldrick-Rab observes that the study identifies no overall impacts of the voucher offer, but that the authors “report and emphasize large positive impacts for African American students, including increases in college attendance, full-time enrollment, and attendance at private, selective institutions of higher education.”

This strong focus on positive impacts for a single subgroup of students is not warranted. Goldrick-Rab notes four problems:

• There are no statistically significant differences in the estimated impact for African Americans as compared to other students;
• There is important but unmentioned measurement error in the dependent variables (college attendance outcomes) affecting the precision of those estimates and likely moving at least some of them out of the realm of statistical significance;
• The authors fail to demonstrate any estimated negative effects that could help explain the average null results; and
• There are previously existing differences between the African American treatment and control groups on factors known to matter for college attendance (e.g., parental education).

“Contrary to the report’s claim, the evidence presented suggests that in this New York City program, school vouchers did not improve college enrollment rates among all students or even among a selected subgroup of students,” Goldrick-Rab writes.

Consequently, this new study’s contribution to discussions of education policy is the opposite of what its authors intend. Goldrick-Rab concludes that the report “convincingly demonstrates that in New York City a private voucher program failed to increase the college enrollment rates of students from low-income families.”

You have read this article Brookings Institution / college attendance / experimental studies / Harvard University / matt chingos / paul peterson / vouchers / what works with the title vouchers. You can bookmark this page URL http://apt3e.blogspot.com/2012/09/experimental-study-finds-vouchers-don.html. Thanks!

Vouchers and College Attendance: Puzzling Findings Deserve Much Caution

Several months ago I described the problems in a study that seemed to have great policy relevance, but little empirical support for its contentions.  Sadly, examples of studies like these abound in education, and another is currently making headlines.  "Vouchers Boost Blacks' College Enrollment Rates," claim the stories-- and boy do the effects seem large! A "24 percent increase" in college attendance among black recipients of those vouchers-- what a dream. And it must be an accurate statement, right, since this was an experiment?

Well, not necessarily.

Too many practitioners, policymakers, and even researchers are far too inclined when they hear the words "randomized trial" to ignore the usual concerns about the reliability and validity of estimated program effects.  After all, it's "gold standard," and touted by the Institute of Education Sciences as being the most valid to get a sense of how well programs work. Unfortunately, its usefulness is a bit more limited than that-- first, experiments don't always work as planned in creating equivalent initial groups for later comparison, and second, they often tell us only how well the intervention worked under a set of very specific conditions and circumstances, that are often crucial but rarely described in detail.  Moreover, unless they are really carefully planned in advance, their post-hoc analyses can get particularly squirrelly when it comes to estimating different effects for different people.

For these reasons, I'm not sharing in the wild enthusiasm over the new Brookings study by Paul Peterson and Matt Chingos that purports to show that vouchers provide a big boost to college attendance to a very at-risk group: African-Americans.

I started laying out these concerns a few days ago via Twitter, but am restating and summarizing them here, in case it's useful to those who don't spend all of their time obsessing about methodology and need to know what really works in education.

Here are three reasons why the findings don't pass my sniff test:

(1) The estimated average treatment effect of offering the voucher is null.  Since the effects of receiving the voucher is positive and large for one group- African-Americans-- this implies that the effects must be negative for another group, and yet this is never mentioned.  Why? It's rather unusual to show effects for only selected groups, and not for all of them. Most importantly, it goes against best practices.

(2) The only subgroup with effects, African-Americans, is a group that doesn't seem to have equivalent treatment and control groups before the offer of the voucher.  If anything, the treatment group students seem more inclined to college attendance independent of the voucher, given that more of their parents have bachelor's degrees (while other factors are also imbalanced, this one is a known drive of college attendance, among the most important).  While the authors attend to this issue a bit, and try one kind of sensitivity analysis to adjust for it, in their text they fail the potential flaws all of the cautions they deserves-- even going so far as to making this finding the main highlight of the paper.

(3) In the paper and the press the authors stress the effects of receiving a voucher but voucher receipt is not randomly assigned.  So if you are excited about the experimental component-- in a study that claims to be "The first to measure the impact of school vouchers on college enrollment" -- you need to know that the main result (for example, see paragraph 1 here) isn't experimental. This is a quasi-experimental approach and is subject to the usual kinds of errors.

Are these flaw par for the course, and thus no big deal? I don't think so.  There was an evident PR effort behind this report, and it's led to widespread reporting of a study that really needs more vetting.  Words like "the Brookings Institution at Harvard" (sidenote: huh?) give it more credibility than it deserves at this stage, and the term "experiment" makes folks feel overly confident in the results.

Now, all that said, I do understand how these things can happen.  Since they suggest differential responsiveness to programs (and thus the potential for cutting costs while increasing program effectiveness), subgroup analyses are quite seductive and compelling, as are randomized trials themselves. Last year, my colleagues and I wrote about some tentative findings from our study of financial aid that suggested effect heterogeneity. Prior to the release, we extensively vetted those findings with colleagues, and ran at least five different sensitivity analyses.  After publication of the working paper, which we were careful to describe as "in progress," we sought even more feedback and advice-- and got a crash course in the enormous difficulty in disentangling effect heterogeneity from heterogeneous treatments. Truth is, the work is still ongoing.  And that's an incredibly important and valuable part of the research process, and one we all should wish and allow for-- it makes the work better.

So, here's to hoping that this is what will happen next for this voucher study.  Instead of rolling full steam ahead thinking vouchers will magically boost college attendance for black students everywhere, let's support the authors working through all potential alternative explanations for these odd results, and then replicating their experiment.  Again, my own experience suggests replication is critical, revealing the processes and contexts under which effects occur and are more reliable.  We should all demand it, especially from high-profile studies like these.

You have read this article Chingos / college / experiment / NYC / Peterson / vouchers with the title vouchers. You can bookmark this page URL http://apt3e.blogspot.com/2012/08/vouchers-and-college-attendance.html. Thanks!