A Little Information Could Go A Long Way

THIS GUEST POST COMES FROM ROBERT KELCHEN, DOCTORAL CANDIDATE IN EDUCATIONAL POLICY STUDIES AT UW-MADISON.

In a new report, Filling in the Blanks: How Information Can Affect Choice in Higher Education, Andrew Kelly and Mark Schneider of the American Enterprise Institute examine the role that information can play in the college choice process. One thousand parents in five states were asked which of two similar colleges they would recommend to their high school-age child. Half of the parents were given information about the colleges’ six-year graduation rates, while half were not. The researchers found that parents who were provided information about graduation rates were fifteen percentage points more likely to recommend the college with the higher graduation rate to their child, with larger differentials for parents who reported having less information about colleges and who had lower levels of education.

The intervention shows the importance of providing salient information to the parents of high school students. However, because parents in the study were making a theoretical decision instead of an actual decision that would affect their child, they had less of an incentive to think as carefully about their choice. This might result in effects that are larger than in real life, especially where parents have evenmore information about the two colleges being compared. A logical next step would be to repeat this experiment with high school students to see if the results significantly differ. Encouraging or requiring colleges to publicize their graduation rates may lead parents and students to choose colleges at which the student is more likely to graduate, as they take this information into account. In any case, even a small effect of additional information can make this low-cost intervention sound public policy.

You have read this article American Enterprise Institute / college / college entry / graduation rate / higher education with the title American Enterprise Institute. You can bookmark this page URL http://apt3e.blogspot.com/2011/01/a-little-information-could-go-long-way.html. Thanks!

Abandon All Hope (For Reform) Ye Who Enter Here!

At first glance, one might dismiss a recent policy brief authored by a former Bush Administration official as a partisan diatribe against the American Recovery and Reinvestment Act (ARRA) and the Obama Administration. After all, a chief conclusion of the brief authored for the American Enterprise Institute by Andy Smarick (former Deputy Assistant Secretary in the Spellings-era Education Department and in 'W's White House with the Domestic Policy Council), is: "It appears all but certain that the ARRA’s $75 billion in formula-based education programs are a lost cause for education reform. These funds have been used almost exclusively to fill budget holes, and cash-strapped states and districts will likely use what remains of these funds for similar, reform-averse purposes."

Abandon all hope (for reform) ye who enter here!

That quoted summary language in the paper *is* perhaps a bit over the top. A "lost cause"? Really? And that's certainly been the takeaway of some blog accounts of this paper (such as this). But that's not really what Smarick is saying nor is it the most important part of this AEI brief. And, as much as he is making that point, his 'lacking in reform' criticism is directed more at the 50 states than at the federal government.

Economic stimulus and a minimization of a short-term funding cliff were among the main aims of ARRA and its education-focused formula dollars. I don't think anyone seriously expected differently. If you read the ARRA web page, it largely spells this out. Now, the Education Department did envision that State Fiscal Stabilization Funding would be used to promote reform as well, and despite an initial look by the GAO, some dollars may accomplish reform, but how on earth could there yet be any real evidence of reform let alone impact when the 2009-10 school year has just begun in most places?!? In addition, as Smarick notes, the economic downturn and its effect on state budgets was far worse than anticipated at the time that ARRA was enacted in early 2009, which lessened the likelihood of these dollars doing anything less than filling holes.

Smarick's take on the competitive aspects of ARRA -- the Race to the Top and the Investing in Innovation (I3) funds -- is generally fair and balanced. He raises important questions about the general risks to any reform push, and specifically to ARRA. Smarick identifies several factors that may reduce the likelihood that competitive dollars will further education reform: on-going state budgetary challenges, resistance to specific reform components, and lack of faithful and vigorous implementation. He warns of "Trojan horse" applications where states will seek the money, but won't use it for reform. Of course, unmentioned are a whole host of other potential roadblocks, such as resistance from school districts, lack of buy-in from teachers and school administrators, lack of capacity to implement reforms, consultants and subcontractors who can't deliver promised expertise or technical assistance, data systems that cannot accurately match student and teacher data, etc.

Read the brief. Or check out a summary at Flypaper.

You have read this article American Enterprise Institute / Andy Smarick / ARRA / GAO / Race To The Top / stimulus / U.S. Department of Education with the title American Enterprise Institute. You can bookmark this page URL http://apt3e.blogspot.com/2009/09/abandon-all-hope-for-reform-ye-who.html. Thanks!

Sorting, Selection, and Success

Cross-posted from Brainstorm, over at the Chronicle of Higher Education.

The latest report from the American Enterprise Institute, Diplomas and Dropouts, hits one nail on the head: plenty of students starting college do not finish a degree. Access does not equate with success, and partly as a result, U.S. higher education is perpetuating a lot of inequality.

What do we do about this? The authors identify a key fact: “analysis of graduation rates reveals wide variance among institutions that have similar admissions standards and admit students with similar track records and test scores.” They interpret this to mean that “while student motivation, intent, and ability matter greatly when it comes to college completion, our analysis suggests that the practices of higher education institutions matter, too.”

This is a pretty common argument made by many policy institutes and advocacy organizations, including but not limited to the Education Trust and the Education Sector. I understand their goal—to make sure that colleges and universities can’t hide behind the excuse of “student deficits” in explaining low graduation rates, and instead get focused on the things they can do something about. In some ways that mirrors efforts over the last fifty years to focus on “school effects” in k-12 education —witness the continuing discussion of class size and teacher quality despite evidence that overall variation in student achievement is much more attributable to within-school differences in student characteristics than to between-school differences (school characteristics). Like many others, I read those findings to say that if we really want to make progress in educational outcomes, we must address significant social problems (e.g. poverty, segregation) as well as educational practices. Don’t misinterpret me- it’s not that I think teachers don’t matter. It’s simply a matter of degree—where and how can we make the biggest difference for kids, and under what circumstances.

Unlike k-12, access in higher education isn’t universal and competitive admissions processes and pricing structures result in lots of sorting of kids into colleges and universities. As a result, they differ tremendously in the students they serve. In turn, as the AEI report admits, this necessarily shapes their outcomes.

The problem is, all this sorting (selection bias) has to be properly accounted for if you want to isolate the contributions that colleges make to graduation rates. (I’ll qualify that briefly to add that the role college enrollment management —tuition setting, financial aid, and admissions— plays in the sorting process is quite important, and is under colleges’ control.) But if you want to isolate institutional practices that ought to be adopted, you first have to get your statistical models right.

Unfortunately, I don’t think the AEI authors have done that. To be sure, they try to be cautious, pointing out colleges that look “similar” but have extremely different graduation rates (rather than modestly different ones). But how they reached “similarity” leaves a lot to be desired. It seems to rest entirely on level of selectivity and geographic region. Their methods don’t begin to approach the gold standard tools needed to figure out what works (say, a good quasi-experimental design). Important student-level characteristics (socioeconomic background, high school preparation, need for remediation, etc) aren’t taken into account. Nor are many key school-level characteristics (e.g. resource levels and allocations). In sum, we are left with no empirical evidence that the numerous other plausible explanations for the findings have even been explored.

I’m not surprised by this, but have to admit that I’m a bit bummed. Yes, I “get” that AEI and places like it aren’t research universities. Folks don’t want to spend long periods of time conducting highly involved quantitative research before starting to talk policy and practice. But I don’t see how this approach is moving the ball forward—sure it gets peoples’ attention, but it’s not compelling to the educated reader—the one who votes and takes action to change the system. Moreover, it doesn’t get us any closer to the right answers, or provide any confidence that if we follow the recommendations we can expect real change.

There have been solid academic studies of the causes for variation in college graduation rates (here’s one example). They struggle with how hard it is to deal with the many differences among students and colleges that are not recorded – and thus not detectable—in national datasets. If we want better answers, we need to start by investing in better data and better studies. In the meantime, I think skipping the step of sifting and winnowing for the most accurate answers is inadvisable. Though, sadly, unsurprising…

You have read this article American Enterprise Institute / college degree / graduation rate / higher education / research with the title American Enterprise Institute. You can bookmark this page URL http://apt3e.blogspot.com/2009/06/sorting-selection-and-success.html. Thanks!

School Vouchers Are No Silver Bullet

Here is a compelling article by the American Enterprise Institute's Rick Hess about school choice in light of Milwaukee's experience with vouchers over the last two decades. His analysis is not as rosy as you might initially expect from a center/right researcher. But he does continue to see the potential in market-based reforms as long as they are envisioned more expansively and are accompanied with a focus on program growth, provider quality, professionalism, innovation, and accountability.

Many of Hess's proposed elements are exactly the kinds of reform that the Milwaukee voucher community fought tooth and nail over the years -- a fight led by politicized organizations such as School Choice Wisconsin, the Alliance for School Choice, and Advocates for School Choice. All of these organizations are interconnected and are bankrolled by national right-wing foundations, including Milwaukee's own Bradley Foundation. These groups have seemed to care little about the quality of education provided to Milwaukee's largely low-income and minority students -- but have willingly used these kids as political props at pro-voucher rallies during the school day -- and have made their sole focus the preservation of an unregulated and unfettered school voucher system in the city of Milwaukee. That changed a little bit a few years ago when media stories about graft, incompetence, and the utter absence of teaching and learning emerged at a number of Milwaukee voucher schools.

As education policy advisor to Wisconsin Governor Jim Doyle from 2004 to 2006, I was involved in some of these skirmishes with the voucher leadership. Doyle was always clearly on the side of public education and generally steered a wise course through choppy political waters on this issue. However, I continue to believe that a 2006 compromise [2005 Wisconsin Act 125 summary] that injected some of the first ever accountability and quality control into Milwaukee's voucher program still gave away too much. As examples, I'd offer up the evaluation currently being conducted by the privately-funded, voucher-friendly School Choice Demonstration Project, the trouble that state officials are having in getting data from that entity, and the lack of a requirement that voucher schools (funded with more than $100 million in state monies) administer a standardized assessment that would provide parents, policymakers and researchers with comparable student achievement data.

Hopefully, moving forward, all involved will take some of Hess's recommendations under advisement. For now, the Milwaukee voucher program lives on in its current form.

Here are a few excerpts from Hess's article:

Nearly two decades have passed since the enactment of the landmark Milwaukee Parental Choice Program by the Wisconsin legislature. The program and its many supporters had hoped this experiment in school choice would lead the way in transforming American schools. But it is by now clear that aggressive reforms to bring market principles to American education have failed to live up to their billing.
...
Today, the Milwaukee voucher program enrolls nearly 20,000 students in more than 100 schools, yet this growing marketplace has yielded little innovation or excellence. The Milwaukee Journal Sentinel recently described 10 percent of voucher schools as having “alarming deficiencies.” These include Alex’s Academics of Excellence, which was launched by a convicted rapist, and the Mandella School of Science and Math, whose director overreported its voucher enrollment and used the funds to purchase two Mercedes. Veteran Journal Sentinel writer Alan Borsuk has opined, “[Milwaukee Parental Choice Program] has preserved the status quo in terms of schooling options in the city more than it has offered a range of new, innovative, or distinctive schools.”
...
We should have no difficulty conceding Milwaukee’s disappointing record while remaining coolly confident that sensible K–12 market reforms have the potential to boost productivity, spur purposive innovation, provide more nuanced accountability, and make the sector a magnet for talent. Failures to dates should not be read as indictments of market reform but of the notion that “parental choice” programs represent a coherent approach to improving our schools. Reaching that goal will require approaching educational deregulation with an agenda much broader than simply increasing parental choice.
...
The lessons are increasingly clear. If school choice is to enjoy a brighter future than wave upon wave of supposed school reforms past, it is time for reformers to fight not just for choice but for good choices.

You have read this article American Enterprise Institute / Jim Doyle / Milwaukee Journal Sentinel / Milwaukee Parental Choice Program / Rick Hess / school choice / voucher / Wisconsin with the title American Enterprise Institute. You can bookmark this page URL http://apt3e.blogspot.com/2008/09/school-vouchers-are-no-silver-bullet.html. Thanks!