A Little Information Could Go A Long Way

THIS GUEST POST COMES FROM ROBERT KELCHEN, DOCTORAL CANDIDATE IN EDUCATIONAL POLICY STUDIES AT UW-MADISON.

In a new report, Filling in the Blanks: How Information Can Affect Choice in Higher Education, Andrew Kelly and Mark Schneider of the American Enterprise Institute examine the role that information can play in the college choice process. One thousand parents in five states were asked which of two similar colleges they would recommend to their high school-age child. Half of the parents were given information about the colleges’ six-year graduation rates, while half were not. The researchers found that parents who were provided information about graduation rates were fifteen percentage points more likely to recommend the college with the higher graduation rate to their child, with larger differentials for parents who reported having less information about colleges and who had lower levels of education.

The intervention shows the importance of providing salient information to the parents of high school students. However, because parents in the study were making a theoretical decision instead of an actual decision that would affect their child, they had less of an incentive to think as carefully about their choice. This might result in effects that are larger than in real life, especially where parents have evenmore information about the two colleges being compared. A logical next step would be to repeat this experiment with high school students to see if the results significantly differ. Encouraging or requiring colleges to publicize their graduation rates may lead parents and students to choose colleges at which the student is more likely to graduate, as they take this information into account. In any case, even a small effect of additional information can make this low-cost intervention sound public policy.

You have read this article American Enterprise Institute / college / college entry / graduation rate / higher education with the title graduation rate. You can bookmark this page URL http://apt3e.blogspot.com/2011/01/a-little-information-could-go-long-way.html. Thanks!

Spin Cycle

Education Next apparently has provided a platform for school choice advocate George Mitchell to shill for voucher schools outside of the state of Wisconsin. Here is his latest spin on a study that shows the high school graduation rate to be 12 points higher in seven Milwaukee voucher schools compared with 23 Milwaukee public high schools.

The Milwaukee Journal Sentinel story by Erin Richards provides the crucial quote regarding causation from the study's author, John Robert Warren, a sociology professor at the University of Minnesota:

"We still don't know whether it's going to the voucher school that causes you to be more likely to graduate, or if it's something about the kinds of families that send their kids to voucher schools would make them more likely to graduate," he said.

Then there's the whole question of which and how the voucher and public high schools were chosen for purposes of comparison. More questions than answers. Unlike Mitchell, I neither see this report as providing "another piece of evidence suggesting that urban students benefit when afforded more educational options," nor "new data" to encourage President Obama and Education Secretary Duncan to take "a second look at the power of parent choice."

The study was funded by the voucher-advocacy group School Choice Wisconsin, run by Mitchell's wife, Susan. The Mitchells have split from national school choice leader Howard Fuller who is devoting his current efforts to furthering accountability and quality in the Milwaukee Parental Choice Program.

After the spin cycle, be sure to rinse.

For past perspective on Voucher Inc, please visit here.

You have read this article graduation rate / high school / Milwaukee / research / school choice / voucher / Wisconsin with the title graduation rate. You can bookmark this page URL http://apt3e.blogspot.com/2010/02/spin-cycle.html. Thanks!

Premature Conclusions: More Money, No More Grads?

Some members of the media are covering the release of a new Canadian study, associated with the Educational Policy Institute, that examines the effects of a financial aid program on college-going and completion among low-income students. Researchers at the Measuring the Effectiveness of Study Aid Project tried to isolate those effects by examining what happened following a change in student aid policy in Quebec that increase aid eligibility and decreased reliance on loans. By comparing student outcomes both before and after the policy change, and comparing the outcomes of similar student in Quebec to those in other provinces (where such reforms did not occur), analysts attempted to establish a causal effect of aid.

They conclude that the policy affected access (increasing overall enrollment among students from families making less than $20K per year by 4-6 percentage points), and persistence (increasing retention rates by 6 percentage points) but did not affect graduation rates--at least within the 4-year window of time during which graduation was measured.

While noting that the null findings may stem from that short period of observation, the researcher still goes on record with this conclusion: "These results therefore cast doubt on the efficacy of this reform in particular, and of needs-based grants in general, to improve graduation rates."The headline over at Inside Higher Ed reads "More Money Doesn't Equal More Graduates."

This is a distinctly premature and irresponsible conclusion. First, as one of my graduate assistants James Benson pointed out, "if the percentage of college-eligible students that enrolled in college increased by 5 percent, and the persistence and graduation rates remained entirely static, then the program produced a net gain in the proportion of young adults completing semesters and degrees."

Furthermore, there are many reasons why an effect might not be estimated properly in this study. As my colleagues Doug Harris, Phil Trostel, and I explained in a recent paper, a simple correlation between aid receipt and college success is likely to be negative because students from low-income families, in the absence of aid, are for a variety of reasons less likely to succeed. Unless researchers can convincingly account for all of those reasons – and we argue that very few do – the estimated effects of aid are likely to look smaller than they really are. This study is not very convincing and really doesn't move far beyond a correlation, for many reasons. For example, as another graduate assistant, Robert Kelchen, indicates:

1. The comparison groups (Quebec vs. other provinces) have very different rates of financial aid take-up prior to the reform. This calls the validity of the comparison into question. It's also too bad the researcher didn't see fit to post his tables on the website, since we cannot see whether the differences post-treatment are significant.

2. Quebec saw increases in the enrollment rates of high-income students following the reform, in addition to increases in the enrollment rates of low-income students. If financial aid was the real driver, it shouldn't have affected the (ineligible) high-income students.

These are but a few examples-- if a full research paper (such as would be submitted for academic review) was available, I bet we'd have more concerns.

This is a case of the press jumping the gun and running with a story, and a headline, not supported by the empirical work done by the researchers. We're in a recession, and aid programs cost a lot of money. We do need to know if they work, and in particular if they are cost-effective. But the estimation of impacts should be done more carefully, and results discussed in a much more responsible manner. Sexy but un-informed headlines will do little good-- perhaps even casting a shadow on an effective program, reducing its ability to maintain funding. All of us studying financial aid have an obligation to do much, much better.

You have read this article financial aid / graduation rate / Quebec with the title graduation rate. You can bookmark this page URL http://apt3e.blogspot.com/2009/09/premature-conclusions-more-money-no.html. Thanks!

Where Have You Been?

A spate of recent articles, including those covering Bill Bowen and Mike McPherson's new book (which I promise to review just as soon as my copy arrives), have left me a bit perplexed-- wondering aloud "where have you all been?" The punchline each time is that a fair proportion of adults starting college are not finishing. Yes, and duh. This is not new, and if it's news well I guess it's only because we've deliberately kept our heads in the sand.

But there's no way that folks like New York Times reporter David Leonhardt have been deliberately oblivious, and yet he's writing about low college completion rates as if they've just been unearthed. In a recent blog post, Kevin Carey implied the same-- just as he did in a recent American Enterprise Institute report. But this has been a prominent topic of discussion for years--maybe a decade plus! Just look at Kevin's own 2004 report A Matter of Degrees (which received plenty of media coverage), or the Spellings Commission report, or Claudia Goldin and Larry Katz's book. I know I could go back several more years and find plenty more evidence.

I think it's one thing to imply something is new when it isn't (because again, maybe you just didn't know, or you feel the issue still is widely known enough and want to beat the drum more), and it's another thing entirely to claim that policymakers still aren't paying attention. In Leonhardt's case, he's simply wrong when he says the current Administration isn't focused on college completion. Um, how about that $2.5 billion Access and Completion Fund, part of Obama's original budget proposal? What about the performance (outcomes)-based components of the new community college monies contained in HR 3221? Foundations like Lumina and Gates have been beating this drum for years, and those in the Administration are well aware. No one in DC is saying institutions should continue to be judged solely based on enrollment (even enrollment of disadvantaged groups). There is plenty of ado about completion rates. The question is now, what exactly are the best solutions? That's a debate that needs to be richer and more visible, since the answers are far from clear-- and we'd be terribly wrong to simply resort to NCLB-style responses that remind me of my toddler: "Institutions bad. Do wrong. I punish you and you do better. Now." Let's direct our energies toward really identifying the sources of the problems, and developing a sense of how reforms can be most effective. When I get a chance to read the new Bowen and McPherson book, I'm hoping I come away with new ideas on how to do that.

You have read this article Barack Obama / college access / college completion / graduation rate / New York Times / President with the title graduation rate. You can bookmark this page URL http://apt3e.blogspot.com/2009/09/where-have-you-been.html. Thanks!

Sorting, Selection, and Success

Cross-posted from Brainstorm, over at the Chronicle of Higher Education.

The latest report from the American Enterprise Institute, Diplomas and Dropouts, hits one nail on the head: plenty of students starting college do not finish a degree. Access does not equate with success, and partly as a result, U.S. higher education is perpetuating a lot of inequality.

What do we do about this? The authors identify a key fact: “analysis of graduation rates reveals wide variance among institutions that have similar admissions standards and admit students with similar track records and test scores.” They interpret this to mean that “while student motivation, intent, and ability matter greatly when it comes to college completion, our analysis suggests that the practices of higher education institutions matter, too.”

This is a pretty common argument made by many policy institutes and advocacy organizations, including but not limited to the Education Trust and the Education Sector. I understand their goal—to make sure that colleges and universities can’t hide behind the excuse of “student deficits” in explaining low graduation rates, and instead get focused on the things they can do something about. In some ways that mirrors efforts over the last fifty years to focus on “school effects” in k-12 education —witness the continuing discussion of class size and teacher quality despite evidence that overall variation in student achievement is much more attributable to within-school differences in student characteristics than to between-school differences (school characteristics). Like many others, I read those findings to say that if we really want to make progress in educational outcomes, we must address significant social problems (e.g. poverty, segregation) as well as educational practices. Don’t misinterpret me- it’s not that I think teachers don’t matter. It’s simply a matter of degree—where and how can we make the biggest difference for kids, and under what circumstances.

Unlike k-12, access in higher education isn’t universal and competitive admissions processes and pricing structures result in lots of sorting of kids into colleges and universities. As a result, they differ tremendously in the students they serve. In turn, as the AEI report admits, this necessarily shapes their outcomes.

The problem is, all this sorting (selection bias) has to be properly accounted for if you want to isolate the contributions that colleges make to graduation rates. (I’ll qualify that briefly to add that the role college enrollment management —tuition setting, financial aid, and admissions— plays in the sorting process is quite important, and is under colleges’ control.) But if you want to isolate institutional practices that ought to be adopted, you first have to get your statistical models right.

Unfortunately, I don’t think the AEI authors have done that. To be sure, they try to be cautious, pointing out colleges that look “similar” but have extremely different graduation rates (rather than modestly different ones). But how they reached “similarity” leaves a lot to be desired. It seems to rest entirely on level of selectivity and geographic region. Their methods don’t begin to approach the gold standard tools needed to figure out what works (say, a good quasi-experimental design). Important student-level characteristics (socioeconomic background, high school preparation, need for remediation, etc) aren’t taken into account. Nor are many key school-level characteristics (e.g. resource levels and allocations). In sum, we are left with no empirical evidence that the numerous other plausible explanations for the findings have even been explored.

I’m not surprised by this, but have to admit that I’m a bit bummed. Yes, I “get” that AEI and places like it aren’t research universities. Folks don’t want to spend long periods of time conducting highly involved quantitative research before starting to talk policy and practice. But I don’t see how this approach is moving the ball forward—sure it gets peoples’ attention, but it’s not compelling to the educated reader—the one who votes and takes action to change the system. Moreover, it doesn’t get us any closer to the right answers, or provide any confidence that if we follow the recommendations we can expect real change.

There have been solid academic studies of the causes for variation in college graduation rates (here’s one example). They struggle with how hard it is to deal with the many differences among students and colleges that are not recorded – and thus not detectable—in national datasets. If we want better answers, we need to start by investing in better data and better studies. In the meantime, I think skipping the step of sifting and winnowing for the most accurate answers is inadvisable. Though, sadly, unsurprising…

You have read this article American Enterprise Institute / college degree / graduation rate / higher education / research with the title graduation rate. You can bookmark this page URL http://apt3e.blogspot.com/2009/06/sorting-selection-and-success.html. Thanks!

Thumb On The Scale: High School Graduation Rates

Saturday’s editorial in the Wisconsin State Journal – a daily newspaper serving Madison (for you non-Badgers) – is merely the latest calling for consistency and transparency in the public reporting of high school graduation and dropout rates. In this version, Wisconsin is the whipping boy for inflating its graduation rate. The Wisconsin Department of Public Instruction says it's 91.2%, while the U.S. Department of Education says it's 85.8%, and the Editorial Projects in Education Research Center says it's 77.3%.

Back in 2005, under the leadership of then-National Governors Association (NGA) Chairman and Virginia Governor Mark Warner, the nation’s Governors signed onto a Compact that called for a consistent measure for calculating high school graduation rates. Specifically, the Graduation Counts Compact calls for “a standard, four-year adjusted cohort graduation rate. States agree to calculate the graduation rate by dividing the number of on-time graduates in a given year by the number of first-time entering ninth graders four years earlier.” Joining the Governors in supporting this approach were national organizations including the American Federation of Teachers, the National Education Association, the Alliance for Excellent Education, the Council of Chief State School Officers, and the Education Trust. Despite that seemingly high level of consensus, a number of states continue to conduct business as usual.

There are some honest reasons for this, such as the inability of state data systems to track students who transfer schools, districts, and even states and separate them from those who drop out of school completely. Some states are continuing to overcome this obstacle. In addition, many Governors don’t control their state department of education and can’t snap their fingers and make this happen. Further, more than a dozen of the Governors who signed this compact three years ago are no longer in office. However, the NGA is working to ensure that governors who have taken office since 2005 agree to the compact’s terms.

There’s most certainly some bureaucratic heel-dragging going on and politics at foot. In fact, two states--North Dakota and South Dakota--have dropped out of the compact all together.

Nonetheless, some progress has occurred. According to a 2006 NGA report, 13 states reported their graduation rate according to the Compact formula; two states used the Compact formula with local cohort data; and one state reported a sophisticated estimate consistent with the recommendations. By 2010, 39 states planned to report a graduation rate using the Compact definition. Two states have even codified the Compact rate: Colorado did so through state board regulations and Maryland through legislation.

On April 1st of this year, U.S. Secretary of Education Margaret Spellings issued proposed regulations that would require all states and high schools to use the NGA-proposed high school graduation rate calculation by 2011. This action may impel recalcitrant states to get off the dime.

This issue clearly isn't liberal or conservative, Republican or Democrat. If we truly are going to put the interests of students first, we need to capture accurately the magnitude of this problem. Perhaps doing so would impel policymakers to take more seriously their duty to reform high schools and support the teachers and administrators who staff them. Currently, too many high school dropouts are forgotten within data reporting systems much as they are forgotten by the school systems that serve them poorly.

If we can’t honestly collect and report basic data about the education policy issues we’re trying to address, why even bother with data or research at all? Let’s manage and legislate based on whims and fancies—and blatantly false perceptions. At least we'll feel good.

-----------------------------------

For those of you craving more on this topic, check out this 2006 report from the NGA Center for Best Practices, this April 2008 policy brief from the Alliance for Excellent Education or this August 2007 brief from The Education Trust.

You have read this article graduation rate / high school / National Governors Association / Wisconsin with the title graduation rate. You can bookmark this page URL http://apt3e.blogspot.com/2008/05/thumb-on-scale-high-school-graduation.html. Thanks!

The Graduation Rate Gap: Minding What Matters?

One of the dirtiest secrets in higher education is that getting in--the focus of 18 years of hard work in some families--is no guarantee of college success. The sad truth is that at many public four-year colleges and universities fewer than half of entering freshmen complete a degree within six years. And in some cases those low average rates conceal incredible disparities based on race and family income.

Kevin Carey of Education Sector has done a remarkable job at turning the nation's attention to this often overlooked problem in recent years. He helped create College Results Online which enables the average Joe to spend hours of fun picking out individual schools and seeing just how low their graduation rates can go. (Seriously, go check it out-- it's FUN!)

Most recently Carey issued Graduation Rate Watch, a report that uses six years of federal data to look at changes over time in the black-white gap in college completion. His primary point: between 2002 and 2006 some schools made significant strides in closing their gaps, while others did not. And in fact, some schools had (and have) no gap at all. Therefore, Carey concludes, the answers are clear: too many colleges are indifferent to the plight of minority students, they know what they need to do, and we just need to push them to do it.

Kevin and I agree on most things. We agree that the feds and states are paying far too little attention to low rates of college success. We agree that disparities in degree completion rates are unacceptable. And we agree that institutions of higher education need to be a part of the solution But on one crucial point we disagree: what should be done.

Drawing on a case study of a single college (Florida State) and its lovely college-support program CARE, Kevin contends that the solution to closing the college achievement gap lies in "creating incentives for institutional leaders to act on the knowledge that already exists." His goal in making this kind of statement is to get educators to step up-- they know what needs to be done, and what remains is to take action.

But the researcher in me hesitates to endorse this plan. Here are a few reasons why:

1. There is little evidence that differences in students' college graduation rates are primarily or even substantially attributable to school-level factors. Go back to the Coleman report--his intelligent hypothesis was that inequities in schools due to segregation were driving black/white differences in educational outcomes. But instead he found that family background and other student-level characteristics drove far more of the variation in student achievement. Yes, some school-level factors matter--among them teacher quality--but empirical evidence indicated that changing schools themselves would not diminish educational inequality much.

Why would we think things would be different in higher education? The typical argument goes like this: We know that institutional choices and policies (financial aid, academics, advising, faculty, incentives, student engagement, etc) matter in terms of student learning, retention, and attainment. Given our diverse, decentralized higher education "system" there is substantial inter-institutional variation in the quality of those choices and policies. Therefore, "institutional effects" should be significant and real.

Well, they may be significant, they may be "real," but there is little hard data to say that they are large and should be the main focus of policy reform. I'm not saying that colleges and universities aren't different from one another in ways that matter, but because higher education is not compulsory and its gates are closely guarded by the gods of test scores and $$ the level of stratification in the college entry process is so intense that students attending different schools are very different from one another. There have been very few studies of institutional effects in higher ed that deal with that selection issue appropriately. I strongly suspect that if a good study were done (and given data constraints it'd be hard to do), we'd find that policymakers would get the biggest bang for their buck by working to reduce race, class and gender disparities in who goes to college and where they go.

2. Another important concern: There can be unintended consequences when you push colleges to close achievement gaps. Take this scenario: You've got 30 percentage point gap in college completion, with 40% of black students finishing a degree in 6 years, compared to 70% of whites. You want to close that gap. So you tell your admissions director to solve the problem. And he does: by being more selective in which black kids he admits. Admitting only very highly-prepared black students may mean a decline in the number of black kids on campus, but it's likely to increase the completion rate of those who are admitted. And if he coupled his efforts with a greater effort at targeted recruiting of minorities (including directing merit-based scholarships at them) , he might succeed in simultaneously increasing diversity and completion rates. Not a bad thing per se (perhaps) but definitely not what Carey has in mind...

3. There is not a shred of evidence that FSU's CARE program actually works to increase student retention or achievement. No evaluation that takes into account who they serve to start with, nada. Yes, they've got data indicating that CARE students start with lower entering test scores than other students and end up with higher retention rates, but NO--this doesn't automatically mean that success is due to the program. What if CARE students are simply more motivated than other students-- meaning that they were more likely to be retained from the start? We don't have any way to account for that in the absence of a good ol' random assignment trial where we assign students at random to participate in CARE. Now that would be worth doing!


Of course I understand that Kevin's goals are different from mine. Ultimately he operates in a world where being 51% sure that you're right when making policy recommendations is sufficient-- after all, if we don't advocate for changes in colleges, there might not be attention paid to higher ed at all (or dollars spent). But in my little research world, I need to be much more confident that the recommendation I'm making will be effective in creating the kind of change we aim for--AND, I want to focus on what will make the biggest difference. And there's simply little hard evidence to suggest that adopting a program like FSU's would help to close the black/white gaps in completion at other schools.

Why not--instead of "tinkering at the margins" with school reform--tackle the larger and more systemic problems like ending poverty and racial segregation? (Why is it that educators and educational policymakers have a hard time seeing what happens outside of schools as being part of their domain?) Why not make higher education compulsory, and in doing so greatly increase the motivation to make high school meaningful? Why not re-envision the entire system and create a p-16 structure which compels higher ed and secondary ed to take an interest in one another?

The last thing we need to do is fan the flames of fears about accountability, encouraging colleges and universities to turn even more inward, engaging in ego-think, and directing their attention at self-preservation rather than student success. We do want them to do more-- and they can be an important part of the process of change-- but they need direct their attention at the things that matter most.

You have read this article college / college completion / Education Sector / graduation rate / higher education / Kevin Carey with the title graduation rate. You can bookmark this page URL http://apt3e.blogspot.com/2008/04/the-graduation-rate-gap-minding-what.html. Thanks!