Live By The Sword, Die By The Sword Redux

A USA Today investigation calls into question "dramatic" improvements in student test scores in select District of Columbia schools due to an "abnormal pattern" of erasures. This occurred during Michelle Rhee's tenure as DC schools chancellor.

Among the 96 DC schools that were flagged for wrong-to-right erasures by the city's testing contractor in 2008 "were eight of the 10 campuses where Rhee handed out so-called TEAM awards 'to recognize, reward and retain high-performing educators and support staff'.... Rhee bestowed more than $1.5 million in bonuses on principals, teachers and support staff on the basis of big jumps in 2007 and 2008 test scores.

In 2008, to her credit, then-DC state superintendent (now Rhode Island education commissioner) Deborah Gist recommended that large test score gains in certain schools be investigated, but as USA Today reported, "top D.C. public school officials balked and the recommendation was dropped."

Such allegations and instances of cheating are not unique to Washington DC of course. In 2010, a New York Times article chronicled erasures in Houston and noted investigations in Georgia (including a criminal probe in Atlanta), Indiana, Massachusetts, Nevada, and Virginia.

This latest development, however, adds a new wrinkle to my 2009 post, "Live By The Sword, Die By The Sword."

Michelle Rhee and other education reform advocates have publicly argued that student performance as measured by test scores is basically the be all and end all....

Student learning, school leadership and teaching cannot be measured and judged good or bad based on a single set of test scores. Test scores must be part of the consideration -- and supporting systems such as accountability, compensation and evaluation must be informed by such data -- but they should not single-handedly define success or failure.

When such huge stakes are placed on a single metric, it raises the likelihood of monkey business. Although it is highly likely this is what occurred in DC, a former employee of DC Public Schools (who tweets as @EduEscritora) makes several smart observations on her blog:

[T]he fact that the number of flagged schools decreased so precipitously from 2008 to 2009 is encouraging, even if we don’t know why that happened.

The decreasing number of schools also doesn’t support the claim that the pay-for-performance system now in place under IMPACT has resulted in cheating; 2010 was the first year that IMPACT existed, and that had the fewest number of flagged schools out of the three years in the study and the fewest number of schools with over 50% of the classrooms flagged – only two!

The problem for an advocate like Michelle Rhee is that she has chosen to largely define success based on a single metric: the test score. If many of these DC test-score gains turn out to be illusory and succumb to what some are calling the "Erase To The Top" scandal, it may spell further trouble for Rhee as a spokesperson for the school reform movement. (Rhee has claimed the largest NAEP score gains in the nation under her leadership, although other analyses have shown that increases began and were larger under Rhee's predecessors.) Her credibility already has been questioned by some as a result of alleged embellishments on her resume about her own teaching record. Without credibility, it is impossible to sell one's wares to anyone but true believers.

From a PR standpoint, this erasure story would seem to call for a measured response that carefully chronicles whatever steps, if any, were taken by DCPS at the time to address the unusual frequency of erasures. Instead, through a spokesperson, Michelle Rhee chose to 'shoot the messenger,' bombastically placing USA Today among the "enemies of school reform." [UPDATE: From the Washington Post's Jay Mathews: "Rhee calls her remarks on test erasures 'stupid'"]

Given Rhee's rhetoric, her policies in DC, and her current focus as head of StudentsFirst (which increasingly appears to be working solely with Republican governors and legislators at the state level), Michelle Rhee has largely pinned her credibility to the test score. If she had chosen to sit on a stool with more than a single leg, she might be sitting more comfortably right now and might not be engaged in a such a precarious and delicate balancing act. No doubt by taking on teacher tenure, she would have made enemies no matter what else she said or did. However, if she touted a more nuanced view of school improvement and student success and didn't poo-poo collaboration, she might not face a growing anti-Rhee cottage industry and her new organization might have had a chance to be a true non-partisan force in education reform.

You have read this article accountability / Michelle Rhee / test scores / USA Today / Washington DC with the title test scores. You can bookmark this page URL http://apt3e.blogspot.com/2011/03/live-by-sword-die-by-sword-redux.html. Thanks!

Tuskegee and the Obama Effect

By now you've all heard the fascinating news of a study (New York Times: "Study Sees An Obama Effect As Lifting Black Test-Takers") that seems to demonstrate an Obama effect on the black/white gap in test scores. In short, a team led by a Vanderbilt University researcher administered a series of 20 questions (drawn from verbal section of the GRE) shortly before Obama's nomination and again after his acceptance and then again after the election. Black performance on the test improved after Obama's acceptance, and rendered the black/white gap in test performance nonsignificant.

I'm the first to admit the potential for an Obama effect. Every time I hear him speak I think of the power of a role model, and dream of possible studies that could uncover such an effect.

But in this case, I'm not so sure what's being captured is an effect of Obama on the confidence of black students in their academic performance. Here's why:

(1) The students taking the test at each administration were different students. If the same kids took the test repeatedly, obviously we'd expect their scores to increase.

(2) According to the lead researcher, in a personal communique with me, while the pool of potential participants was constructed at time 1, the actual sample at each time was based on volunteers offered a monetary incentive to participate (what size incentive? I don't know).

There are more critical pieces of information missing as well:

(a) Whether the reasons for participation vs. non-participation differed by race, and are correlated with test-taking ability.

and

(b) Whether the rates of participation were similar for both racial groups.

What we do know is that ever since the Tuskegee Study of Untreated Syphilis (TSUS), African Americans are less likely than Whites to volunteer for participation in research. Given the known gaps in achievement, if they knew anything about what the study required they may've also simply lacked the confidence to participate. This is completely understandable. The question is, could it influence the findings in this study? Are there other plausible explanations for the change in test scores observed in the study?

Yes. Let me suggest just a few.

(1) A disproportionate effect of the economy on black's financial status. The study took place during a year of steady decline in the economic standing of many Americans. Is it possible that the money offered for participation wasn't enough to offset the concerns of higher-achieving black students about research (or to offset the opportunity costs associated with participation)? But that by time 2, the money was simply worth more (e.g. more effective as an incentive) and induced greater participation of black students? I'm positing that during the period whites were both less affected by changes in the economy and overall less averse to volunteering to take a test.

(2) An effect of Obama on black's trust in society, including researchers. So at time 1 the black students in the pool are generally more suspicious and only the lower-achievers are affected by the monetary incentive enough to overcome that suspicion and take the test. At time 2, they're feeling more goodwill towards the world, and higher-achieving black students are willing to participate.

(3) Maybe higher-achieving black students, when asked twice to do a study, tend to do it? I don't know if nonrespondents at time 1 were asked again.

These are just three ideas about how sample selection could bias these results. I have many more. What about the gender composition of the samples? ( Black men have lower test scores on average and are generally less likely to participate in studies. )

I want to quantify the good feelings we're all having in the post-Bushie world too. I get the motivation. But I don't think we should get too carried with feel-good stories on studies that have not yet undergone peer review.

You have read this article achievement gap / Barack Obama / New York Times / Obama effect / test scores with the title test scores. You can bookmark this page URL http://apt3e.blogspot.com/2009/01/tuskegee-and-obama-effect.html. Thanks!