Baking Bread Without The Yeast

Among my son's favorite books are the ones in Richard Scarry's Busytown series. In What Do People Do All Day?, Able Baker Charlie puts too much yeast in the dough, resulting in a gigantic, explosive loaf of bread that the bakers (and Lowly Worm) need to eat their way out of.

The opposite problem -- a lack of yeast -- is present in Michelle Rhee's recent op-ed in Education Week. In it, she limits her call to "rethink" teaching policy to "how we assign, retain, evaluate, and pay educators" and to "teacher-layoff and teacher-tenure policies." (And she casts the issue of retention purely as one about so-called "last-in, first-out" employment policies rather than about school leadership, collaboration or working conditions.)

The utter absence of any focus or mention of teacher development either in this op-ed or in her organization's (StudentsFirst) expansive policy agenda leaves me wondering if Rhee believes that teachers are capable of learning and improving. If Rhee indeed does believe that new teacher induction and career-long professional development have value, then why does she consistently ignore it in her public statements and in her organization's strategic priorities? The alternative, of course, is a view that teachers are static beings, incapable of improvement. They are either born effective or ineffective. "Mr. Anderson's value-added score is an 18, thus he is an ineffective teacher and should be fired because his inability to teach cannot be ameliorated." We, of course, know this not to be the case. This alternative view also involves a strategy of simply trying to hire and fire our way to success. From research and international exemplars, I think most of us understand such a narrow approach to be ineffectual, albeit politically attractive in some quarters, especially among the Republican governors that Rhee is assisting exclusively.

High-quality development opportunities for teachers are like the yeast that helps the bread to rise. Comprehensive teacher induction has been shown to accelerate new teacher effectiveness and increase their students' learning. Likewise, personalized and purposeful professional development also can strengthen teaching skills and classroom impact.

It seems to me that a stated policy goal should be to ensure that as many as teachers as possible successfully pass educator evaluations being developed across the nation. Too many advocates such as Rhee appear to be eager to fire more teachers rather than make investments and restructure schools to maximize their effectiveness. A critical role for policy then would be to re-define teacher development in a way that raises the quality bar and invests public dollars in programs and approaches shown to have the desired impact on teaching and learning. Isn't that something we all can agree with?

Teachers are tremendously influential -- and we should do everything we can to unleash their full power. On teacher effectiveness, I'm unwilling to settle for half a loaf.

You have read this article Education / Education Week / induction / mentoring / Michelle Rhee / professional development / reform / StudentsFirst / teaching with the title Education Week. You can bookmark this page URL https://apt3e.blogspot.com/2012/01/baking-bread-without-yeast.html. Thanks!

Jumping to Conclusions

In a recent post on Education Week's Inside School Research blog, Debra Viadero offers a caution about President Obama's support for community colleges. Pointing to her recent article on community college research that indicated how much more we need to know about how best to improve completion rates in that sector, she questions whether the president would be wiser to place his bets on career colleges. She says that a recent study by the Educational Policy Institute (EPI) and an ongoing program of research by James Rosenbaum and colleagues support her contention that community colleges ought to take cues from career colleges.

In my opinion, this talented reporter is jumping to conclusions.

Yes, the graduation rates at two-year for-profit colleges exceed those at two-year public colleges. No one disputes that. That does not necessarily mean, however, that career colleges are outperforming community colleges, or that community colleges should take steps to become more like career colleges. The plausible alternative explanations for the differences in results are numerous. For example, the students attending the two types of colleges may differ in important yet unmeasured ways, ways that are associated with chances of graduation (what researchers refer to as selection bias). Is one group more economically or educationally advantaged? More motivated? More apt to have a family, nighttime work, or receive tuition support from an employer? It's also possible that the differences in graduation rates stem from constraints that community colleges face but career colleges do not-for example, inadequate resources or a lack of control over mission or governance. It's one thing to point to differences in practices between the community colleges and for-profit colleges, and another thing to attribute those differences to variation in the "will" or intentions of practitioners, rather than attribute them to under-funding and all that comes with it.

Establishing that community colleges have poorer graduation rates than career colleges for reasons they can and should do something about requires evidence that the two differ on one or more key aspects that is causally linked to college completion. Say we knew that smaller classes caused better student retention-and community colleges have larger classes than career colleges. We'd then be able to say, there's something community colleges ought to fix. But we don't have evidence that that's the case.

Instead, research simply establishes that (a) career and community colleges have different graduation rates and (b) career and community colleges (sometimes) employ different institutional practices. Rosenbaum and his colleagues have done a nice job, as Viadero notes, of documenting the latter-using qualitative methods, mostly at colleges in the Chicago area. But they have not shown that those practices cause observable differences in graduation rates. Moreover, while they've produced one paper indicating that differences in the student populations at career and community colleges do not appear to account for disparities in outcomes, that analysis is based solely on a limited set of observable characteristics-and therefore don't rule out the possibility that different levels of student motivation, for example, are really the culprit. Just think about how students get to college-many at career colleges are actively recruited (sometimes off their living room couches) while many at community colleges effectively wander in the door. Why would we think, then, that career and community colleges are serving the same kinds of people and producing different results?

There's another consideration Viadero neglects, and that's college costs. Students at career colleges leave with far more debt than students at community colleges. Data from the 2007-2008 National Postsecondary Student Aid Study reveal that 61% of community college students graduate with less than $10,000 in debt, compared to only 22% of students graduating from 2 year for-profits. In contrast, 19% of graduates from 2 year for-profits have $30,000 or more in student loans, compared to only 5% of community college graduates. Nearly all students (98%) finishing at 2 year for-profit colleges have taken on a loan, compared to just 38% of community college graduates. Is that a problem? Is it offset by higher rates of graduation? The answers are far from clear. Absent better ones we shouldn't be relying on evidence like EPI's---a study of career colleges' high graduation rates that was supported by the Imagine America Foundation, formerly the Career College Foundation, established in 1982 as the research, scholarship and training provider for the nation's career colleges. Full-text of that study wasn't even placed online for researchers to fully vet!

Community colleges have a long, rich history of serving this nation. Sure, there's room for improvement, but without more solid evidence of which changes are needed let's not jump to conclusions and tout the for-profits as a model to which they ought to aspire. We might end up in a bigger mess than we're already in.

POSTSCRIPT:

I have now obtained a copy of the full EPI report. My suspicions were correct: the authors use nothing more than simple descriptive comparisons of students' characteristics and degree completion rates (calculated using NCES's DAS system, likely without propering weighting) to support their causal claims about the "benefits" of attending community college. For example, they write "The report suggests that career colleges work harder to provide appropriate student services and support" but present no data on institutional services or effort expended, particularly any tied to student outcomes. Their final conclusion -- "statistically, not only do students attending career colleges perform as well as or better than many other students attending comparative public institutions, but they persist in and complete their education while typically being more economically, educationally and socially challenged than other students"-- is based on nothing more than comparisons of sample means (no regression, no nothing). C'mon folks, this ain't the kind of research any consumer ought to be taking seriously. Glad to see Kevin Carey agrees.

You have read this article Barack Obama / community college / Education Week / Educational Policy Institute / James Rosenbaum / President with the title Education Week. You can bookmark this page URL https://apt3e.blogspot.com/2010/01/jumping-to-conclusions.html. Thanks!

"No Effects" Studies

Education Week's got this article out about randomized trials producing "no effects." According to the article, these null findings are raising eyebrows and "prompting researchers, product developers, and other experts to question the design of the studies, whether the methodology they use is suited to the messy real world of education, and whether the projects are worth the cost, which has run as high as $14.4 million in the case of one such study."

Wow, is that ever a disappointing reaction. Here's why:

1. We should be psyched, not upset, that studies with null effects are being released. That is not always the case. Publication bias, anyone? I've often thought that studies demonstrating null effects need to be publicized even more widely than those that find positive or negative impacts. Too many places out there are at the behest of funders, and can't release null findings. Too many assistant professors don't get tenure because they "didn't find anything." Are you kidding me? If it's a current practice and you learn it doesn't produce any effects, either way, it needs to be out there. We should learn as much from null findings and "worst practices" as we do from "statistically significant" impacts and "best practices."

2. Saying that experimentation isn't suited to the "messy real world" is a cop out. It lumps many different kinds of experiments into one category-- the good, the bad, and the ugly. Field experiments, lab settings, cluster-randomized trials with volunteer districts, and student-level randomized experiments with participants selected via administrative data-- these are very different animals. Each approach has a differential potential for generalizable results (external validity) and varying levels of challenges to internal validity as well. I'll grant you, experiments that rely on volunteer samples probably can't help us much in education-- since in real life programs aren't applied to students, families or schools who volunteer--they apply to everyone. This is especially a problem when we try interventions to close achievement gaps-- African-Americans who volunteer for studies are very, very different from those who do not (Tuskegee anyone?).

3. Doing experiments well costs a LOT of money. Putting trials on tight budgets helps to ensure they aren't run well--PIs cannot build the kinds of relationships that promote treatment fidelity, cannot collect high-quality data, and cannot get inside the black box of mechanisms--and instead are stuck simply estimating average treatment effects. No drug works for everyone, and no drug works in the exact same way for everyone-- the medical community knows this, and uses larger samples to make identifying differential and heterogeneous effects possible. When is Education going to catch up?

4. One thing I do agree with this article on. The model IES is using needs some revisions. I heard William T. Grant president Bob Granger give a great talk at SREE recently, where he made the point that the usual 'try small things then scale them up' model isn't going anywhere fast. We need to know how current policies work as currently implemented-- at scale. Go after that, spend what's necessary to conduct experiments with higher internal AND external validity, and support researchers to do this who reject old models and try new things. I promise you, we'll get somewhere.

You have read this article Education Week / Institute for Education Sciences / research with the title Education Week. You can bookmark this page URL https://apt3e.blogspot.com/2009/03/effects-studies.html. Thanks!

Measurement Is Not Destiny

Stephen Sawchuk has written an excellent cover story ("Stimulus Bill Spurs Focus on Teachers") in this week's edition of Education Week. It discusses the federal stimulus legislation which directs states to abide by the equitable teacher distribution provisions in the No Child Left Behind Act -- as well as to improve teacher effectiveness -- in exchange for state stabilization funds and the opportunity to apply for competitive grants as part of Secretary Duncan's "Race To The Top Fund."

With regard to teacher effectiveness, there's just one little problem. There's no definition in federal law -- let alone in state laws -- about what that actually means. From Education Week:

Several states, and some districts, now endorse performance-based teacher evaluations that define good teaching, determine which teachers exhibit such practices, and identify those who fall short for assistance. Others are reorienting professional development toward sustained school-based approaches that researchers say are more likely to change teacher behavior and improve student achievement than “one shot” workshops.

Some efforts to improve teacher effectiveness have proved politically challenging. The federal Teacher Incentive Fund, a performance-pay program, has promoted interest in using test scores to estimate teacher effectiveness. That approach has generally not been favored by teachers’ unions. The tif program received an additional $200 million in the stimulus.

Additionally, a limited number of states have the ability to match teacher records to student data, and even those with the technical capacity have not always used their data to estimate teacher effectiveness. The unions fear such links could ultimately be used to establish punitive policies, and they have successfully lobbied legislators to curb the use of “teacher effect” data in some states. ("Growth Data for Teachers Under Review," Oct. 12, 2008.)

But the possibilities of “value added” are enticing to policymakers. Officials in Tennessee, the lone state that has incorporated teacher-effect data into personnel decisions, are awaiting new data that will reveal whether efforts to attract effective teachers to the most challenged schools have improved results, said Julie McCargar, the state director of federal programs.

This is a huge issue – and it will be interesting to see if the U.S. Department of Education focuses its regulatory definition and its expectations of states – like so many others – simply on measuring and identifying and perhaps rewarding effective teachers. The logical and more purposeful next step, of course, is to look at what behaviors, characteristics, or knowledge make certain educators more effective and then determine how to scale up approaches to initial training or on-going professional development programs to help make the vast majority of teacher candidates, beginning teachers and veteran teachers better. I have no insider knowledge about the Department's thinking around all this, but I’m always astonished at the wealth of policymakers, policy organizations, and foundations that never seem to get past square one on this topic.

Measurement is not destiny.

If all we do is use value-added metrics to determine who the best teachers are and pay them more money for being better, we will be sacrificing the quality of public education for a short-sighted reform. While more money might keep some effective educators from leaving a particular school or district, or from leaving the profession entirely, it won't do anything to make existing and future teachers a whit better.

The teacher effectiveness conversation must be about more than value-added measurement and performance pay, although it can certainly include those elements. It can't be simply about rewarding the good and getting rid of the bad. Fundamentally, it must be about a concerted human capital strategy to use existing knowledge as well as future data and research to strengthen teacher preparation, induction and professional development to improve the skills and abilities of all teachers. Hopefully, the Department's focus on teacher effectiveness will impel such an effort.

We can do better -- by learning from the best teachers and finding ways to replicate their success. Now, that would be effective.

You have read this article Education Week / federal / stimulus / teacher distribution / teacher effectiveness / teacher pay / teacher preparation / teacher quality / U.S. Department of Education / value added with the title Education Week. You can bookmark this page URL https://apt3e.blogspot.com/2009/03/measurement-is-not-destiny.html. Thanks!

Blagojevich, Day Two

Alyson Klein of Education Week's Campaign K-12 blog offers up a post reporting that the Illinois Education Association -- the state's NEA affiliate -- is calling for Illinois Governor Rod Blagojevich's resignation, following his arrest on corruption charges yesterday ('Illinois Governor Arrested on Corruption Charges'). Join the club.

Alyson's blog post underscores the education-related point I made in my post yesterday: that Blagojevich was decent with respect to providing funding for education, but not tackling vexing reform issues.

If Blagojevich is ousted from his post because of the allegations, it doesn't sound like education organizations in the state will miss him much.

"According to our members it would not be a huge blow," said Ben Schwarm, an associate executive director for the Illinois Association of School Boards. He said some districts had been angry with Blagojevich for his refusal to support a dedicated funding mechanism for education, although he did acknowledge that schools had gotten spending increases during the governor's tenure.

Ken Swanson, the president of the IEA, agreed that Blagojevich had been "a roadblock" to educators' push to get a designated revenue source for schools. The union endorsed neither the governor nor his opponent during Blagojevich's 2006 re-election bid.

Here's the IEA statement.

Oh, by the way, happy birthday, Rod!

You have read this article Education / Education Week / Illinois / Illinois Education Association / Rod Blagojevich with the title Education Week. You can bookmark this page URL https://apt3e.blogspot.com/2008/12/blagojevich-day-two.html. Thanks!

Federal Funding for Teacher Quality Innovation?

This is a follow-up to my post of two weeks ago about the use of Title II, Part A funding under NCLB. In these tight economic times, it is inevitable that the focus will move from spending more money on education to spending existing dollars more wisely. Currently, most school districts are not using these federal dollars in particularly innovative, let alone effective or impactful ways.

An article by Stephen Sawchuk ('Grants in NCLB to Aid Teaching Under Scrutiny') was published in this week's edition of Education Week. In part, it discusses the findings of a recent Education Sector report on this topic.

For those of you who aren't Ed Week subscribers and may not be able to access the story, here is a peek at the story:

The Teacher and Principal Training and Recruiting Fund—better known as Title II, Part A of NCLB—is the federal government’s second-largest K-12 investment, after the Title I grants for disadvantaged students. Ninety-five percent of the funds flow to school districts, and they come with few strings attached.

Although the fund has promoted some promising local practices, Title II, in general, “is not especially aligned with leading-edge [teacher-quality] efforts, and it’s the federal government’s big entry in this sweepstakes,” said Andrew J. Rotherham, the co-director of Education Sector, a Washington think tank, and the report’s author.

...

In his paper, Mr. Rotherham stakes out one conceptual approach that Mr. Obama and legislators could consider when they revise the program as part of the reauthorization of the NCLB law: to transform Title II into a fund for seeding innovations to the education human-capital continuum, and to disallow a handful of currently authorized activities, including class-size reduction.

...

Nationally representative U.S. Department of Education survey data show that districts in 2007-08 spent 6 percent of their Title II funds on professional-growth initiatives—such as mentoring programs or incentives for teachers to pursue certification from the National Board for Professional Teaching Standards—and 4 percent on recruitment, including performance-based pay and teacher loan-forgiveness programs.

More than three-quarters of districts’ Title II allocations subsidize professional development and smaller class sizes. In his paper, Mr. Rotherham deems those activities “low leverage” because they typically lack quality-control mechanisms and reinforce traditional human-capital structures, rather than altering them.

You have read this article Andy Rotherham / Education Sector / Education Week / ESEA / No Child Left Behind / teacher effectiveness / teacher quality / Title II with the title Education Week. You can bookmark this page URL https://apt3e.blogspot.com/2008/12/federal-funding-for-teacher-quality.html. Thanks!

Teacher Quality and Title II

Education Week published an incredibly important story this week by Stephen Sawchuk ("Spending On Federal Teacher-Quality Funds Questioned") and Education Sector issued an incredibly important report (Title 2.0: Revamping The Federal Role in Education Human Capital) on Title II, Part A dollars in No Child Left Behind. The notion of better utilizing existing resources is especially critical in light of the economic downturn and budgetary challenges which will make new resources harder to come by.

As I wrote in two recent posts ($29 Billion Buys You A New Education System? and Will The New President Support New Educators?), it is abundantly clear that Title II, Part A's $3 billion are not flowing toward the most impactful initiatives in schools and districts. Most are going to class-size reduction and professional development (of questionable quality).

In a nutshell, Education Sector recommends shifting "the federal government’s role from enabler of existing activities largely irrespective of quality to a driver of reform through strategic investments in new initiatives, institutions, and policy schemes to recruit, train, support, and evaluate and compensate teachers."

Easy stuff, right?

You have read this article budget / Education Sector / Education Week / ESEA / NCLB / No Child Left Behind / teacher quality / Title II with the title Education Week. You can bookmark this page URL https://apt3e.blogspot.com/2008/11/teacher-quality-and-title-ii.html. Thanks!

Read It Here First

Please don't confuse this blog with a breaking news operation. It's hard to be on top of everything amidst being parents, holding full-time jobs, and trying to follow the pennant races (Go Sox!). But sometimes, despite doing most of our blogging after the sun's gone down, we're way ahead of the curve.

Such was the case with the TEACH Grants. Sara was all over this issue back in April, prior to the U.S. Department of Education's release of regulations. I notice that Education Week picked up the story in the September 15, 2008 issue, six months later.

Now, Stephen Sawchuk is an excellent reporter (formerly with EdDaily), but my wife scooped you on this one, dude!!! Check out some of Stephen's blogging at Teacher Beat.

You have read this article college / College Cost Reduction and Access Act / Education Week / loan / TEACH grant / teacher education with the title Education Week. You can bookmark this page URL https://apt3e.blogspot.com/2008/09/read-it-here-first.html. Thanks!