Live By The Sword, Die By The Sword?

The problem with Jay Mathews' defense ("Measuring Progress At Shaw With More Than Numbers") of a Washington, DC school principal who did not demonstrate student learning gains at his school after one year is that the principal operates within an accountability system that demands such a result. In this case, both Mathews -- and DC Schools Chancellor Michelle Rhee, as described in Mathews' WP column -- are right not to have lowered the boom on Brian Betts, principal of the DC's Shaw Middle School at Garnet-Patterson, based on a single year's worth of test scores.

The state superintendent of education's Web site says Shaw dropped from 38.6 to 30.5 in the percentage of students scoring at least proficient in reading, and from 32.7 to 29.2 in math.

But those were not the numbers Rhee read to Betts over the phone.

Only 17 percent of Shaw's 2009 students had attended the school in 2008, distorting the official test score comparisons. Rhee instead recited the 2008 and 2009 scores of the 44 students who had been there both years. It didn't help much.

The students' decline in reading was somewhat smaller; it went from 34.5 to 29.7. Their math proficiency increased a bit, from 26.2 to 29.5. But Shaw is still short of the 30 percent mark, far below where Rhee and Betts want to be....

Despite the sniping at Rhee, the best teachers I know think that what happened at Shaw is a standard part of the upgrading process. I have watched Betts, his staff, students and parents for a year. The improvement of poor-performing schools has been the focus of my reporting for nearly three decades. The Shaw people are doing nearly everything that the most successful school turnaround artists have done.

They have raised expectations for students. They have recruited energetic teachers who believe in the potential of impoverished students. They have organized themselves into a team that compares notes on youngsters. They regularly review what has been learned, what some critics dismiss as "teaching to the test." They consider it an important part of their jobs.

That's how it's done, usually with a strong and engaging principal like Betts.

Mathews' take -- including consideration of contextual factors, such as the fact that only 17% of the school's students had attended the prior year and the contention that school turnaround requires more than a single year -- is how the education world should work. Embrace the complexity of learning and trying to measure it! To do so would disallow the use of single-year changes in test scores for making high-stakes decisions about schools and individual school personnel. It would also remove the unrealistic pressure on school turnarounds to bear fruit in a single year. Test scores would be used responsibly in combination with other data and evidence to paint a fuller picture about individual school contexts and inform judgments about school leadership and student success.

But Michelle Rhee and other education reform advocates have publicly argued that student performance as measured by test scores is basically the be all and end all. According to this Washington Post story ("Testing Tactics Helped Fuel D.C. School Gains"), Rhee supports strengthening No Child Left Behind to "emphasize year-to-year academic growth." Such a stance creates a problem for such reformers when they are leading a district and staking their leadership on uncomplicated test score gains. Others will assess their leadership and judge their success by this measure -- an ill-advised one in its simplest form.

I would argue that, in addition to doing the right thing (as happened in this instance), reform advocates and school leaders like Rhee also have a responsibility to say and advocate for the right thing. They have a responsibility to be honest about the complexity of student learning and the inability of student assessments to accurate capture all of the nuance going on within schools and classrooms. While the reformers' challenge of the adult-focused policies of the educational status quo is often warranted, some reforms -- accountability, chief among them -- have been taken too far. Student learning, school leadership and teaching cannot be measured and judged good or bad based on a single set of test scores. Test scores must be part of the consideration -- and supporting systems such as accountability, compensation and evaluation must be informed by such data -- but they should not single-handedly define success or failure.

The complexity as presented by Mathews in his article -- and, more importantly, by existing research (such as by Robert Linn, Aaron Pallas, Tim Sass, and embedded within Sunny Ladd's RttT comments) about year-to-year comparisons of both overall test scores and test score gains -- strongly suggests that educational accountability systems should be designed more thoughtfully than they have been to date, but unfortunately that does not seem to be the direction that policymaking is headed at either the federal or state levels. Part of being more thoughtful is moving away from NCLB-style adequate yearly progress and toward a value-added approach, but thoughtfulness also requires not making high-stakes decisions based exclusively on volatile student data. Do I hear "multiple measures"? Sure, but Sherman Dorn offers some provocative thoughts on this subject in a 2007 blog post.

With regard to educational accountability, policymakers first should do their homework -- and then they clearly have more work to do in creating a better system and undoing parts of the existing system that aren't evidence-based and accomplish only in simplifying a truly complex art: learning.

-------------------

For those of you that have gotten this far, there's a related post on the New America Foundation's Ed Money Watch blog discussing a new GAO report that analyzes state spending on student assessment tests -- $640 million in 2007-08.

The increasing cost of developing and scoring assessments has also led many states to implement simpler and more cost-effective multiple choice tests instead of open response tests. In fact, although five states have changed their assessments to include more open response items in both reading and math since 2002, 11 and 13 states have removed open items from their reading and math tests, respectively over the same time period.... This reliance on multiple choice tests has forced states to limit the content and complexity of what they test. In fact, some states develop academic standards for testing separately from standards for instruction, which are often un-testable in a multiple choice system. As a result, state NCLB assessments tend to test and measure memorization of facts and basic skills rather than complex cognitive abilities.

------------

And here's a new story hot off the presses from Education Week. It discusses serious questions raised about New York City's school grading system.

Eighty-four percent of the city’s 1,058 public elementary and middle schools received an A on the city’s report cards this year, compared with 38 percent in 2008, while 13 percent received a B, city officials announced this month.

“It tells us virtually nothing about the actual performance of schools,” Aaron M. Pallas, a professor of sociology and education at Teachers College, Columbia University, said of the city’s grades.

Diane Ravitch, an education historian at New York University, was even sharper: She declared the school grades “bogus” in a Sept. 9 opinion piece for the Daily News of New York, saying the city’s report card system “makes a mockery of accountability.”

But Andrew J. Jacob, a spokesman for the New York City Department of Education, defended the ratings, even as he said the district’s demands on schools would continue to rise next year....

The city employs a complex methodology to devise its overall letter grades, with the primary driver being results from statewide assessments in reading and mathematics, which have also encountered considerable skepticism lately.

The city’s grades are based on three categories: student progress on state tests from one year to the next, which accounts for 60 percent; student performance for the most recent school year, which accounts for 25 percent; and school environment, which makes up 15 percent.

Mr. Pallas of Teachers College argues that one key flaw with the city’s rating system is that it depends heavily on a what he deems a “wholly unreliable” measure of student growth on test scores from year to year that fails to account adequately for statistical error.

You have read this article Aaron Pallas / accountability / Assessment / Helen Ladd / Jay Mathews / Michelle Rhee / New York City / No Child Left Behind / reform / Robert Linn / Tim Sass / Washington DC / Washington Post with the title Assessment. You can bookmark this page URL http://apt3e.blogspot.com/2009/09/live-by-sword-die-by-sword.html. Thanks!

Genesis of A Firewall

A question has been raised regarding the genesis of Wisconsin's restrictive law preventing the consideration of student achievement in the evaluation of teacher performance. Well, it dates back 18 years to 1991. Section 633m of Act 269 (page 131 of the PDF file), a huge omnibus bill, put this restriction into state law. After some minor wordsmithing in Act 16 in 1993, the relevant section of Wisconsin's current statute (118.30 (2) (c)) reads:

The results of examinations administered under this section
to pupils enrolled in public schools, including charter schools,
may not be used to evaluate teacher performance, to discharge,
suspend or formally discipline a teacher or as the reason for the
nonrenewal of a teacher’s contract.

I wonder if any long-time Wisconsinites could provide background detail on how this got into legislation. That was way before my time in the Badger State. UPDATE: A reliable source tells me that this biennial budget bill is where the 8th- and 10th-grade state assessment (the Wisconsin Knowledge and Concepts Examination) was established.

Also, has anyone answered the question regarding the vintage of California's alleged law that may render it ineligible for Race To The Top funding? (Never mind, this news story provides the answer: 2006. A decent year in Napa, not so much in Sonoma.)

It has been widely reported that New York's firewall preventing the consideration of student performance in teacher tenure decisions, passed in 2008, is set to expire in 2010.

You have read this article ARRA / Assessment / California / New York / Race To The Top / teacher evaluation / Wisconsin with the title Assessment. You can bookmark this page URL http://apt3e.blogspot.com/2009/08/genesis-of-firewall.html. Thanks!

Where's That Dutch Kid?

Gotham Schools reports that some in New York State don't believe that the state's law that restricts student assessment data from being used in teacher tenure decisions will hamper the state from securing Race To The Top funding. Is this just wishful thinking or is this whole issue being oversimplified by proposed federal RTTT regulations?

New York State’s tenure law, passed last year under pressure from teachers unions, says student test score data can’t be the sole determinant of whether a teacher gets tenure. But three top officials — teachers union president Randi Weingarten, Board of Regents Chancellor Merryl Tisch, and incoming State Education Commissioner David Steiner — are arguing that the law will not disqualify New York from the fund.

“It is our firm belief that the language of Race to the Top funding does not preclude New York,” Steiner said today. “New York has a law on the books that relates strictly to tenure.”

Weingarten noted that a second section of the same law explicitly requires teachers’ annual evaluations, which take place even after they receive tenure, to be based in part on how they use test score data to improve their instruction.

“The way in which teachers use data in their classroom instruction is specifically included in the definition of what confers tenure onto a classroom teacher,” she said. ”How teachers use data is one of the criteria for getting tenure. Just not the data in and of itself.”

NY UPDATE: Charlie Barone says BS.

Likewise, in Wisconsin -- another state singled out by Education Secretary Arne Duncan for having a "ridiculous" law that restricts the use of student assessment data in teacher evaluations -- the Governor's office says that the law only applies to data from the state assessment. Assumedly, other assessment data could be used instead, although that creates costs and logistical hurdles for school districts, some very small. From the Milwaukee Journal-Sentinel:

According to Chapter 118.30(2)(c) of the Wisconsin State Statutes, "the results of examinations to pupils enrolled in public schools, including charter schools, may not be used to evaluate teacher performance, to discharge, suspend or formally discipline a teacher, or as the reason for the nonrenewal of a teacher's contract."

By Friday afternoon, state Sen. Randy Hopper (R-Fond du Lac) and Rep. Brett Davis (R-Oregon) had announced plans to introduce legislation that would change Chapter 118.30(2)(c) to eliminate the prohibition on using state testing in teacher evaluations.

But according to Gov. Jim Doyle's office, the Wisconsin statute is not at odds with the state's Race to the Top eligibility.

"Our reading of the current law is that it only prohibits the use of the WKCE (Wisconsin Knowledge and Concepts Examination) in evaluating teachers, and that other student assessments may be used to evaluate teachers," said Lee Sensenbrenner, a spokesman for Doyle's office.

Sensenbrenner said the governor will be putting together a comprehensive application for the Race to the Top competition that puts the state in a position to succeed.

As part of that, he said, the state would "review the existing law to see if any changes need to be made to strengthen our competitive position."

UPDATE: On Teacher Beat, Stephen Sawchuk has a pithy update on this issue -- and the pleadings of California, New York and Wisconsin about how this really isn't a problem. Really, it isn't!

You have read this article ARRA / Assessment / New York / Race To The Top / stimulus / student / teacher evaluation / U.S. Department of Education / Wisconsin with the title Assessment. You can bookmark this page URL http://apt3e.blogspot.com/2009/07/where-that-dutch-kid.html. Thanks!

Fight The Power!

Dana Goldstein raises some serious questions and concerns in her American Prospect article ("Testing Testing") about the process of developing national academic standards. The process is dominated by three organizations--two (ACT and the College Board) with a proprietary interest in ensuring that assessments are a featured component of any final product.

The problem is that the initiative's co-signers aren't just state governments--they are also testing groups: Achieve, a nonprofit that advocates for more effective standardized tests; the College Board, maker of the SAT; and ACT, which administers a competing college-entrance exam. Right now, the College Board and ACT have little engagement with the K-12 education sector. They do, however, have ample experience creating and administering national exams. And there is little doubt that one goal of this national-standards process is to create standardized tests--not one single national test but perhaps two or three options from which states can choose.

As oligopolists, it makes total sense for the College Board and ACT to be eyeing, together, expansion into the immense K-12 assessment market. But given these testing companies' track records, it is worth asking if this is a wise idea. A number of studies have found SAT scores are far less effective than high school grades in predicting how well students will perform in college, and professors say standardized-test prep does little to teach students the research and critical thinking skills they will need at the college level. Because of these shortcomings, an increasing number of colleges--led by the giant University of California system--have made standardized test scores optional for admission.

It would be a shame if national education reform further cemented a system in which passing standardized tests is the goal of learning.

While others (including Dan Brown) have pointed out that only one classroom teacher has a seat at the table, Goldstein follows the money, so to speak. I am disappointed, although not surprised, that the national organizations leading this effort have basically turned it over to Testing, Inc. The corporate boards of both the Council of Chief State School Officers and the National Governors Association are littered with representatives of the student assessment industry--ETS, McGraw Hill, and Pearson in the case of CCSSO; and ACT, the College Board, ETS, and Pearson in the case of NGA. To their defense, both CCSSO and NGA list these organizations directly on their respective web sites. As a former employee at NGA, I also can honestly say that their funding did in no way impact the substantive advice provided to the nation's Governors when I worked there. But does it provide these companies ready access to Governors and their senior staff at regular meetings? Sure. Does it raise questions about their role in this standards-setting process and create the appearance of bias? Absolutely.

Of the 29 slots on the mathematics and English-language arts Work Groups, 15 are taken by employees or affiliates of ACT and The College Board. Another seven slots are occupied by Achieve, Inc. (Some individuals serve on both Work Groups.) Of the remaining seven slots, two are filled by America's Choice, two by Student Achievement Partners, and single seats by a communications firm, a consultant, and a professor. In addition, 37 individuals serve on twin Feedback Groups for both math and English/LA standards. They are overwhelmingly higher education faculty. Of the 19 members of the math feedback group, 15 represent higher education with a single k-12 teacher in the mix. Of the 18 members on the English/LA feedback group, 14 are professors and there is one "instructional performance coach" from a public charter school as well.

The decision to cut k-12 educators out of the standards development process contrasts sharply with the rhetoric of President Obama and Secretary Duncan about including educators in the development of education reforms. Indeed, it would "be a shame" if Testing Inc. rode this gravy train to the (hopefully not) inevitable conclusion suggested by Goldstein's article. Of course, in the end, it is the product rather than the process that really matters. In this case, one can hope that some of the participants' potentially parochial and proprietary interests don't define the outcome or the intent of the entire effort. The standards should be developed based on what is best for students and how such standards can best be utilized by educators -- not to ensure their ease in being converted into multiple-choice tests.

Hat tip to TWIE.

UPDATE: See Education Week story (7/30/2009).

You have read this article academic standards / ACT / American Prospect / Assessment / College Board / Council of Chief State School Officers / Dana Goldstein / National Governors Association / teacher with the title Assessment. You can bookmark this page URL http://apt3e.blogspot.com/2009/07/fight-power.html. Thanks!

Cheerleading for NCLB

I guess my reaction to today's Washington Post shout out to No Child Left Behind ("'No Child' in Action") from former U.S. Education Secretary Margaret Spellings is a question: "If NCLB's accountability alone is such a silver bullet, then how come test scores at the high school level didn't improve?"

Although Spellings mentions that NCLB requires math and reading tests in grades 3-8, it is quite disingenuous of her not to mention that such tests were also required in high school. If the achievement gains aren't sustained through high school, what real difference does it make?

The wise Aaron Pallas offers his take on this issue ("Wishful Thinking"), calling into question Spellings's claims:

But what portion of those trends can be attributed to NCLB? Margaret Spellings refers to changes since 1999, which is convenient for her story, because there were sharp increases in grade 4 reading between 2000 and 2002, and in grade 4 and grade 8 math between 2000 and 2003. But NCLB was signed into law in January, 2002; the first final regulations dealing with assessment were issued in December, 2002; and initial state accountability plans were approved by the U.S. Department of Education no later than June, 2003. The 2003 main NAEP was administered between January and March of 2003. Is it realistic to claim that NCLB affected scores before the 2003 NAEP administration? I, and a great many other analysts, think not.

Only in Margaret Spellings’ world can NCLB affect NAEP scores for the four years before the law was passed and implemented. Now that’s wishful thinking.

UPDATE -- Diane Ravitch comes to similar conclusions in her blog post.

Thus, when one looks at the patterns, it suggests the following: First, our students are making gains, though not among 17-year-olds. Second, the gains they have made since NCLB are smaller than the gains they made in the years preceding NCLB. Third, even when they are significant, the gains are small. Fourth, the Long Term Trend data are not a resounding endorsement of NCLB. If anything, the slowing of the rate of progress suggests that NCLB is not a powerful instrument to improve student performance.

Caveat emptor.

You have read this article Aaron Pallas / accountability / Assessment / data / Margaret Spellings / NAEP / NCLB / No Child Left Behind with the title Assessment. You can bookmark this page URL http://apt3e.blogspot.com/2009/05/cheerleading-for-nclb.html. Thanks!

"Illusory" Test Score Gains

Here is a neat little story from today's Washington Post that puts my recent post about Maryland test scores into some context.

Recent reports from the Thomas B. Fordham Institute and [Bruce] Fuller's group, Policy Analysis for California Education, have concluded that most recent gains on state tests are illusory, reflecting better test-taking skills or lower standards rather than increased knowledge. Another study, from the Center on Education Policy, concluded that the gains seemed genuine but did not necessarily reflect greater learning.

You have read this article adequate yearly progress / Assessment / AYP / Maryland / NCLB / No Child Left Behind / Washington Post with the title Assessment. You can bookmark this page URL http://apt3e.blogspot.com/2008/07/test-score-gains.html. Thanks!

Student Achievement in Maryland

In Maryland, the glass is half full ... or is it half empty?

From 2007 to 2008, the share of students statewide who were judged proficient or better rose six percentage points in reading and four points in math, to 82 percent and 76 percent, respectively, on the Maryland School Assessments.

But

State reading and math tests taken by Maryland students were shortened and tweaked this year, leading some critics to question whether the shifts contributed to surprisingly strong gains in achievement.

Is this uptick in student achievement in Maryland legit? Or is this another example of a state gaming the system [see here and here] when it comes to NCLB's adequate yearly progress requirements?

Ed Week's David Hoff provides a nice summary of the trickery employed by states in this blog post from November 2007.

--------------

ADDED: Read Eduwonk's take on this.

You have read this article adequate yearly progress / Assessment / Maryland / NCLB / No Child Left Behind with the title Assessment. You can bookmark this page URL http://apt3e.blogspot.com/2008/07/student-achievement-in-maryland.html. Thanks!

Teaching To The Test

Thursday's Christian Science Monitor ran a provocatively titled op-ed--"Good Teachers Teach to the Test." In it former Los Angeles teacher and UCLA lecturer Walt Gardner contends that teaching to the test is "eminently sound pedagogy." That's educator-speak for "it's the right thing to do."

In the era of No Child Left Behind, many educators and observers have bemoaned teaching to the test. Gardner makes the case that teaching to the test in the right way is about much more than teaching kids to fill in tiny bubbles and master only basic skills and knowledge. Sadly, that does happen in some schools.

If a test measures what we want and expect students to learn and be able to demonstrate, then why not teach to the test? Flying in the face of conventional wisdom, Gardner argues that it's entirely possible for a highly skilled educator to design an assessment integrated with their lesson plan and the overall curriculum.

You have read this article Assessment / No Child Left Behind / teacher / Test with the title Assessment. You can bookmark this page URL http://apt3e.blogspot.com/2008/04/teaching-to-test.html. Thanks!