Crash Testing the MCPS Hype

At this Montgomery County Public Schools website, MCPS proudly points out that it is the home of 34 National Blue Ribbon Schools.

Blue Ribbon Schools are awarded by the U.S. Department of Education—the feds. The award is a prestigious award denoting high quality schools. Back in the day, I did Blue Ribbon school site visits for the feds, and so, I believe winning denotes that something special is truly present. Click here to read more about the award.

I think it is fair to say that MCPS likes the Blue Ribbon program. And because we have so many winning schools, it is probably also fair to say that MCPS believes that the feds know quality schools when they see them.

So, many of us MCPS-watchers are somewhat surprised when MCPS rejects the suggestion that Jerry Weast-era programs be submitted to the U.S. Department of Education’s What Works Clearinghouse (WWC) for review. In fact, when making this suggestion, some of us have been told by MCPS officials that the feds don’t have a clue about how to conduct research. Really!

According the WWC website, the feds established the WWC “in 2002 to be a central and trusted source of scientific evidence for what works in education.” In short, the WWC independently verifies claims that education curricula and programs raise achievement levels by what they say they raise them by.

In the field of education, it is common for education curricula and programs—being sold by for-profit vendors or being promoted by public school districts simply wishing to praise their own accomplishments—to make public claims that their “stuff” closes achievement gaps between, say, white and black students. Frequently, claims are very specific. A vendor might claim that the use of their product increases reading achievement for fourth graders by X-number of months, or that the use of a specific intervention or program increases high school graduation rates.  From time to time, education curricula and programs come with program evaluation results that support the claims.

So, the genius of the WWC is that it takes these claims and independently checks them out. Independent researchers literally “kick” the tires by reviewing program details and data. When everything is checked, WWC issues a report on effectiveness. These reports are brutally candid. Here is what the WWC said about the popular "I Have a Dream" dropout prevention programs:

“No studies of I Have A Dream that fall within the scope of the Dropout Prevention review protocol meet What Works Clearinghouse (WWC) evidence standards. The lack of studies meeting WWC evidence standards means that, at this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness of I Have A Dream. I Have A Dream is a program that encourages students in low-income communities to complete high school and go on to college. The program guarantees that tuition for higher education will be covered after high school graduation. In addition, it provides participants with tutoring and counseling from elementary school through high school. Each I Have A Dream program sponsors either an entire grade level of students at a low-income public elementary school or an entire cohort of same-age children in a public housing development. … A full-time paid staff member coordinates program activities and serves as a mentor to program participants. A group of sponsors commits to working with the students throughout the life of the program and often provides the program with funding and other resources. ...”

In short, the WWC concludes that there is no empirical evidence to support claims that the "I Have A Dream" program works—prevents, for example, low income students from dropping out of high school. This is important for educators because knowing this literally stops them from replicating a program that is ineffective.

Now, because I said WWC “kicks” tires, here is another way of looking at what the WWC does: Imagine an automobile company designing what it believes is a safer automobile. This auto company builds this new auto and tells the world its new auto is the world's safest car ever. These are pretty bold claims. Now, this auto company has a long tradition of building high quality automobile and initially, no one objects to the claims of the safest car ever. Lots of cars are sold. Yet in the long run, no wise consumer is really going to believe the claims of the safest auto ever without some independent testing. And when it comes to auto safety one has to go through the Insurance Institute for Highway Safety tests—meaning, you just got to crash test the heck out this new car. In the end, IIHS determines how safe this new car really is based on its independent crash tests data. Ultimately, it’s the IIHS results that make or break this new car—not the company hype.

And so, on one level, that is what some of us are asking MCPS to do with the WWC. We simply want MCPS to get busy crash testing and eventually proving that the Weast-era programs worked. Prove the MCPS claims that its curricula and programs closed achievement gaps.

Is the WWC effectiveness bar really too high a bar for a world-class public school district like MCPS to jump over?

By the way, the WWC evolved out of work originally performed by Dr. Rebecca Herman, a nationally-recognized researcher at the American Institutes for Research. Dr. Herman was instrumental in helping the feds establish the WWC. I mention this work because I worked at AIR when Dr. Herman published the original "An Educators' Guide to Schoolwide Reform."

Click here to read the original Herman guide.

This work is noteworthy because the original guide’s clients included the nation's two major teachers unions—the American Federation of Teachers and the National Education Association. And why did the AFT and NEA see value in independently reviewing claims of education curricula and programs? I believe it was because they didn’t want their teachers wasting time on “stuff” that really doesn’t live up to the hype.