On think tanks and research bias
Monday, March 3, 2008Written by: Captain Haddock
Last week’s EdWeek features a guest commentary by CU School of Ed professor (and sometimes HeadFirst blog contributor) Kevin Welner. Co-written with Alex Molnar, an
… [A] phalanx of “one-armed” policy analysts are plying their trade in free-market-oriented think tanks. For these analysts and their think-tank sponsors, privatization is the preordained solution for each new educational problem. Indeed, time spent reading their reports leaves the unmistakable impression that the public nature of public education is the root problem for all that ails schools. Everything else is just a symptom.
Welner and Molnar’s analysis sheds light on one of the eternal elephants in the education policy research room: the conclusions of such research are highly correlated with the researchers’ ideologies and previous research findings. (Put less delicately, this means that much education policy research is biased). The trend is most obvious in politically charged areas like vouchers and charter schools.
Though Welner and Molnar identify a market-oriented bias to the think-tank research they survey, they would do well to extend their investigations to research organizations of all political persuasions. I’m willing to bet the work of, say, the American Federation of Teachers might be just a wee bit “one-armed”, too.
March 3rd, 2008 at 11:34 am
“Their review of 18 think tanks concludes that these organizations are heavily biased on the side of market solutions.” Quite the astonishing conclusion they reached, huh?
“Though Welner and Molnar identify a market-oriented bias to the think-tank research they survey, they would do well to extend their investigations to research organizations of all political persuasions. I’m willing to bet the work of, say, the American Federation of Teachers might be just a wee bit ‘one-armed’, too.” I’m glad to hear someone else make this point.
Bias, bias everywhere. Can’t we just measure each individual study on its scientific merits?
March 3rd, 2008 at 5:32 pm
Welner and Molnar provide a critically needed service as many of these reports from so-called think tanks are held up and marketed to policymakers as good research to base decisions on. Many of these studies, however, contain pretty shoddy research and poorly drawn conclusions - read some of the reviews. No one (and I believe Welner and Molnar will attest) thinks that AFT or NEA studies are unbiased - but strictly pointing out biases is not the sole point of the think tank project. It is meant to inject some sort of accountability into a fairly large and influential area of the market for education research that is currently no holds barred. Conservatives should welcome the accountability and not dismiss this as being liberally baised - but I guess they have a hard time swallowing what they dish out.
March 3rd, 2008 at 6:49 pm
Thanks for the attention. I guess.
Uncle Charley will be happy to hear that measuring each study on its “scientific” (we prefer “empirical” and “logical”) merits is precisely what the Think Tank Review Project — the focus of the commentary Cptn. Haddock mentions — is designed to do. Reports from think tanks are assigned to independent, academic reviewers, who write what we hope are readable but empirically grounded reviews (akin to the peer reviews written for journal articles). This effort is one of the projects of the Education and the Public Interest Center, which I direct.
While we haven’t reviewed anything from the AFT, we have reviewed reports from the Center on Education Policy (generally identified as on the left) and RAND (generally identified as centrist or non-partisan). But the reality is that free-market think tanks make up about 4/5th of Captain Haddock’s elephant. I think it important to avoid suggesting a (false) equivalency of quantity or policy impact. (Issues of quality are certainly debatable; we’ve found considerable variation, even among organizations with similar advocacy agendas.)
Anyway, this mention of the work of the Education and the Public Interest Center makes me realize that I need to write a “HeadFirst” entry about EPIC. Alan, it’ll be on its way soon. In the meantime, please feel free to visit us at epicpolicy.org.
March 3rd, 2008 at 8:01 pm
Addendum:
For non-subscribers to Ed Week, you can read the full commentary cited by Cptn. Haddock at http://epsl.asu.edu/epru/articles/EPRU-0802-408-OWI.pdf
March 4th, 2008 at 4:11 pm
One issue that seems empirically testable here is the number of “reports” or studies produced each year, and by what kind of group. Kevin, I don’t see this on your website - I see the number you reviewed, and you estimate in your comments above that 4/5 come from conservative organizations - what is the evidence for that? I’m not doubting it, just want to see how verifiable it is.
So, what is the universe (and I know that is hard, and would require ruling some documents out as not “studies” or “research” or whatever). But, you must have some metric in mind about what is worth reviewing. Similarly, if Uncle Charley could keep track of the union and/or liberal think tank reports with a similar metric, we could at least see how much “stuff” is out there.
I do think having all of these reports “reviewed” or analyzed or discussed is important, simply because some, perhaps many, bad ones leak out into the public domain, especially through media outlets that don’t have high bars. This distracts from high quality studies that might have useful information on “what works and what doesn’t,” and just confuses debate.
Kevin seems to be operating based upon an assumption that conservative think tanks dominate this discourse - some of his critics disagree. But, this could, and I think should, be determined, at least roughly.