Why IRB Matters: A Real-Life Example from a Political Science Experiment Gone Wrong
Why IRB Matters: A Real-Life Example from a Political Science Experiment Gone
Melissa R. Michelson, Menlo College
October 31, 2014
Details continue to emerge about a get-out-the-field experiment that came to light last Friday. The experiment, fielded by three political science professors, involved the sending of mailers to registered voters in California, Montana, and New Hampshire. Here, I review the facts of the case as we now know them, review the ongoing controversy and ethical issues that have been raised, and possible implications. The flawed experiment is not only bad news for the involved political scientists, but for the field more broadly. All should take note.
The Facts of the Case (So Far)
The lead researcher of the project is Adam Bonica, and assistant professor of political science at Stanford University.
His collaborators include Jonathan A. Rodden, Professor of Political Science at Stanford University and Senior Fellow at the Hoover Institution, and Dartmouth College Assistant Professor of Government Kyle Dropp.
According to the researchers' website, the question being investigated was whether voter turnout could be increased by providing more information about candidates in certain kinds of races. In other words, would registered voters who received the mailers be more likely to vote and more likely to complete their ballots. The second question relates to the issue of drop-off on down-ballot contests; often, voters will make a choice in a top-ballot race such as a gubernatorial or U.S. Senate contest, but decline to vote in a less salient contest further down the ballot, such as for the judiciary or for local water board. A number of scholars continue to investigate both how to increase voter turnout and also how to increase ballot completion. Here's the document explaining the experiment.
The experiment was conducted in three states, including Montana (100,000 mailers), New Hampshire (66,000 mailers) and California (143,000 mailers). The mailers provided a visual of an ideological spectrum (liberal to conservative), including placement on that continuum of President Barack Obama and former Republican Presidential candidate Mitt Romney, as well as the candidates in the targeted down-ballot election contest. According to the researchers' website, the information was based on "candidates' ideological positions based primarily on their fundraising activities," using publicly disclosed campaign finance records.
The Media and Public Reactions
The ensuing controversy has included legal charges from Montana Secretary of State Linda McCulloch, demands for an apology from U.S. Senator Jon Tester, letters of apology from the Presidents of the involved institutions, Stanford University and Dartmouth College, and a slate of blog posts, newspaper articles, and social media discussions. An interview with the Montana Commissioner of Political Practices Jonathan Motl with News Talk KGVO's Jon King can be heard here.
You can read my initial reactions to the controversy here; there have been insightful and informative reactions from other political scientists on other blogs and media outlets, including Dan Drezner, Thomas Leeper, Chris Blattman, and Paul Gronke. The issue has also been covered by the The Upshot and Talking Points Memo (here and here).
Most reactions address the stated intent of the experiment and on the ethical questions raised by the use of the state seal, the number of mailers used, and the insertion of ideological ratings in a non-partisan election. Some question whether Bonica had a conflict of interest due to his recent start up, Crowdpac, which rates political candidates on an ideological spectrum using a similar methodology.
By far the majority of the criticism aimed at the experiment relates to the mailers sent to Montana voters. The mailers included ideological labeling of candidates for two nonpartisan elections to the Montana Supreme Court. One of those judicial races, between incumbent Justice Mike Wheat and challenger Lawrence VanDyke, is hotly contested, and has seen significant outside spending. Many observers, myself included, questioned the decision to insert ideological labels into a non-partisan race. This conflicts with the longstanding tradition in the West, dating from the Progressive Era, of maintaining an independent judiciary. For a good summary of local outrage, see this piece in the Montana Standard.
This vision of judicial elections as nonpartisan may be more myth than reality. Political Scientists Chris W. Bonneau and Damon M. Cann, using mixed-methods including an experiment, find in a 2013 Political Behavior piece that "nonpartisan elections for judges fail to meet their stated goal of minimizing the role of political factors like partisanship and ideology in judicial selection." Melinda Gann Hall, in her new book Attacking Judges, finds that attack ads reduce the incumbency advantage in nonpartisan state supreme court elections. More about the true nature of judicial elections, and why they are in fact intensely partisanship, can be found in this 2010 interview with Professors Bonneau and Hall.
So perhaps judicial elections are not as non-ideological or nonpartisan as some of us would like to believe. But the Montana mailers still struck many locals as inappropriate, and in this case, I believe, appearances matter. The appearance of researchers from elite, out-of-state institutions overtly inserting ideological ratings into a non-partisan judicial election was roundly and loudly criticized by residents of the Glacier State, to the extent to some cases of suspecting ulterior motives and unsavory conspiracies by unnamed big spenders. The Cowgirl Blog raised several questions related to Crowdpac; further questions are raised at Flathead Memo. Cowgirl Blog notes:
Why was Adam Bonica in Montana earlier this year trying to sell his for-profit services through CrowdPAC at a FollowtheMoney.org conference in May of 2014? Why did the site montanans4justice.com, which until yesterday had posted a graphic nearly identical to the fake voter guide mailers, suddenly scrub that graphic from the site?
The only the difference was that the Montanans4Justice graphic and the fake voter guides is that Montanans4Justice used little pictures of the candidates heads showing how close or how far a candidate's head was to Obama. But as soon as the mailergate story broke, the graphic on Montanans4Justice was obliterated from that site.
As the Flathead Memo reported, the anti-Wheat site "Montanans4justice… was registered on 3 September 2014 by an anonymous party." And even though the graphic was removed as of this posting, it still contains references to the exact same partisanship metrics used by the mailers and the supposed "experiment" - CrowdPAC's DIME method, as well as criticism for Wheat and praise for VanDyke. CrowdPAC launched the same day.
Questions raised in Montana also related to the size of the experiment. Paul Gronke notes:
There are 671,031 registered voters in Montana, so this mailer was sent to 15% of the electorate. Depending on how many of the recipients had already intended to vote, using the 2010 turnout as a baseline, as much as half the total voting population received this mailer!
In the Citizens United era, many independent, dark money groups engage in campaigning for and against candidates and ballot measures. Does that mean that political science experiments should not be criticized for trying (however inadvertently) to change election outcomes? Again, the appearance of misconduct is part of the problem, and feeds into the public perception of political science professors as conducting work of questionable validity and questionable value.
A third issue relates to the actual material included in the mailers. McCulloch has charged that the use of the state seal on the mailers, and the wording, "2014 Montana General Election Voter Information Guide," violate various state laws. McCulloch has filed a legal complaint charging that the mailers also violate three campaign laws: 1) A ban on "fraudulent contrivance" that could cause a person to vote a certain way; 2) A prohibition on the dissemination of information that gives incorrect or misleading election procedures; and 3) A requirement that a person or group engaging in political activity register with the state.
The mailers in New Hampshire used the state flag rather than the state seal, while the California mailers also used an official state seal, according to one news article. The California pieces were sent to voters in the 4th and 25th Congressional Districts, targeting runoff elections in which both candidates are Republicans, a result of California's fairly new Top-Two-Vote-Getter primary system. The California mailers may have broken state law, according to Talking Points Memo.
"It is a violation of California government code to use the state seal," Nicole Winger, a spokesperson for the California secretary of state's office, told TPM. When asked if the mailers did appear to bear the state seal, she said: "Oh yeah. That looks like the state seal."
The Broader Implications
One of the biggest misjudgments made by the Bonica team was choosing to not submit the experiment for review at Stanford. As Stanford announced on their website:
Faculty members at universities like Stanford and Dartmouth have certain rights and responsibilities in connection with the conduct of research.
On the one hand, they are granted freedom of inquiry and publication as part of their right to academic freedom, in which "freedom of inquiry, thought, expression, publication, and peaceable assembly are given the fullest protection," in accordance with Stanford policy.
At the same time, researchers have certain responsibilities that go with those rights, including the requirement that their research comport with established professional ethics. In addition, research universities have review boards, known as Institutional Review Boards, that seek to protect human research participants by ensuring their rights are protected, that research is guided by ethical principles, and that research complies with applicable laws, policies and regulations. Information about Stanford's process is available here. The election study at issue was not submitted to Stanford's Institutional Review Board, which is required by the university's normal process for studies of this nature.
Often, we political scientists gripe about the IRB process (as do our colleagues in other fields). It can be time-consuming, and frustrating. Sometimes, IRB members seem to us to be making unreasonable demands for revisions to our proposed research projects, or to be taking too long to approve time-sensitive ideas. I've personally conducted well over 300 field experiments. Some of those experiments had to be changed, or cancelled, because of IRB concerns. I can understand the temptation to sidestep the process altogether, as the Bonica team did here. But that temptation should be resisted. IRB review is important: often we have blinders on when it comes to the possible shortcomings of our own ideas.
U.S. Senator Jon Tester sent a strongly worded letter to the presidents of Stanford and Dartmouth on October 24 that began, "As I am sure you are now aware, Montanans recently received a misleading campaign flier in their mailboxes that sought to inject partisanship into non-partisan Montana Supreme Court elections." Tester goes on to denounce the "so-called research project," and adds, "Efforts to undermine elections in Montana - whether by fraud or merely by poorly-designed experiments - must not be tolerated."
This harms the discipline. Political science experimenters, and political scientists more broadly, are being portrayed as unethical and as inappropriately meddling in real elections. Dartmouth President Philip Hanlon and Stanford President John Hennessy have issued an open letter of apology to the people of Montana, and have plans to also apologize to California voters.
Bonica and his collaborators will pay a price for this controversy, including the loss of usable data from the Montana and California mailers, and there may be fines that result from the charges of violating state laws. That said, their careers are not over; they will conduct other research, publish other papers, and move forward.
The greater price is that being paid in terms of public perceptions of political science field research: that political scientists are unethical, engage in fraud, and are improperly meddling in elections. Future experiments will now have more difficulty getting approved and funded. Just last year, the Coburn Amendment struck a major blow to political science National Science Foundation grants. Political science was dismissed as wasteful and not in the national interest. Scandals of this sort fan these anti-political science research flames.
Political science research often impacts the real world. My get-out-the-vote experiments, for example, aim to increase turnout. This is a blatant and upfront attempt on my behalf to impact the real world, stemming from my belief that increased participation is desirable. Others might disagree with that, preferring that votes only be cast by those who care enough to motivate themselves to get to the polls.
I make every effort to engage in GOTV work that is ethical: the work is non-partisan and non-ideological, and I mobilize individuals regardless of their political party affiliation or ideology. For example, for next week's city council election in Atherton, California, home of Menlo College, I've worked with my students to register the student body to vote and to participate. Will that impact the outcome?
Possibly. In a town with about 4,900 registered voters, now including 288 at our campus address (students and on-site faculty and staff), Menlo College is 6% of the electorate. In a close race, that could be the margin of victory or defeat. None of the outreach aims to influence vote choice, the project has been approved by Menlo IRB, and the work is being done with the knowledge of the city council candidates. I believe what we're doing is ethical, even if it might, through heightened participation by the Menlo College community, change the outcome of the election. Others might disagree; Dawn Teele claims in Field Experiments and Their Critics that ethical field research requires that the research itself leave no trace.
Experiments that strike the public as unethical, possibly illegal, and deceptive, are harmful to our image as a discipline and will make it more difficult for future studies to obtain funding and IRB approval. Doug Chapin argues that this experiment amounts to political science "blowing its close-up".
Intentionally or not, these researchers have caused serious harm. Possibly to the people of Montana, California and New Hampshire, and definitely, IMHO, to political science.
Dr. Melissa R. Michelson is Professor of Political Science at Menlo College. Her publications include Living the Dream: New Immigration Policies and the Lives of Undocumented Latino Youth (Paradigm Publishers, 2014) and Mobilizing Inclusion: Transforming the Electorate Through Get-Out-the-Vote Campaigns (Yale University Press, 2012).