Myth-Busting Monday
Myth-Busting Monday
Flawed Research: John Lott’s complex and erroneous defense of Plassmann and Whitley’s 2003 Stanford Law Review article.
This article was originally published on GVPedia.
Summary:
On April 16, 2003, Yale researcher Ian Ayers and Stanford researcher John Donohue published a critique in the Stanford Law Review of researcher John Lott’s work in an article, “Shooting Down the ‘More Guns, Less Crime’ Hypothesis.” This was the first installment of a four-paper series debating the impact of right-to-carry laws.
Analyzing new data released after Lott’s initial work in his book, More Guns, Less Crime, Donohue and Ayres uncovered that right-to-carry (RTC) laws do not decrease crime as Lott’s initial theory claimed. Donohue and Ayers stated in their paper, “No longer can any plausible case be made on statistical grounds that shall-issue laws are likely to reduce crime for all or even most states” and suggested such laws might increase violent crime.
Lott was invited to include a response in the same issue of the Stanford Law Review. Lott co-authored a response with economists Florenz Plassmann and John Whitley, but despite being the lead author, Lott eventually removed his name from the paper due to a single word in a 120 page paper, according to the editor of the journal (that word has not been publicly disclosed).
The response, published under Plassmann and Whitley’s names only, introduced new data in an attempt to support the initial claim (RTC laws reduce crime).
Upon analyzing Plassmann and Whitley’s new data, Ayers and Donohue found errors that, in fact, rejected the claim that more guns equals less crime.
While Plassmann admitted that Ayres and Donohue had correctly identified a substantial error in the paper, Lott continued to deny that the error caused any results to change, and publicly provided a data table to support his claim.
Lott’s table, however, still contained substantial data errors.
Lott quietly made a series of “corrections” to the original data table and backdated the changes, but even after the corrections the table still contained errors.
Flawed Research:
On April 16, 2003, the Stanford Law Review published “Shooting Down the ‘More Guns, Less Crime’ Hypothesis” by researchers Ian Ayres and John Donohue. John Lott, along with researchers Florenz Plassmann and John Whitley, wrote a reply to Ayres and Donohue in an attempt to confirm the “more guns, less crime” hypothesis.
After examining Lott’s reply, Ayres and Donohue discovered numerous coding errors (mislabeling data) and empty spreadsheet cells. When these errors were corrected by Ayres and Donohue, Lott’s own model showed that right-to-carry laws did not reduce crime and even increased some categories of crime.
In a September 2003 email exchange with Tim Lambert, a computer scientist at the University of South Wales, Plassmann admitted that correcting the coding errors changed the paper’s conclusions.
Immediately before publication, Lott removed his name from the final paper, citing disagreements with the editor over a single word in the 120 page paper. Ben Horwich, President of the Stanford Law Review, explained, “In the end, I had consented to everything that Lott wanted, save one word. He decided to withdraw his name from the paper.”
The Stanford Law Review series sparked a debate over whether coding errors found by Ayres and Donohue in Plassmann and Whitley’s response changed the conclusion that RTC laws are beneficial. After correcting Lott’s coding errors, Ayres and Donohue presented a data table which clearly showed that the corrections significantly change the results.
In response, Lott posted a “corrected” table on his website. This “corrected” table, however, showed the same values that Donohue and Ayers had. In other words, the coding errors were removed, but Lott still claimed the correctly coded data showed statistically significant drops in rape, murder, and robbery.
As Chris Mooney explained in Mother Jones, Lott “omitted a key calculation regarding statistical significance used in the Plassman-Whitley paper.” To sway the results in his favor, Lott removed the clustering to make the results look statistically significant once again.
Mooney confronted Lott about this change during two telephone interviews and several email exchanges. Lott replied with a series of contradictory claims throughout a prolonged email discussion with Mooney. Eventually, Lott refused to admit any error on his part, and instead had his webmaster try to take the entire blame. Even months after the incident, neither Mooney nor Lambert received an adequate explanation of what actually transpired.
Lott’s Counter:
On June 23, 2015, John Lott posted the following counter on his website to the questions by Ayers, Donohue, Mooney, and Lambert which were compiled in an Armed With Reason article by Evan DeFilippis and Devin Hughes.
Lott posted:
“The reply to Ayres and Donohue in the law review was by Florenz Plassmann and John Whitley. I had helped them out, and Whitley notes, ‘We thank John Lott for his support, comments, and discussion.’ There were minor data errors in the additional years that they added from 1997 to 2000, but those errors didn’t alter their main results that dealt with count data. They had accidentally left 180 cell [sic] blank out of some 7 million cells. Donohue has himself made much more serious data errors in his own work on this issue. For example, he repeats the data for one county in Alaska 73 times, says that Kansas’ right to carry law was passed in 1996 and not 2006, and made other errors. I did co-author a corrected version of the Plassmann and Whitley paper that fixed the data errors and is available here. But DeFilippis and Hughes can’t even get it straight what paper I co-authored.”
Rebuttal:
In his counter, Lott attempts to downplay his role in the initial paper despite the original draft listing his name first and statements made by the Stanford Law Review editor indicating he was an extensive part of the project.
It is a well established fact that Lott was the lead author on the paper, but then removed his name during the editing process.
Lott also claims that the data errors were minor because they did not alter the main results of the study. The errors in Plassmann and Whitley’s paper that Lott initially co-authored did cause the results of their main regression to significantly change. Lott initially denied the errors in Plassmann and Whitley’s paper, but eventually did. Even then, Lott tried to downplay their importance and obfuscate their effect in a series of “corrections.”
The “corrected version” Lott currently links to on his website is not corrected. Even after multiple iterations, Lott’s “corrected version” still contains data errors. Instead of correcting mistakes, this version only made the paper less rigorous.
As Ayres and Donohue concluded in their critique of Lott published in Stanford Law Review:
“In the wake of some of the criticisms that we have leveled against the Lott and Mustard thesis, John Lott appeared before a National Academy of Sciences panel examining the plausibility of the more guns, less crime thesis and presented them with a series of figures showing year-by-year estimates that appeared to show sharp and immediate declines in crime with adoption of concealed-carry laws. David Mustard even included these graphs in his initial comment on the Donohue paper in the Brookings book that PW [Plassmann and Whitley] refer to repeatedly in their current response. But Donohue privately showed Mustard as well as the Brookings editors that the graphs were the product of coding errors in creating the year-by-year dummies, and in the end Mustard conceded and withdrew them from his comment on Donohue. Now PW respond to our paper with an array of regressions that purport to support their thesis, but again are utterly flawed by similar coding errors. We previously made no mention of the initial National Academy of Sciences/Brookings comment error, since we know how easy it is to make mistakes in doing this work. But for the second time Lott and coauthors have put into the public domain flawed regression results that happen to support their thesis, even though their results disappear when corrected. Claiming we misread our results in the face of such obvious evidence to the contrary and repeatedly bringing erroneous data into the public debate starts suggesting a pattern of behavior that is unlikely to engender support for the Lott and Mustard hypothesis. We feel confident concluding that we have indeed shot down the more guns, less crime hypothesis. Perhaps PW can now assist in laying it to rest.”
Sources: