Myth-Busting Monday
Myth-Busting Monday
Flawed Research: Lott’s Board Member Carlisle Moody misreads his own analysis.
This article was originally published on GVPedia.
Summary:
A 2011 study by Stanford Law School professor and economist John Donohue and his colleagues attempted to replicate the results of a National Research Council (NRC) review on the effects on crime of right-to-carry (RTC) laws. Researcher John Lott provided NRC the dataset upon which the study was based.
Donohue et al. found flaws in the NRC dataset provided by Lott, and concluded that the most significant effect of concealed carry laws is an increase in aggravated assault.
In an attempt to refute Donohue’s findings, Carlisle Moody, a board member of Lott’s Crime Prevention Research Center, co-authored a paper that claims RTC laws significantly reduce murder, and have no clear effect on aggravated assaults.
Moody’s results, however, support Donohue’s findings that RTC laws increase aggravated assaults. This stemmed from Moody et al. misreading their own data table.
Lott defends Moody from accusations that he misread his own analysis, but Lott fails to address the main points of criticism.
Flawed Research:
In February 2014, Carlisle Moody, a board member of Lott’s Crime Prevention Research Center, and three co-authors released “The Impact of Right-to-Carry Laws on Crime: An Exercise in Replication.”
The paper was in response to a 2011 study by John Donohue and his colleagues that attempted to replicate the results of a National Research Council (NRC) review on the effects of RTC laws on crime. Donohue’s study concluded that there were flaws in the dataset used by the NRC, which was provided by Lott.
Rather than reducing murders, as the NRC study found, Donahue’s paper found that the most significant effect of concealed carry laws is an increase in aggravated assault.
Moody’s paper states, “Once corrected for omitted variables, the most robust result, confirmed using both county and state data, is that RTC laws significantly reduce murder. There is no robust, consistent evidence that RTC laws have any significant effect on other violent crimes, including assault.”
However, Moody and his co-authors misread their own analysis.
As Table 3 on page 7 of Moody’s paper (pictured below) demonstrates, the increase in aggravated assault for county level data is statistically significant, yet is not bolded by the authors like the other statistically significant findings.
In statistics, a result is usually considered significant if there is a less than 5% chance that the result is due to random chance, meaning it has a “t-statistic” greater than 1.96. A significant result means that the authors of a study can put a higher degree of confidence in their finding.
As the table below shows, the “stat” for the “post-law trend” for “Assault” (highlighted with a red box) has t-statistics of 2.8 and 2.25 for the general and specific model respectively. Further, the result itself is a positive number, indicating an increase in assault.
Nowhere in the Moody paper does it explain why significant t-stats are un-bolded, and it remains undiscussed in the conclusion despite the fact that it directly undermines the thrust of their entire paper.
The paper actually supports Donohue’s finding that RTC laws significantly increase aggravated assaults.
Had Moody and his co-authors reported their own results correctly, they would have been left with the puzzling conundrum of concealed carry laws both reducing murder and increasing aggravated assaults. This finding contradicts well-established criminological facts and suggests the paper relies on deeply flawed statistical modeling choices.
It’s worth noting that the paper states, “The authors thank The Crime Prevention Research Center for its support.”
Lott’s Counter:
In an August 2016 post on his website, Lott counters that Moody et al. did not misinterpret their own results as claimed by Evan DeFilippis and Devin Hughes in a ThinkProgress article.
Lott writes, “DeFilippis and Hughes seem to accuse a lot of people of deliberately deceiving others. But DeFilippis and Hughes don’t seem to understand what is meant by ‘robust results, confirmed using both county and state data.’ MMZA weren’t claiming that there wasn’t a single result showing a significant effect for aggravated assaults. What they were saying is that the results weren’t consistent across different estimates.”
Rebuttal:
Lott’s counter deflects from the criticism and does not address the claim that Moody et al ignored a statistically significant finding that undermines their argument that more guns means less crime.
Moody’s paper highlights significant results, which is a finding with a t-statistic over two. Table 3 clearly shows a statistically significant finding that right-to-carry laws are associated with more aggravated assaults. The authors fail to report the significant finding which dramatically alters their conclusion. Lott’s counter does not address these basic points.
Sources:
Carlisle Moody, Thomas Marvell, Paul Zimmerman, and Fasil Alemante, “The Impact of Right-to-Carry Laws on Crime: An Exercise in Replication,” Review of Economics & Finance, February 2014
Abhay Aneja, John Donohue, and Alexandria Zhang, “The Impact of Right-to-Carry Laws and the NRC Report: Lessons for the Empirical Evaluation of Law and Policy,” American Law and Economics Review, Fall 2011
Evan DeFilippis and Devin Hughes, “The GOP’s favorite gun ‘academic’ is a fraud,” ThinkProgress, August 12, 2016
John Lott, “Response To Evan DeFilippis And Devin Hughes’ Newest Claims At Think Progress,” Crime Prevention Research Center, August 31, 2016
Image by Gerd Altmann from Pixabay.