Disinformation Is Once Again in Front of the Supreme Court

John Lott and Carlisle Moody’s latest brief relies heavily on their own flawed research.

Nov 29, 2023

By: Devin Hughes

Earlier this month, the Supreme Court heard oral arguments in United States v. Rahimi, a case that cites the Second Amendment to overturn the prohibition of firearms for those under Domestic Violence Restraining Orders (DVROs).

The Crime Prevention Research Center (CPRC), led by John Lott, has filed an amicus brief filled with disinformation, just as it did in the landmark 2021 case, New York State Rifle & Pistol Association v Bruen.

In his brief, Lott and his colleague, Carlisle Moody, argue that the gun prohibition attached to DVROs should be struck down as there is no empirical data that such measures save lives, an argument that is in stark contrast to the majority of academic literature on the subject.

Lott and Moody’s brief criticizes reliable, peer-reviewed research, brings in gun policies unrelated to DVROs, and relies heavily on their own flawed research.

First, Lott and Moody argue that:

“While gun control advocates argue that gun bans won't work unless you ban guns everywhere in a country, every single time around the world that countries banned all guns or all handguns, murder/homicide rates have risen. That has been true even in island nations with no neighbors supplying guns. One would think that out of randomness, one time, they would go down or at least stay the same.”

While it is unclear what complete gun bans have to do with a case about Domestic Violence Restraining Orders, what is clear is that Lott and Moody’s claim is objectively false.

As GVPedia has previously detailed, a case study of Japan from 1946 banned the possession of firearms and swords by private citizens in principle, although the possession of hunting guns and artistic swords was permitted with a license. In 1958, changes were implemented to prohibit carrying guns and swords, regardless of whether the carrier was licensed. Murders in Japan have fallen by more than 75% since 1955, with no upward spikes as the chart below reveals.

A 2006 study estimated gun ownership in Japan at 1 in 175 households. Today, Japan has one of the lowest homicide rates in the world, and gun violence is exceptionally rare. In 2017, only three people were fatally shot. Lott and Moody’s statement, “every place that has banned guns (either all guns or all handguns) has seen murder rates go up,” is disproved by Japan.

Also, the Solomon Islands was in the middle of a civil war when its regulations banning firearms were passed, but once the war was over and the island nation was able to enforce its ban, even Lott’s own data on the Solomon Islands shows a substantial reduction in violence.

Lott and Moody go on to state that:

“For example, while the largest-ever survey of 120 academics on gun control found that academics were overall quite skeptical that gun control lowered murder rates or the frequency of mass public shootings, the differences between criminologists, economists, and public health researchers were stark.”

GVPedia has also proven this statement completely false.

Lott’s survey of experts is not the largest of its kind. A May 2014 Harvard Injury Control Research Center survey about firearms and suicide was completed by 150 firearms researchers, including Lott. In contrast to Lott’s survey of 120 academics, 84% of respondents to the Harvard survey agreed that having a gun in the home increases the risk of suicide.

A July 2014 follow-up survey about concealed carry laws was completed by 140 researchers, with a majority believing more permissive concealed carry laws have not reduced crime rates. In subsequent follow-up surveys, a similar or greater number of researchers completed the questions than participated in the Lott survey. Furthermore, the Harvard study found much greater support for gun violence prevention policies than Lott’s study.

In order to obtain a clear sense of Lott and Moody’s statistical analysis of DVRO laws, I asked three leading experts in the field for their thoughts: Dr. Cassandra Crifasi and Dr. Daniel Webster of Johns Hopkins University, and Dr. John Donohue of Stanford University.

All of them responded, pointing out a virtual cornucopia of errors in Lott’s analysis. I’ve included their full responses at the end of this article as they delve into complex econometrics, but I’ll summarize my key takeaways here.

Lott and Moody badly mischaracterize previous econometric literature, which leads to them choosing an econometric model that is known to be weak. Further compounding this is using data that is known to have major holes, rather than a corrected version of the same data set. Finally, their methodology for choosing control variables is quite suspect.

Combined, the weak modeling choices, poor methodology, and improper data set render their analysis functionally useless in determining whether DVRO gun restriction laws reduce violence.

With these massive flaws and weaknesses, along with objective falsehoods earlier in the brief, it is far better to rely on existing peer reviewed research, which finds that such orders save lives. Hopefully Lott and Moody’s falsehoods will have no bearing on the Supreme Court’s ruling, but such hopes are best not left to chance.

Appendix:

From Dr. Cassandra Crifasi:

My quick take:

They used standard SHR data (rather than imputed) which is known to have significant issues with missing data.
They appear to have considered policy variables (1970-2023) in years where they didn't have outcome data (1984-2018).
They used TWFE with negative binomial which is a weak analytic method.
The way they operationalized the IV makes little sense.

I would say this is generally on par with his usual approach to analyzing data.

From Dr. Daniel Webster:

What is really weird is that they are measuring the strength of a DVRO gun restriction law based on its duration. They are critical of the prior PH studies because we treated DVRO and specific provisions using dummy variables. Yet that is exactly what they do with the federal DVRO policy. But for state policies...

""We measure the strength of state domestic violence protective orders (DVPO) by the duration of the order... we had to make several assumptions. For example, since we don’t know the respondent’s age or the probability of the order being appealed or otherwise reduced or rescinded, we assume that a permanent order lasts 25 years. Since the numerical values are necessarily arbitrary, we did robustness checks with these long-duration order values doubled and halved to see if the results changed. (The results are not sensitive to these assumptions.)"

I just don't get that. That, alone, may explain why their findings are at odds with our (Zeoli et al. 2018) and other studies.

Just highlighting below the part of the assumption in Lott's models that is way out of line with reality and logic. I can only conclude that he gives a value of 1 the first year a DVRO firearm state prohibition is in place and ramps up to 25 the 25^th year it is in place. So the protective effect in the 25^th year is 25 times larger than in the first year.

Well, the number of people eligible for a DVRO prohibition each year does not ramp up in a linear fashion in this way. DVROs are temporary in nature and usually expire within 12 months. Some of the orders are renewed, but most are not. So, unlike with DV misdemeanor firearm prohibitions — which could be viewed to increase exposure every month/year the prohibition is in place as more people are prohibited — with DVRO firearm prohibitions there may be a slight increase in people exposed/protected by a DVRO firearm restriction from year 1 to year 2, because DVRO restrictions expire with the orders.

In other words, the yearly change in people restrained/protected should only increase in an appreciable way in years 1 and maybe 2 — much more like the 0/1 dummy that Zeoli et al. and most other studies estimate DVRO firearm restrictions.

From Dr. John Donohue’s team:

This amicus brief in support of Rahimi draws heavily upon a forthcoming paper that attempts to estimate the benefits of banning individuals who are subject to domestic violence protection orders from possessing firearms, by regressing SHR homicide data for 1976-1980 and 1984-2018 on the interaction of federal law with state law variation.

In the brief, the authors start by reasoning that people under civil restraining orders are unlikely to obey restrictions on possession: “Someone who is willing to commit a serious assault or murder is already facing a significant prison sentence, a life sentence, or the death penalty. The additional penalties for illegally obtaining a gun or violating a protective order are unlikely to provide marginal deterrence… Thus, the law is most likely to restrain only those who are most law-abiding and fail to restrain the most dangerous.”

They also mention the rise in murder rates after handgun bans in Chicago and Washington, D.C., as well as the fact that “every single time around the world that countries banned all guns or all handguns, murder/homicide rates have risen… even in island nations with no neighbors supplying guns.”

Finally, they mention that the Peltzman effect might reduce the benefits of restraining order possession limitations: “enacting safety measures can cause people to engage in offsetting behavior,” citing seatbelts/airbags and COVID-19 awareness campaigns increasing risk-taking.

The authors go on to say that current literature neglects causality, focuses on state firearm surrender laws rather than federal policy, and tends to use a simple dummy variable rather than accounting for variation in stringency. They code state domestic violence protective orders (DVPO) by the duration of the order, and create a dummy for federal law 18 U.S.C. § 922(g) with value zero before 1994 and value one afterward. Since the federal law has no impact in the absence of a state law, their coefficient of interest is that of the interaction term between the state order duration and the federal law dummy — the additional effect of § 922(g)(8) given an existing state protection order.

In the first econometric faux pas of the brief, Moody and Lott claim that “The de Chaisemartin and D’Haultfoeuille criticism does not apply to our model because the test variables are not simple dummies, instead they are the product of a continuous variable (state order duration) and a dummy for 1994.”

This appears to be a gross misunderstanding of the contribution of de Chaisemartin and D’Haultfoeuille, which focuses not on the binary nature of treatment, but on the elimination of contaminated two-way comparisons from the ATE. In fact, de Chaisemartin et al. (2023) and Callaway et. al (2021) extend their methods for estimation of dynamic effects under staggered adoption to continuous-treatment settings. Since it is reasonable to expect the effect of the (interaction between state and) federal law to increase over time, as more restraining orders are enacted (particularly the permanent ones), these new techniques seem particularly relevant here.

Further, the authors justify the use of a negative binomial model because their outcomes — domestic homicides, domestic homicides of women, domestic homicides committed by firearms, and femicides by a domestic partner using firearms — are count variables.

However, Wooldridge (1999), Allison (2002), Greene (2005), and Guimarães (2008) have found that “fixed-effects” negative binomial estimators do not in fact absorb time-invariant variables, have no known robustness properties, and have thus long fallen out of favor. Perhaps the authors should instead attempt a Poisson approach, which can be robust to serial correlation as well as Overdispersion.

Third, the authors use the “general to specific” modeling methodology, first running a regression with many potentially relevant controls, then retaining only the significant ones to increase the precision of their estimates. Compared to principled covariate selection based on existing literature, this procedure is not empirically responsible or theoretically defensible, and it preserves different controls in the specific models in Table 2 (domestic murder and domestic femicide generally) and Table 3 (domestic gun murder and domestic gun femicide).

The Table 2 specific model controls for prisoners per capita, construction employment, alcohol consumption, the Fryer crack index, the poverty rate, lagged FS/S ratio, and percent of the population that are Black men aged 15-34. The Table 2 general model includes the aforementioned as well as police per capita, effective abortion rates (extrapolated from Donohue-Levitt data for 2015-2018), unemployment, employment, military employment, density, income, real welfare payments per capita, and percent aged 15-34.

The Table 3 specific model controls for alcohol consumption, the Fryer crack index, the poverty rate, lagged executions per capita, and percent of the population that are Black men aged 15-34. The Table 3 general model includes the aforementioned as well as police per capita, prisoners per capita, effective abortion rates (extrapolated from Donohue-Levitt data for 2015-2018), unemployment, employment, military employment, construction employment, density, income, real welfare payments per capita, lagged FS/S ratio, and percent aged 15-34. Besides, the standard errors on estimates produced by the “specific” model are not even consistently smaller than those of the general model.

Ultimately, Moody and Lott find that neither the application of the federal restriction to temporary/emergency orders (median 14 days) nor final orders (median 1 year) has a preventive effect on domestic homicide, domestic femicide, domestic gun homicide, or domestic gun femicide. They do misinterpret the incident rate ratios in Table 2, saying, “For example, the coefficient on final order years… is .993 indicating that a one-year increase in the length of the order would reduce domestic murders by .007 percent,” when it actually means a reduction of 0.7%.

Nonetheless, their results are robust to specifying outcomes as per capita rates rather than counts, dropping 1976-1980 to run a balanced panel from 1984-2018, and using different sentence lengths for ambiguous permanent restraining orders. However, failing to control for the number of each type of restraining order issued strikes me as a major oversight, since that is the channel through which domestic violence may be avoided.

If these orders were less commonly granted after the federal law went into effect, because judges were reluctant to infringe upon Second Amendment rights, their protective effect might not be observable in the bottom-line homicide frequencies.

Again, the highly questionable econometric “choices” made in the forthcoming paper and “justified” in this brief — standard TWFE in a staggered setting, negative binomial rather than the better-understood Poisson, and willy-nilly covariate selection — may drive the surprising null finding rather than achieve an unbiased estimate of the true effect of § 922(g)(8).

Devin Hughes is the President and Founder of GVPedia, a non-profit that provides access to gun violence prevention research and data.

Image by Mark Thomas from Pixabay.