Myth-Busting Monday: A Few Brief John Lott Data Manipulation Updates
Among all the recent political noise, gun violence disinformation is still spreading
While global turmoil has largely supplanted gun violence in the headlines over the past few months, that does not mean the purveyors of gun violence disinformation have been silent.
Chief among these disinformation-spreaders is John Lott, President of the Crime Prevention Research Center, whom we have debunked numerous times previously. Here are a few brief updates on some of his recent falsehoods that are receiving traction in the media.
Myth: Armed Civilians Are Stopping Active Shooters
In a widely circulated, self-published paper released in early May, Lott and his co-author, Dr. Carlisle Moody, argue that armed civilians are more effective at stopping active shooting incidents than police and suffer fewer casualties. However, the paper’s conclusions are the product of substantial data manipulation.
Back in 2022, Lott claimed that the Federal Bureau of Investigation (FBI) was missing dozens of cases in which a “good guy with a gun” had stopped an active shooter event. As we reported at the time, Lott achieved this result by covertly modifying the FBI’s definition of an active shooter event. While the FBI uses “active shooting” to refer to attempted mass public shootings that aren’t part of another crime, Lott extends it to all shootings in public that aren’t part of another crime.
Lott then applies his revised definition to only cases involving a defensive gun use (DGU). The result is that Lott deliberately excludes thousands of shootings in which a DGU did not occur from his analysis.
While one can argue whether Lott’s definition is better than the FBI’s there are only two possible conclusions:
The FBI definition is proper, in which case Lott is falsely adding incidents that aren’t real active shootings.
Lott’s definition is proper, in which case Lott is failing to include thousands of applicable shootings that don’t have a DGU.
The end result in either scenario is data fraud.
Lott and Moody’s paper from May builds on this fraudulently constructed data with a statistical analysis finding that civilians are better at stopping active shootings than police. Yet this conclusion is pre-ordained by the underlying data manipulation, but now with a new statistical analysis on top of the foundation.
Sadly, Lott is functionally immune from the consequences of data fraud as the head of his own organization, as well as hesitation by the courts to get involved in what they perceive as academic disputes. However, it is surprising that Dr. Carlisle Moody would participate in this enterprise given that he is still bound by the ethics code of William & Mary (though this is not the first time Moody has provided false data and analysis to support Lott).
A Convenient Switch in Crime Narrative
Crime is surging and the FBI is covering it up… is what Lott was arguing last October in the middle of the 2024 Election. As we reported at the time, these incendiary — and objectively false — claims were universally adopted by right-wing media, the Trump campaign itself, and boosted by Elon Musk, the richest man in the world, on X and with his political PAC.
Now, eight months later, Lott’s tune has changed: “As Deportations Rise, The U.S. Is On Track For The Lowest Murder Rate On Record.” No data has emerged between October and now to account for such a dramatic shift. The only difference is that Donald Trump is now in the White House, Lott takes the FBI at its word, and the fact that the FBI was reporting record drops in violent crime last year are ignored.
The national crime numbers are now useful for Lott’s political narrative, and with no evidence, he is claiming that the years-long downward trajectory in crime is the result of Trump’s draconian immigration actions over the past few months.
Unsurprising, but still shameful.
Surveys of Statistically Rare Events
For the final entry, Lott is correct about a survey, but misses the broader implications.
He is currently on the warpath against a survey that contends 18 million American adults were present at mass shootings. To demonstrate that the survey’s figure is an overestimate, he produces his own similarly worded survey that finds similar topline results, but then asks the participants if they were injured in the shootings. The results indicate that the number who replied “yes” is around 71 times higher than those who were shot during mass shootings.
This is substantial evidence that the original survey’s results shouldn’t be trusted. However, the problem rests not on the surveyors themselves or anything they did wrong with the questions. Rather, it is the direct result of asking about statistically rare events in a survey.
Because of the nature of false positives and false negatives (for a more detailed breakdown, check out Part 3 of our DGU series), these surveys are almost always going to overestimate the rare event. And the rarer the event, the worse the overestimation gets, and this is true for all such surveys.
In particular, and what Lott fails to grasp, is that this artifact of surveys is at the core of the defensive gun use debate. Lott is a major proponent of the claim that there are millions of DGUs each and every year. Yet, these numbers come from surveys of statistically rare events. Indeed, if we applied Lott’s own analysis of this mass shootings survey to those of DGU surveys, even by the most extreme estimate of gunshots in DGU cases (which is by Lott himself), Lott’s DGU numbers are off by at least 62 times based on observable data, and likely much more. Not far off the error rate Lott alleges in the mass shootings survey.
The moral of the story: don’t trust surveys of statistically rare events, unless they are backed up by substantial empirical data. And don’t trust John Lott.
Devin Hughes is the President and Founder of GVPedia, a non-profit that provides access to gun violence prevention research and data.
Photo by Markus Winkler; via Pexels.
You wrote: "As we reported at the time, Lott achieved this result by covertly modifying the FBI’s definition of an active shooter event. While the FBI uses “active shooting” to refer to attempted mass public shootings that aren’t part of another crime, Lott extends it to all shootings in public that aren’t part of another crime."
What sources do you have for the FBI definition?
When I did a quick search, this definition came up on multiple FBI websites: "The FBI defines an active shooter as one or more individuals actively engaged in killing or attempting to kill people in a populated area."
That seems to support Lott's definition much better than your interpretation limiting it only to "mass" shootings.
See https://www.fbi.gov/file-repository/reports-and-publications/2024-active-shooter-report/view
One of the major difficulties with trying to have a meaningful discussion of violence involving guns is incomplete, nonexistent, or biased data sets. And the later is made much worse by the total lack of agreed upon definitions. We lack the most basic of data, the number of deaths by gun annually. The FBI data is based on incomplete voluntary reporting by some but not all law enforcement agencies. And the CDC figures are based on an extrapolation of data collected from a limited and unknown list of hospitals. We also lack accurate data on the total number of functioning firearms in private hands, the number of individuals who own firearms and their demographic make up, household where guns are present and data relating to monthly and annual firearms sales.
As to the rest of the data being used, it is all being gathered and compiled by biased parties. The reality is that the only individual or groups involved in this discussion are from either the pro or anti gun camps. Mainly because they are the only groups interested in the subject enough to spend time doing the research and data collection. Since the two sides lack common definitions for what is being gathered, comparing data sets is often an apple and oranges exercise.
The net result is that both sides have collected a large amount of sometimes conflicting data, upon which they have based conflicting conclusions. Given that both sides are doing this it is somewhat less than genuine to label the conclusions of either side as disinformation.