Myth-Busting Monday
John Lott’s Missing Survey on Defensive Gun Use
This article was originally published on GVPedia.
Summary:
Since 1996, John Lott has claimed that surveys indicate the “vast majority” of defensive gun uses (DGUs) consist of brandishing, but not firing, a gun.
Beginning in February of 1997, Lott began to specifically claim that 98% of DGUs consist of brandishing, but not firing, a gun.
Lott consistently cited Gary Kleck, the Los Angeles Times, and Gallup for this 98% figure. However, after he was notified that his citation was inaccurate by sociologist Otis Dudley Duncan in 1999, Lott began attributing the 98% figure to a survey he claims he conducted in the Spring of 1997.
Lott said he conducted the survey in Spring 1997, yet referenced the statistic from his alleged survey on February 6, 1997, months before he alleges he completed the survey.
Lott has been unable to provide any substantive evidence that he conducted the 1997 survey.
Lott claims he conducted a new survey in 2002 to “replicate” the 98% finding, but there is evidence that Lott has misrepresented those results as well.
David Hemenway, Ph.D., Professor of Health Policy and Director of the Harvard Injury Control Research Center, said of Lott’s survey, “For readers to accept the results requires complete faith in Lott’s integrity, that he will always conduct careful and competent research. Lott does not merit such faith.”
Jon Wiener, a historian at the University of California, Irvine, concluded his analysis of Lott with, “The conclusion seemed obvious: Lott had never done the national survey. He was lying.”
Lott’s Claim:
In a 1996 article in the journal Agenda, Lott said, “… polls of American citizens undertaken by organizations like the Los Angeles Times and Gallup showing that Americans defend themselves with guns between 764,000 and 3.6 million times each year, with the vast majority of cases simply involving people brandishing a gun to prevent attack.” The same sentence appears in a 1997 symposium paper written by Lott in which he cites research by Gary Kleck and Marc Gertz.
Since 1997, John Lott has consistently claimed that 98% of defensive gun uses require only brandishing the gun, not firing it.
During testimony to a Nebraska legislative committee in February 1997, Lott repeated the claim and added that: “There are surveys that have been done by the Los Angeles Times, Gallup, Roper, Peter Hart, about 15 national survey organizations in total that range from anything from 760,000 times a year to 3.6 million times a year people use guns defensively. About 98 percent of those simply involve people brandishing a gun and not using them.”
In the 1998 edition of his book, More Guns, Less Crime, Lott attributes the 98% claim to “national surveys.” Lott writes, “If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.” Lott referred to this statistic more than four dozen times in publicly available sources since 1997.
The Facts:
Lott attributes the 98% statistic to fifteen national survey organizations including the Los Angeles Times and Gallup, but none of the surveys asked about brandishing a gun. Otis Dudley Duncan, a sociologist, explains: “Lott repeatedly made erroneous statements about the findings of other researchers. None of the national surveys that he cited by name actually had any figure at all for merely brandishing or firing. One of them, the Roper survey (which was mentioned in the Feb. 6, 1997, Nebraska testimony) never even did any survey on defensive gun use. Of the polls that did collect data on firing, none of them obtained a figure anywhere near Lott’s 2 percent.
So everything he has said about surveys on this topic done by others is utterly, totally false. There is no room even for reasonable doubt.”
The original source of the 98% statistic probably comes from Gary Kleck. In 1988, Kleck wrote: “over 98 percent … of the one million estimated defensive gun uses… involved neither killings nor woundings but rather warning shots fired or guns pointed or referred to.”
It’s important to note that Kleck’s 98% claim includes firing warning shots and instances when the person fires and misses, not just instances where the firearm is only brandished. Gun rights supporters who cite Kleck’s research frequently omit that the claim included warning shots and missed shots.
Between 1997 and 2000, Lott is one of the writers who appears to omit the important information that Kleck’s research includes warning and missed shots, not simple brandishing. Lott repeatedly attributes to Kleck’s research the claim that 98% of DGUs involve only brandishing firearms. In addition to citing Kleck in the previously mentioned 1997 symposium paper, Lott wrote in a February 2000 article, “Kleck’s study of defensive gun uses found that ninety-eight percent of the time simply brandishing the weapon is sufficient to stop an attack.”
In March 2003, this sentence was deleted after Duncan and other scholars pointed out Lott’s error.
Lott’s Counter:
After Lott was criticized by fellow researchers, he claimed the 98% finding came from his own study rather than from Kleck and other national surveys. In the second edition of More Guns Less Crimes released in 2000, Lott changed this attribution from “national surveys” to “a national survey that I conducted.” In a 2000 article written for The Criminologist, Lott elaborates on his new position, explaining that the 98% figure derives from a study he conducted in the first three months of 1997, surveying a representative sample of 2,424 people.
To support this claim, in an email written to Professor James Lingdren, a law professor at Northwestern University, Lott explains: “I am willing to bet that I don’t start mentioning this [98%] figure until the spring of 1997. If I use it before I said that I did the survey, I will say that they nailed me. But if I only started using it about the time that I said that I did the survey, I think that it would be strong evidence the other way.”
Rebuttal:
Scholar Otis Dudley Duncan accepted Lott’s challenge and discovered a reference to the 98% statistic on February 6, 1997, which was months before Lott claims to have completed the survey.
The evidence overwhelmingly suggests Lott did not conduct the survey and has been repeating data he fabricated.
In a message to Lingren, Lott argued that “The overwhelming majority of the survey work was done at the beginning of the period over which the survey was done. It has obviously been a while, but my recollection is that the small number of people surveyed after the first four or five weeks (mainly January 1997) did not include any more defensive gun uses.”
Lindgren then explains why Lott’s defense is dubious:
“While again this story is certainly possible, Lott himself gave spring 1997 as the time before which he should not have been discussing the 98% figure. Additional matters bear mentioning. It hardly matters whether all of the defensive gun uses were found in the first 4-5 weeks of the study, since Lott could not have known that at the time he spoke about the results unless data collection were complete. If data collection were partial, the precise percentage of defensive gun uses would have been higher with partial data. Collecting so much data in 4-5 weeks would have been unusual for unpaid volunteers who were full-time undergraduate students at the University of Chicago at the time, unless there were a very large number of volunteers.”
“As I discuss below in the section on technical problems with the study,” Lindgren continued, “Lott’s numbers suggest that only half, an 1/8th, or 5/8ths of a respondent reported certain kinds of defensive uses. The partial respondents necessary to support Lott’s percentages would be most likely to result from some extreme demographic weights being applied after the data collection were complete and the results were compared with the characteristics of the adult population. If the study were not complete, it would be very unlikely that someone would have weighted the results against the general population before knowing how skewed his sample was. Such weighting is not easy and would have been a colossal waste of time before data collection were complete, since they would need to be redone at the end of data collection. Last, of course, Lott does not mention that he is reporting partial data in his February 6, 1997 testimony.”
When pressed for evidence of the survey, Lott has claimed all of the following:
His hard drive crashed in June of 1997, erasing all evidence of the survey.
He paid for the survey from his private funds, but has failed to provide receipts. Therefore, no expense information is available to substantiate the fact that any survey was ever administered
Employee records do not exist because the survey was completed by unpaid, full-time undergraduates at the University of Chicago in their junior and senior years.
Telephone records do not exist because Lott instructed the students to use their own telephones. He subsequently reimbursed the students out of his own funds.
He does not remember the names, contact details, or faces of a single student volunteer. Therefore, students cannot be contacted to corroborate their involvement in the survey’s administration.
He does not remember the questions asked on the survey.
He had no discussions with anybody about sampling design.
He did not retain any of the tally sheets because they were lost in an office move in 1997.
After investigating the controversy, Lingdren wrote in a report that “all evidence of a study with 2,400 respondents does not just disappear when a computer crashes.” Lindgren explains that to lose every conceivable form of hard evidence — phone records, funding, tally sheets, potential communication with consultants, records of employees — is essentially impossible:
“Having done one large survey (about half the size of John Lott’s) and several smaller surveys, I can attest that it is an enormous undertaking. Typically, there is funding, employees who did the survey, financial records on the employees, financial records on the mailing or telephoning, the survey instrument, completed surveys or tally sheets, a list of everyone in the sample, records on who responded and who declined to participate, and so on. While all of these things might not be preserved in every study, some of them would almost always be retained or recoverable.”
Lingdren pressed Lott further, asking him how he drew the sample for the survey. Lott explained that he used a CD-ROM, but that he can’t remember where the CD is or how he obtained it. Lingdren suggested he email all the students at the University of Chicago from 1997 to 1998. Lott resisted, however, saying that he had “serious questions about how complete the University’s alumni records are.”
Lott’s response to his critics has been to assemble anecdotal claims from various tangentially related colleagues. Duncan and Lambert investigated Lott’s evidence and discovered written statements that don’t provide proof of Lott’s claims.
For example, one of Lott’s colleagues claimed that, “John told me that he had conducted a survey in 1997.” Notice, however, this is very different from “John told me in 1997 that he was conducting a survey.”
Lott did, however, find David M. Gross who claims to have been interviewed in 1997 as part of the survey. Mr. Gross is a former board member of the National Rifle Association and founding director of the Minnesota Gun Owners Civil Rights Association. Jon Wiener, writing in his 2004 book, Historians in Trouble: Plagiarism, Fraud, and Politics in the Ivory Tower, observed: “It seems unlikely, to put it charitably, that [David Gross] would turn up in a random sample of a few thousand people out of the 300 million Americans.”
Lott also went to great lengths to remind readers that the statistic is a small portion of his book: “The reference to the original survey involves one number in one sentence in my book,” and “there have been many claims that I didn’t conduct a survey in 1997 that was reported in one sentence on page 3 of my book, More Guns, Less Crime.”
One wonders why someone who ostensibly went through a herculean effort to conduct a huge survey with his own money in three months’ time would then marginalize his own work by insisting that it’s only “one number in one sentence” in one book.
Donald Kennedy, the late editor-in-chief of Science magazine, noted that Lott’s many excuses “does not restore life to the data — which, ‘far from being one number in one sentence’, were at the center of controversy between Lott and his critics.”
Lott’s Counter:
According to Lott, he undertook another survey in the fall of 2002 with the goal of validating his original 98% DGU claim.
In his 2003 book, The Bias Against Guns, Lott discusses the new survey, writing: “… the survey I conducted during the fall of 2002 indicates that simply brandishing a gun stops crime 95% of the time, and other surveys have also found high rates.” He claims that 1,015 people were interviewed, but Lott does not report the full survey results in the book.
In the appendix, Lott writes, “Overall the survey results here are similar to the one I conducted primarily during January 1997 which identified 2.1 million defensive gun uses, and that in 98 percent of them, the gun was simply brandished.”
Rebuttal:
Many studies of DGU have been published over the past 30 years. None substantiated Lott’s finding that 98% of DGUs only involve brandishing a gun.
The majority of similar surveys found that rate to be typically in the 60-80% range. In addition, online reports of Lott’s November 2002 study suggest that his own study finds a much lower brandishing rate. In a critical review of Lott’s Bias Against Guns, Harvard Professor David Hemenway writes: “Exactly 1,015 people were interviewed (p. 259). But unfortunately Lott’s book abruptly ends before he gives the survey results. Internet reports say that 7 of his 1,015 respondents claimed a self-defense gun use (13 uses), and in one of these the respondent shot his gun. This is not 98% or 95%.”
Most importantly, “his survey is not nearly large enough to provide precise estimates of the percentage of self-defense gun users who merely brandish the firearm.” Hemenway continues, “In his analyses, Lott virtually always uses complicated econometrics. For readers to accept the results requires complete faith in Lott’s integrity, that he will always conduct careful and competent research. Lott does not merit such faith.”
Lott’s highly questionable loss of all 1997 survey information, his inability to provide credible answers to his critics, and his heavy reliance on complicated econometrics earned him a spot in Jon Wiener’s book, A Human Enterprise: Controversies in the Social Sciences, under the header: “Outright Lies.”
Wiener, a historian at the University of California, Irvine, concluded his analysis of Lott with, “The conclusion seemed obvious: Lott had never done the national survey. He was lying.”
Sources:
John Lott, More Guns Less Crime, 1998
John Lott, More Guns Less Crime (2nd Edition), 2000
John Lott, The Bias Against Guns, 2003
John Lott, “Concealed Handgun Laws Can Save Lives,” Agenda: A Journal of Policy Analysis and Reform, 1996
John Lott, “Does Allowing Law-Abiding Citizens to Carry Concealed Handguns Save Lives?,” Valparaiso University Law Review, 1997
John Lott, “Reply to Otis Duncan’s Recent Article in The Criminologist,” The Criminologist, Sept/Oct 2000
Gary Kleck and Marc Gertz, “Armed Resistance to Crime: The Prevalence and Nature of Self-Defense with a Gun,” Journal of Criminal Law and Criminology, Fall 1995
Tim Lambert, “Does simply brandishing a gun cause attackers to flee 98% of the time?,” ScienceBlogs, Sept. 13, 2002
Otis Dudley Duncan, “Gun Use Surveys: In Numbers We Trust?,” The Criminologist, Jan/Feb 2000
Tim Lambert and Otis Dudley Duncan, “Statements by John R. Lott, Jr. on Defensive Gun Brandishing,” ScienceBlogs, Oct. 17, 2002
Tim Lambert, “Tim Lambert’s comments on the Duncan-Lott exchange,” ScienceBlogs, Oct. 17, 2002
Donald Granberg and John F. Galliher, A Most Human Enterprise: Controversies in the Social Sciences, 2010
David Hemenway, “Book Review,” Harvard School of Public Health, 2003
Otis Dudley Duncan, “Duncan Comments,” ScienceBlogs, Jan. 16, 2003
“GVPedia explains…Defensive Gun Use,” GVPedia University, Feb. 24, 2020
Evan DeFilippis and Devin Hughes, “The GOP’s favorite gun ‘academic’ is a fraud,” ThinkProgress, Aug. 12, 2016
Image by Peter Timmerhues from Pixabay.