Welcome to the fourth article in Puppycide Database Project's ongoing series, "Statistics are Misleading 100% of the Time". In this article, we pick up where we left off from Part Three in our analysis of the widely-cited claim that "half of all police shootings involve dogs". In order to understand both how the citation became so widely accepted, and whether or not it is accurate, Part Three explained several concepts that are fundamental to statistical analysis, such as representative sampling, bias and probability. For new readers, we suggest starting at least at Part Three in order to gain the benefit of all of our research concerning the "half of all police shootings involve dogs" statistic.
In Part One of our series, PuppycideDB discussed how important statistics are in our attempts to understand trends in the use of lethal force by police. Part Two examined (and ultimately debunked) the statistic that "police shoot a dog every 98 minutes" made popular by the marketing efforts of a recently released documentary film.
When the last article of Statistics are Misleading 100% of the Time ended, we had just completed an initial examination of a pair of factual claims from a Department of Justice whitepaper titled The Problem of Dog-Related Incidents and Encounters that ultimately caused news organizations and activists across the United States to believe that half of all intentional police shootings involve a dog. In that initial examination, we took the pair of claims at face value, but argued that even if correct, they did not provide a representative statistical sample and as such could not be used, on their own, to justify a national trend.
However, further examination reveals that there are good reasons to question the legitimacy of the pair of claims made by The Problem of Dog-Related Incidents and Encounters related to the rate of police use-of-force toward animals. Our argument, then, is two-pronged: first, the numbers used as the basis for the "half of all police shootings involve a dog" statistic are invalid. Second, even if they were valid, they would not prove what the authors of Problem believed they do.
We begin once more by repeating the first of these pair of claims:
"[...] nearly three-fourths of the shooting incidents in Milwaukee from January 2000–September 2002 involved shots fired at dogs, with 44 dogs killed by officers during that period."
The authors of Problem provide a citation for this claim: Analysis of Use of Force Incidents Against a Canine in the Milwaukee Police Department, Jan. 1, 2009, through Dec. 31, 2009 along with a URL to download a copy of the source. The source was prepared by Kristin Kappelman, a Research and Policy Analyst with the Milwaukee Fire and Police Commission. It is not unusual for police departments to publish annual reports, however only a small number of police departments publish public analyses of use of force incidents on an annual basis. In fact, the practice is almost completely unheard of, save for police departments that are required to publish such records publicly as part of an agreement with the Department of Justice following a federal investigation of the department, or as part of an effort to avoid being forced into a consent decree arrangement by Federal investigators. Milwaukee Police Department has repeatedly been the target of federal probes for excessive force and civil rights violations.
Before we examine the numbers in the Milwaukee Police Department claim closely, we must point out an obvious error. The specific phrasing was "shooting incidents in Milwaukee", but the authors of Problem use only Milwaukee Police Department's records. There are several other law enforcement agencies with jurisdiction of Milwaukee, including the Milwaukee County Sheriff's Office and the Wisconsin State Patrol. Milwaukee itself is no backwater, and is currently the largest city in the entire state of Wisconsin, with a population of over half a million as of 2010. Problem never explains why they assume that Milwaukee PD is the only law enforcement agency that shot a dog in the city of Milwaukee from January 2000 through September 2002. Any explanation that could be reliably offered would almost certainly have to include a citation from the other police agencies with jurisdiction. The controversial 2010 shooting death of a pair of Tibetan mastiffs in the city of Milwaukee involved a Milwaukee County Sheriff's Deputy on the scene.
Due to this critical omission, we cannot be sure how many police shootings took place in the City of Milwaukee using only the citations provided by The Problem of Dog-Related Incidents and Encounters. But leaving out other police departments is not the only issue with this citation, or even the most damning one.
The Puppycide Database Project has obtained use of force reports for the Milwaukee Police Department dating back to 2007. However, we do not at this time have access to use of force reports from the years 2000 through 2002, the years The Problem of Dog-Related Incidents and Encounters claims for Milwaukee PD. In fact, the report cited by The Problem of Dog-Related Incidents and Encounters does not contain any reference to statistics for shootings of animals by police for the years 2000 through 2002, which makes sense given the title of the report, Analysis of Use of Force Incidents Against a Canine in the Milwaukee Police Department, Jan. 1, 2009, through Dec. 31, 2009. The report only includes information on shootings of animals between January 2009 and June 30, 2010 (the partial 2010 data is included as part of an appendix).
Puppycide Database Project was unable to determine the reason for the discrepancy. It is not a simple typo that exchanged 2000 for 2009: the statistic cited by Problem notes the final month of data as September rather than June. Furthermore, Problem gets the title of the source correct, which eliminates our suspicion that Milwaukee Police Department uploaded a new report to a link that used to forward to information about dog shootings in 2000-2002. It looks like a typo - but an unfortunate and critical typo.
The Puppycide Database Project reviewed the actual statistics contained in the 2009 Milwaukee PD Use of Force Analysis, as well as Milwaukee PD Use of Force reports for 2010, 2011, 2012 and 2013.
During that time frame, Milwaukee Police Department reported the number of animals killed by their officers was as follows:
|41 dogs 15 deer 2 raccoons 1 goose 1 seagull 1 coyote||34 dogs||37 dogs 23 deer 3 raccoons 1 opossum||37 dogs 10 deer 6 raccoons||27 dogs 24 deer 1 raccoon|
We found issues with conflicting data in the Milwaukee Police Department reports for 2009, as well. At least two reports were filed for the year 2009: Analysis of Use of Force Incidents Against a Canine in the Milwaukee Police Department, which was referenced by the The Problem of Dog-Related Incidents and Encounters. Not cited by Problem was An Analysis of 2009 Use of Force Incidents in the Milwaukee Police Department, which was a more general accounting of Milwaukee's use of force toward humans as well as animals.
Use of Force Incidents Against a Canine reported that in 2009 "there were 46 incidents in which a law enforcement employee of the Milwaukee Police Department used force against at least 1 canine". However The Problem of Dog-Related Incidents and Encounters only listed 43 incidents within the same period: "43 [use of force incidents] involved force being used against one or more dogs". The Problem of Dog-Related Incidents and Encounters also references use of lethal force against "15 deer, two raccoons, one goose, one seagull, and one coyote". The report made clear that "all 20 of these incidents involved the use of a firearm."
So someone miscounted. Whats the big deal? The shooting of these 20 other animals was apparently left out of the calculation of total shooting events. Milwaukee Police reported that close to 75% of the uses of force recorded by their officers involved a dog, which is where the "nearly three-fourths" referenced by Problem most likely comes from. But that number changes significantly if we include other animals; even if we are just counting dogs there is a measurable change in the statistics based solely on which Milwaukee PD report is used. The table below was provided in An Analysis of 2009 Use of Force Incidents in the Milwaukee Police Department - note how the total number of dog-related use of force incidents changes again, to 41 incidents involving some 45 dogs shot.
|Target of Firearm||Frequency||Percentage||Result|
|Dog(s)||39||72.2||43 dogs hit|
|Subject||12||22.2||7 subjects hit|
|Subject and Dog||2||3.7||2 dogs hit; 0 subjects hit|
|Gun Pointed at Subject||1||1.9||--|
|Total Number of Incidents||54||100||--|
Now let's compare the percentages when we add the other animals. Because Milwaukee Police provided no data on incidents in which police shot at other animals and missed, or the number of bullets used to kill the other animals, we can only include the data we have access to.
|Target of Firearm||Frequency||Percentage||Result|
|Dog(s)||39||52.7||43 dogs hit|
|Subject||12||16.2||7 subjects hit|
|Subject and Dog||2||2.7||2 dogs hit; 0 subjects hit|
|Gun Pointed at Subject||1||1.3||--|
|Shootings of Other animals||20||27||15 deer, 2 raccoons, 1 goose, 1 seagull and 1 coyote shot|
|Total Number of Incidents||74||100||--|
Suddenly, the percentage of shootings involving dogs plummets from 72.2% to 52.7%. Alternatively, if we calculate the rate of police shootings involving any animal, we arrive at 82.4%.
Whether the rate increases or decreases is not relevant to our point here. It is certainly not the contention of the Puppycide Database Project that police shootings of dogs are not a significant issue or that the DOJ report overstates the risks encounters with police pose to dog owners.
What is relevant is that the rate of shootings involving dogs, as reported by The Problem of Dog-Related Incidents and Encounters, is wrong. At least two critical law enforcement agencies are excluded. The rate is wrong as is the years used in the rate. At best, the statement "nearly three-fourths of the shooting incidents in Milwaukee from January 2000–September 2002 involved shots fired at dogs, with 44 dogs killed by officers during that period" is not proven by the citation provided. At worst, nothing in the statement is correct.
"Dog bite statistics are not statistics, and do not give an accurate representation of dogs that bite." - American Veterinary Medical Association Task Force on Canine Aggression and Human-Canine Interactions
Next we will consider the second claim used by "The Problem of Dog-Related Incidents and Encounters" as a basis for the statistic under review in this article:
"Information furnished by various California law enforcement agencies indicated that at least one-half of all intentional discharges of a firearm by an officer from 2000–2005 involved animals."
We already discussed how, even if correct, this statement could not be used on its own as the basis for a national trend. But is the claim true?
The citation for this second statement comes from a 2007 article written by Lisa Spahr titled “The Canine Factor: To Shoot or Not to Shoot”. Spahr's article was published by the Police Executive Research Forum as part of their monthly newsletter, "Subject to Debate". According to her LinkedIn page, the article was published during an 18 month period during which Spahr was employed by Police Executive Research Forum as a Research Associate. Spahr has also worked for a private military contractor in the missile defense sector, and now works in management for a technology company.
The Police Executive Research Forum provides the following summary of its mission on its website:
"The Police Executive Research Forum (PERF) is an independent research organization that focuses on critical issues in policing. Since its founding in 1976, PERF has identified best practices on fundamental issues such as reducing police use of force; developing community policing and problem-oriented policing; using technologies to deliver police services to the community; and evaluating crime reduction strategies."
The Board of Directors of PERF is composed entirely of Chiefs of Police. PERF's Research Advisory Board is slightly more diverse and includes individuals from academia as well as law enforcement. However, nearly all of those on PERF's Research Advisory Board not currently employed by law enforcement were employed by law enforcement or the military at some point. Remember how we talked about bias earlier? Having a direct professional or otherwise financial stake in the outcome of your own research is the essence of bias (imagine for a moment that the Puppycide Database Project were staffed entirely by Labradors).
We aren't trying to be mean to PERF - in fact, PERF was incredibly generous in providing us with back issues of "Subject to Debate" completely without charge, allowing us to complete our research for this article. Furthermore, we read Lisa Spahr's 2007 article in "Subject to Debate", The Canine Factor: To Shoot or Not to Shoot, and we have no quarrel with the major points in the article. The Canine Factor is well thought out and well argued. Spahr took a brave and controversial position, and it speaks highly of both her and PERF that it was published in spite of PERF's close relationship with law enforcement managers. We highly recommend those interested in the topic of law enforcement violence toward animals obtain a copy of the article.
Unfortunately, The Canine Factor: To Shoot or Not to Shoot also includes the "Half of all police shootings involve a dog" statistic made famous by The Problem of Dog-Related Incidents and Encounters.
Here's what Spahr's article says (emphasis added):
"One news article, citing information furnished by various California law enforcement agencies, reported that at least one-half of all intentional discharges of a firearm by an officer in 2000–2005 involved animals. The same news article reported that two sheriff departments recorded 162 animal shootings in a period of four years (2000–2004). Little more is known about state and national statistics on this topic."
Again we have a citation for this claim; and in fact, this is a new citation not included in The Problem of Dog-Related Incidents and Encounters. This was an immediate red-flag for PuppycideDB researchers. Usually, it is a questionable practice to use a citation of a citation. If you come across a statistic you want to use in one source that itself references another source for that statistic, the preferred practice is to track down the original source. In research, we categorize sources based on how close they are to original documents. Original documents are called primary sources and are distinguished from secondary sources, which comment on or are based on the research of primary sources.
When we are talking about killings of animals by police, a primary source would be a video tape of that killing, a police report or an original interview with an eyewitness. Puppycide Database Project is typically an example of a secondary source, because we compile videos, interviews and police reports and comment on them, count them and create statistics based on the results of our counting. A newspaper article that talks about an incident in which police have killed an animal can vary between being a primary, secondary or even a tertiary source - sometimes in the same article!
Let's consider an example to understand why newspaper articles can manage to fit into multiple source categories simultaneously. Imagine a newspaper article that includes the following elements:
- A photograph of a police officer shooting a dog, recovered from a camera worn by that officer. This is a primary source.
- The reporter has performed a records request and obtained police reports of all of the times the same police department has shot dogs that year. The records themselves are primary sources, but because they are hundreds of pages long the reporter only includes the total number of times that police department has shot dogs in that year. The article's count of police shootings is a secondary source.
- For our purposes, we won't be concerned with tertiary sources, which are usually encyclopedias, dictionaries and similar references.
So what's the big deal with citing a secondary source? Why does the APA Style Guide say they should be "kept to a minimum" and used "only if you are unable to find or retrieve the original source of information"? When you use a secondary source, you must rely on the competency and validity of the work done to acquire that source. You have to believe that your secondary source performed the translation correctly, did not miss count, correctly dotted every "i" and crossed every "t", never fudged a number on the calculator. That is an enormous amount of faith to put in someone.
The problem is compounded when secondary sources continue to cite more secondary sources, creating a chain that grows farther and farther distant from the original source of information or research. A work that engages in this practice tells his readers "I heard this from a guy who heard it from a guy who heard it from a guy". Would you invest your money in a stock tip obtained using such a reference? If not, why would you invest your intellect in a similar swindle?
It is important to keep this in mind as we explore this next - and final - source for the "half of all police shootings involve a dog" claim. At this point we have found the statistic relies on a report (The Problem of Dog-Related Incidents and Encounters) that cites a secondary source (The Canine Factor), which in turn cites a secondary source.
To be fair, when we checked on the reference list provided for The Canine Factor, we found that the citation in question was both behind a newspaper paywall and that both the name of the article and the year it was listed as published were incorrect. These issues could very well account for why The Problem of Dog-Related Incidents and Encounters did not include the original source.
Using the newspaper's archive system requires either the correct title or the correct year to be effective. The Canine Factor lists that source as: Quan, D., “Pet owners find numbers alarming; officers say they act to protect lives.” The Press-Enterprise, November 6, 2006. The Puppycide Database Project was only able to locate the article cited under the date November 6, 2005 and with the title: "Police Shootings: Guns Often Fired at Animals // Canine Casualties // Pet owners find numbers alarming; officers say they act to protect lives" (the "//" were included in the article's listing in The Press-Enterprise).
So what did the original source have to say that got this started?
"When police officers intentionally fired their guns over the past five years [2005-2000], they were aiming at animals at least half the time, according to records from four Inland [Empire California] police agencies." (data in brackets added)
There is no mention of a national trend. The author, Douglas Quan, clearly specifies the statistics he obtained (from primary sources) are related to four police departments. There are 18,000 law enforcement agencies in the United States.
A sample of four departments from California, even with backup from one department from Milwaukee, is not a representative sample of national trends. A sample of four is not even representative of national trends in the state of California. By contrast, the Puppycide Database Project has complete, multiple-year use of lethal force data from nineteen police departments in the state of California and we have refrained from publishing claims about state wide trends.
Let's take a look again at Lisa Spahr's reading of what Douglas Quan just said in her article The Canine Factor:
"One news article, citing information furnished by various California law enforcement agencies, reported that at least one-half of all intentional discharges of a firearm by an officer in 2000–2005 involved animals.
And finally, let's return to The Problem of Dog-Related Incidents and Encounters:
"In most police departments, the majority of shooting incidents involve animals, most frequently dogs. [...] Information furnished by various California law enforcement agencies indicated that at least one-half of all intentional discharges of a firearm by an officer from 2000–2005 involved animals."
If it had stopped there, the issue would be a small one - worthy of a correction note. But shortly after The Problem of Dog-Related Incidents and Encounters was published, the "statistic" became gospel. Over the last eight years it has been reproduced by the most trusted news and animal welfare organizations in the country.
Strangely, some of these organizations have claimed that they are the source of the statistic. One such organization is the American Society for the Prevention of Cruelty to Animals (ASPCA).
The following is an excerpt from the ASPCA's Position Statements on Law Enforcement Response to Potentially Dangerous Dogs:
Perhaps when the ASPCA refers to "our review of public records of firearms discharges by police", they in fact mean their review of the articles we have already mentioned in this post. But ASPCA's choice of words makes it sound like they have conducted their own investigation, using a their own records requests to law enforcement agencies across the country. The ASPCA has never published data related to any such survey, and has provided no clarification as to which law enforcement agencies, or even how many, were included in the survey they allude to in their position statement. It is not simply that the ASPCA has refused to publicize their data or submit that data for peer review. ASPCA's statement lacks even the most basic information about their "study". Nor was ASPCA listed within The Problem of Dog-Related Incidents and Encounters as being involved with the DOJ whitepaper. Between this and the use of identical claims as published in previous articles, it without further clarification from ASPCA it appears that Problem or a related publication was their source of their position statement and they did not, in fact, review any significant number of law enforcement use of force records - certainly not enough to justify the claims in their Statement.
Like the ASPCA, other authors have recycled the claims made by The Problem of Dog-Related Incidents and Encounters and The Canine Factor. The incorrect data referred to in these publications are in turn cited by more news reports, creating a closed loop that confirms and re-confirms the "half of all police shootings" claim. Many of the more recent references to the "half of all police shootings involve dogs" canard make no reference to the DOJ COPS whitepaper or Lisa Spahr's article.
"A recent review of public records by the American Society for the Prevention of Cruelty to Animals concluded that almost half of all firearms discharges by police officers involve the shooting of a dog." - National Review
"Research by the American Society for the Prevention of Cruelty to Animals suggests that half of all police firearm discharges involve the shooting of a dog." Washington Post
"Roughly half of all firearms discharges by police officers involve shooting a canine, according to an ASPCA review of public records." Business Insider
Even when news sources make an apparent effort to dial back claims of national trends, they still manage to get the facts wrong. NBC 12 made the following claim while citing The Problem of Dog-Related Incidents and Encounters:
"In addition, a dog training manual for police published by the Department of Justice showed that most police shooting incidents involve dogs. In Los Angeles, at least one-half of all 'intentional discharges of a forearm by an officer from 2000-2005 involved an animal,' the study said."
Its unclear how NBC settled on the city of Los Angeles, because The Problem of Dog-Related Incidents and Encounters refers only to the state of California, while the original newspaper article Problem itself inevitably relied on for its data by Douglas Quan used police records from San Bernadino and Riverside, each of which are both cities and counties, and both of which are located east of Los Angeles in the Inland Empire metropolitan area of California.
There will be one more brief installment of the "Statistics are Misleading 100% of the Time" series, a Conclusion. In the Conclusion Puppycide Database Project will briefly review our findings over the last four articles, and discuss the impact that the prevalence of the statistics we have discussed has had on the national conversation concerning the use of force by police.