Statistics are Misleading 100% of the Time (Part 2)

This post is Part Two of Puppycide Database Project's ongoing series "Statistics are Misleading 100% of the Time", in which we review - and ultimately debunk - several statistics about police shooting dogs that are widely cited as fact by the most trusted news sources in the world. In Part One of the series, we discussed the important role that statistics play in helping the public understand the complex issues surrounding the use of lethal force by law enforcement. (Click here if you are looking for Part Three, in which we investigate the claim that half of all police shootings involve dogs)

In Part Two, PuppycideDB researchers investigate the claim that "Every 98 minutes a dog is shot by a police officer". We look at how the statistic first found its way to the national press with the help of a pair of young documentary film-makers, and eventually became one of the most widely used references concerning police shootings of dogs in the United States. Newspapers, magazines and nonprofit groups continue to repeat the "98 minutes" statistic: but is it true?

"There are three kinds of lies: lies, damned lies, and statistics." - Unknown

Just a little over two years ago, a Kickstarter campaign was started in order to raise funds for a proposed documentary. The project was to be called Puppycide: The Documentary and would be a one hour long feature film.

The Kickstarter campaign was created by a California-based company called Ozymandias Media, which consists of Michael Ozias and Patrick Reasonover. The claim that a dog is shot by police officers every 98 minutes featured prominently in the film's fund-raising efforts. Ozymandias Media asked for $100,000 to complete the film.

Four days after the start of the first Kickstarter campaign, Breitbart News featured an Op-Ed that carried both Ozias and Reasonover on the byline. The pair used the opportunity to share the 98 minute statistic with potential viewers:

"Police officers unnecessarily shooting dogs has become a silent epidemic. It’s called 'puppycide,' and every 98 minutes there’s another victim".

No source for the claim was provided.

The Kickstarter raised a tidy sum, but not enough to meet the $100,000 goal. The Kickstarter campaign failed on November 15th, 2013 after having raised $60,397 from 958 backers. When a Kickstarter campaign does not achieve its' fund-raising goal, each contribution was refunded to backers. Ozias and Reasonover were back to square one.

Following the disappointment of the first campaign, but perhaps encouraged by their now-proven ability to quickly gather tens of thousands of dollars, Ozias and Reasonover decided that they could make a shorter film with less than $100,000. A week after their first Kickstarter failed, the pair launched a second campaign. This time, they asked for only $40,000, but the film would only be half as long. Just like the first campaign, the claim that a dog was killed by police every 98 minutes was front and center in the promotional fund-raising on their Kickstarter page.

Puppycide Documentary Kickstarter 98 minutes

The second Kickstarter campaign exceeded its goal, raising $45,463 from 693 backers. Those who provided funds were offered incetives based on the size of their contribution: everything from an Executive Producer credit on the film to merchandise like handkerchiefs and sweatshirts. 652 backers were to be provided with a physical copy of the film.

Following the success of the second Kickstarter campaign, the media coverage surrounding Puppycide: The Documentary accelerated. The claim that a dog is killed by police every 98 minutes started to undergo a subtle and gradual change, as well. Its one thing to be raising funds for a film, but now Puppycide: The Documentary was greenlit.

But after Puppycide: The Documentary began production, something strange began to happen. In news coverage mentioning the documentary before the film completed its fundraising efforts, "Every 98 minutes a dog is shot by police" was pretty clearly the tagline for a film. For 1979's Alien, the tagline was "In space, no one can hear you scream." The 1931 classic Frankenstein: had "A monster science created - but could not destroy!" And Puppycide: The Documentary used: "Every 98 minutes, a dog is shot by law enforcement".

After the Kickstarter succeeded, "98 minutes" stopped being used in articles about Puppycide: The Documentary and started being used in articles about dogs who had just been shot by police, or about the phenomenon of puppycide. News organizations began saying the source for the "98 minutes" statistic were not filmmakers, but "animal rights activists" (International Business Times) and eventually an "animal care group" (NBC Miami). The popular liberal news website AlterNet called the film's production company, Ozymandias Media, an "independent research group". There was no mention of any documentary in AlterNet's article.

The statistic ceased to be included in articles about the film "Puppycide: The Documentary", and instead was used in hard news stories reporting animals killed by police.

The appearance of the "98 minute" statistic in AlterNet is a point of interest in itself. Both Patrick Reasonover and Michael Ozias have been described as libertarians in interviews, which may explain why they chose the conservative-leaning Breitbart News to write an Op-Ed piece and why a glowing interview with Reasonover appears in the famed National Review (to their credit, the National Review declined to publish the "98 minute" statistic - but they also declined to ask about it, opting instead to gushingly refer to Reasonover as a "happy warrior"). AlterNet is very much on the opposite end of the of the political spectrum, and spared the filmmakers from any form of partisan sniping. Commentators and news outlets of every political stripe have uncritically repeated the "98 minute" statistic. Ozymandias Media's Kickstarter page has become a trusted, bipartisan source for news media across the country.

This transformation is all the more remarkable as it occurred without the film ever having been released. The documentary has still not been shown to the public or made available for public purchase. The second Kickstarter campaign announced an estimated release date of December, 2014. But come December the only item released was an announcement that the name of the film had been changed from Puppycide: The Documentary to Of Dogs and Men. A new website and a re-vamped trailer accompanied the announcement.

It would be another 11 months before the film premiered on November 1st, 2015 at the Austin Film Festival. As of the time of this writing on December 10th, even the 652 backers who were promised a physical copy of the film by the end of 2014 have yet to receive one, according to the film's Kickstarter page (a smaller number of backers received sweatshirts and stickers in April 2015).

So what is the source for the claim published by Michael Ozias and Patrick Reasonover that a dog is killed by police every 98 minutes?

There does not appear to be a source.

"Statistics are used much like a drunk uses a lamppost: for support, not illumination." - Vin Scully

The day before the editorial that Ozias and Reasonover wrote went to press in Breitbart News, the pair were interviewed by The Atlantic. This was the first time that the 98 minutes claim appeared in a major media outlet that Puppycide Database Project was able to identify. Ozias and Reasonover frankly admitted that they had done no original research before manufacturing the 98 minute statistic:

"We’re planning on doing a lot of Freedom of Information Act requests," Reasonover says. "We think it’s happening way more than the statistic we have [...] We think we can get the data by looking at firearm discharge reports. There might not be a box on reports that says, 'Check here if you shot a dog,' but they will probably include information like, 'I discharged my firearm at a dog.' We’d like to show the scale."

It is clear the speculative tone of Reasonover's quote that he has never even seen a firearm discharge report. The journalist who penned this article in the Atlantic - Mike Riggs - found no apparent issue with publishing the statistics Ozias and Reasonover offered without research or even a citation justifying those statistics. Riggs writes:

"Filmmakers Patrick Reasonover and Michael "Oz" Ozias hope to nail down a rough estimate as part of their research for a documentary called Puppycide."

Just a few sentences later:

"'We think it’s happening way more than the statistic we have.' That statistic, which sits at the top of Puppycide's [The Documentary] Kickstarter page: 'Every 98 minutes, a dog is shot by law enforcement.' Activists came up with that number after tallying accounts of dog-shootings from news stories across the country."

The article provides no clarification as to who those activists are - and The Puppycide Database Project was unable to find any examples of activists stating that a dog is killed by police every 98 minutes before Ozymandias Media's Kickstarter. It is possible that such a claim circulated throughout some ephemeral series of social media posts that are no longer available, but that hardly adds credibility - quite the opposite, in fact. All future references for the 98 minutes claim that provide a citation use either Ozymandias media promotional materials or an interview with Ozias and Reasonover as the source.

We made several attempts to contact Ozymandias Media via the email address posted on their website, beginning over a year ago. In our initial contact, we offered to provide Ozymandias Media with access to our research. We never received a response.

Meanwhile, there has been no slow-down in the use of the 98 minutes statistic. If anything, the reliance on its use by the media and activists is increasing. The Puppycide Database Project found it repeated dozens upon dozens of times - by animal welfare organizations like the National Animal Interest Alliance, on television programs like Inside Edition, in centrist policy journals like the National Journal, inside of technology periodicals like WIRED Magazine, in dog breeder magazines and on environmental websites.

Rarely if ever have we seen the statistic provided to readers critically, it is always presented as the result of some vaguely specified investigation. In the few instances where journalists express some reservation about the accuracy of the "98 minutes" claim, it is only to offer an equally-dubious statistic. In an article with the deliberately (and questionably) provocative title "Why People Care More About Pets than Other Humans", Wired Magazine offered the "98 minute" statistic only to contrast it's resulting annual rate of animal killings with a number proposed by Merritt Clifton, editor of Animals 24-7. Clifton commits essentially the same offense as the Puppycide Documentary filmmakers, attributing 300-500 annual killings of dogs to police based on un-sourced and un-published "analyses of media reports". Wired than goes on to state that if Clifton were correct, it would mean that animals are killed at an annual rate that is "about the same as human cop shootings". Their source for human cop shootings? The totally inaccurate and thoroughly debunked FBI accounting of police justifiable homicides.

Wired Magazine Why People Care More About Pets than other Humans Hal Herzog

The producers of "Of Dogs and Men" have a clear-cut fiduciary interest in drumming up media attention for their work. Each uncritical news story that uses the film as a source is providing the producers with a rave review: even better, they suggest that Ozias and Reasonover's film is the summation of socially relevant research and analysis. By contrast, few newspapers would be interested in a headline like "Film makers confirm cops shooting dogs is really no big deal". When a researcher has a financial stake in a specific outcome of their research, we must pay particular attention to the proof of their claims. Ozias and Reasonover have no proof, so we must disregard their claims - (or at least, the "98 minute" claim - there may be other more specific claims in the film that have been thoroughly researched).

That is not to say that we should ignore the individual stories that Ozias and Reasonover no doubt share through their film: stories of families whose lives have been up-ended as the result of police violence that could very likely have been avoided. We do not wish to give readers the idea that Of Dogs and Men is a bad film, or that it has a bad message, or that Ozias and Reasonover did not put an incredible amount of work into making their film. To the contrary, making any film is an extraordinary undertaking that requires skill and determination. Of Dogs and Men could be a great movie, but its producers are not a great source of statistics.

Films have a powerful ability to communicate ideas and emotions. Movies allow us to travel through space and time, limited only by our imagination. Despite this, there are some things that films are simply not all that good at and that are important.

Films can't do your taxes.

Films are absolute rubbish when it comes to clipping your toe-nails.

And as it turns out, films are no good at math, statistics or long-term research that requires tedious fact- and error-checking.

No matter how compelling the story, or how mind-blowing the special effects, no matter how much we believe or how much we want the end result, no matter how much what we are told fits with what we think ought to be right, movies can't make what is make-believe into what is true.

Of course, all of this leaves one last question un-answered. We established that "Every 98 minutes a police officer shoots a dog" was made famous by a pair of young filmmakers. We also established that this claim was offered with no research or analysis with which to confirm it.

But is it true?

Is a dog really shot by police in the US every 98 minutes?

"Statistical thinking will one day be as neccessary for efficient citizenship as the ability to read and write." - HG Wells

The answer - the only true and honest answer that anyone can provide to this question as of the time of this writing is this: We don't know. No one knows.

There is no complete accounting of the number of dogs shot by police in the United States. No federal government agency requires police to let them know each time they kill a dog. No federal government agency requires police to let them know each time they kill a human being. Puppycide Database Project has a lot of work to do before we can be sure that we can account for every killing.

And it may be impossible to account for every killing. We have learned through our research that police departments across the country regularly destroy lethal force records after a short period of time - typically between 3 to 5 years. We suspect that at least a few police departments or officers within those departments manipulate, hide or even destroy lethal force records. We suspect this because it has been proven this occurs with violent crime records, because police departments have all of the same incentives to manipulate use of force data and because in many instances it would be easier to manipulate use of force data than it would be to manipulate violent crime data. We also suspect that at least sometimes police kill an animal without reporting it to anyone. There are almost 20,000 law enforcement agencies in the US. Almost all of them make it incredibly difficult and expensive to acquire their records related to the use of deadly force. The Puppycide Database Project has no money and a very small number of volunteers.

But few are satisfied with this sort of thing. People demand instant gratification, not answers on the law-away plan! So we will end this article with a brief examination of what we do know.

There are 525,600 minutes in a year. So if one dog is shot by police every 98 minutes, it would mean some 5363 dogs are shot by law enforcement every year.

We also know there are approximately 18,000 law enforcement agencies in the United States. Of these 18,000 agencies, the Puppycide Database Project has complete use of force records related to shootings of animals for only 50 departments. Despite the paltry sample size, this is to our knowledge the largest collection of such records currently available anywhere. Here it is:

City / Police DepartmentStateDog Shootings / YearDates ReviewedTotal Shootings in Sample
Sacramento County SheriffCA1.6923076923[2008-2013]22
Citrus HeightsCA0.1111111111[2006-2014]1
LAPDCA22.1416803954[1/1/1985-3/12/2005,2009-2013]560
OaklandCA8[2009]8
Contra Costa Animal ControlCA0.125[2002-2009]1
San Diego CityCA5.5[2011-2014]22
San Diego SheriffCA7.75[2011-2014]31
CarlsbadCA0[2011-2014]0
Chula VistaCA0[2011-2014]0
CoronadoCA0[2011-2014]0
El CajonCA0.25[2011-2014]1
EscondidoCA0.5[2011-2014]2
La MesaCA0[2011-2014]0
National CityCA1.25[2011-2014]5
OceansideCA0.75[2011-2014]3
Riverside County SheriffCA16.2[2000-2004]81
San Bernardino County SheriffCA16.2[2000-2004]81
Riverside CityCA6.1[2000-2004]30.5
San Bernardino CityCA6.1[2000-2004]30.5
Saint PetersburgFL7[2011]7
DavieFL0[2006-2011]0
SunriseFL0[2011]0
Broward County SheriffFL12[2010,2011]24
Palm Beach County SheriffFL14.1111111111[2001,2004-2012]127
HollywoodFL5[2010,2011]10
AtlantaGA8.3333333333[2010-2012]25
DeKalb CountyGA10.6666666666[2010-2012]32
Gwinnett County GA6.3333333333[2010-2012]19
Clayton County GA3.3333333333[2010-2012]10
Cobb CountyGA2.6666666667[2010-2012]8
Ottumwa IA1.6[2010-2014]8
MarshalltownIA0.4[2005-2014]4
ElginIL5.75[2008-2011]23
ChicagoIL81.333333333[2008-2013]488
OmahaNE22.6524685382[2008/01/01 – 2008/08/20,2007]39
LincolnNE4[2014]4
HastingsNE1.5[2014,2015]3
New York CityNY50.8474576271[1990/01/01 – 2005/03/12,2011,2012]875
RochesterNY19.5[2009 – 2012]78
BuffaloNY25.0909090909[2011/01/01 – 2014/09/01]92
CincinnatiOH7.3636363636[2011/01/01 – 2014/09/01]27
DaytonOH12.25[2011-2014]49
Montgomery County SheriffOH2.3333333333[2012-2014]7
Middletown PoliceOH7.5[2011-2014]30
Butler County SheriffOH23[2014]23
PhiladelphiaPA8[2014]8
Houston Police DepartmentTX40.3162055336[2010/01/01 – 2014/07/23]187
Harris County SheriffsTX8.1925979159[2010/01/01 – 2014/07/23]38
MilwaukeeWI36.2[2011-2014]15
West Virginia State TroopersWV3.75[2009-2013]181
TOTALSN/A10.4738897076N/A3320

Within this sample, a police department shoots an average of 10.5 dogs a year. The sample includes police departments shooting dogs at a alarming rate, like Chicago Police Department which is recorded shooting over 80 dogs a year. The sample also includes multiple departments that for periods of multiple years shot absolutely no dogs, such as Carlsbad, Chula Vista and Coronado.

If this sample were representative of all police departments in the country, it would suggest that police are shooting enormous amounts of dogs - many more than one dog every 98 minutes. If all 18,000 law enforcement agencies in the United States are shooting 10.5 dogs a year, it would put the annual number of dogs shot by police at over 188,000. That rate translates to a shooting every 2 minutes 47 seconds.

So why isn't this series about that? Why are we debunking statistical claims instead of focusing on the even more alarming national statistics that can be deduced from the Puppycide Database?

Our sample, as currently presented, does not adequately address problems of statistical bias. Counter-intuitively, the secrecy with which police departments handle their use of force records have forced us to rely on a substantial amount of data obtained either as part of a Consent Decree agreement between a municipal police department and the Department of Justice, or on bulk Freedom of Information or other types of transparency requests performed by journalists in order to obtain complete numbers of shootings within a single department over a significant period of time. Both of these sources of information tend to include police departments in trouble - agencies that engage in systemic patterns of excessive force. Such problems exist in many police departments, but not all of them.

LAPD consent decree DOJ

The smaller a statistical sample, the less likely it is to be accurate. The larger Puppycide Database contains thousands of records related to individual animal shootings from hundreds of police departments in all 50 states; we suspect that this wider sample, as well as a more varied array of sources of information, provides a more moderate (and possibly more accurate) look at national puppycide trends.

The truth is that determining trends among a population (in this case, the population of dogs shot by police) quickly becomes complex when the size of the population is unknown [PDF]. The Puppycide Database Project is working on an accurate estimate of the total number of annual dog shootings in the United States. Ultimately the answer will come from a substantial and transparent sample of puppycides from across the country, one or more established statistical analysis techniques from the academic literature and the public review of that work by independent researchers.

In Part Three of our series "Statistics are Misleading 100% of the Time", we will investigate another statistic that is even more widely accepted than the claim we reviewed here in Part Two. Is it true that "Half of all police shootings involve a dog"? Who came up with this statistic and what research is it based on? In order to understand the answers to these questions, we will discuss some basic concepts in research and statistics, like primary sources, bias, sampling and randomness. Click here to read the next article in this investigation!

NOTE: Despite the use of the word "Puppycide" in both Puppycide Database Project and the title of Puppycide: The Documentary, there is absolutely no relationship between our our project and the film. PuppycideDB did write a brief article about the film's first trailer.