Statistics are misleading 100% of the time - Conclusion

Over the past week, the Puppycide Database Project has published some 13,000 words as part of our multi-part series, Statistics are misleading 100% of the time. The purpose of our series was to investigate two of the most widely circulated and highly respected statistics published concerning the use of lethal force toward animals by police. These pair of statistics have been repeated for years by some of the most well-known organizations in news, law enforcement and animal welfare. As a result, they have had a significant influence on the national conversation surrounding police shootings of pets.

Our investigation was thorough and our conclusions were largely critical. The first statistic we reviewed, in Part Two of our series, was that "police shoot a dog every 98 minutes". Our research lead us to conclude that this statistic was made famous during the production of an upcoming documentary film, "Of Dogs and Men". We discovered that the statistic had no basis in controlled research, and was originally circulated as part of the marketing and fundraising efforts for the new movie.

In Part Three and Part Four of our study, we reviewed the claim that "half of all intentional police shootings involve a dog". This statistic has been made by the ASPCA, the Department of Justice as well as a variety of respected research and advocacy organizations. Unlike "98 minutes", this second statistic was provided with sources and citations. We researched those sources and determined that they did not prove anything related to national trends in police shootings. We also found that several mistakes had been made in both the primary and secondary sources - shootings were miscounted, years were mis-cited, and jurisdictions were overlooked.

Throughout the series, we made it a point to explain fundamental concepts in statistics, probability and research in order to clearly explain the problems we discovered. Of particular concern were how to determine whether a statistical sample is representative and how to limit or control for bias. These explanations are made in plain, non-technical language that doesn't require any proficiency in math or science to get the hang of. Statistics are used to justify all sorts of claims to us - from selling products to convincing us to vote a certain way. There has always been a lot to gain by misleading people about what can be rationally said about statistics. It is Puppycide Database Project's hope that even those who are not interested in the use of police force can benefit from this series: the mistakes we point out are not specific to puppycide, and are in fact common to all forms of research.

The response to our series has been mixed. The film producers who created "Of Dogs and Men", the ASPCA, the researchers who published a whitepaper for the DOJ that lead to the "half all police shootings involve dogs" statistic being widely accepted - all of these groups have done an enormous amount of work to ensure the public is aware of the growing problem of police shootings of domestic animals. Some have been working on publicizing 'puppycide' for years before Puppycide Database Project was even founded. Is our series, Statistics are Misleading 100% of the Time, an attack on these groups?

We selected the statistics offered by these groups not simply because they have been among the most influential, but because, in general, their work has been the most comprehensive and worthy of citation. The National Canine Research Council, who played a role in in the DOJ whitepaper we discuss in Parts 3 and 4 of our series, was a valuable source for our research on fatal dog attack trends. The Police Executive Research Forum provided us with back issues that made this study possible. And although we have yet to watch "Of Dogs and Men" we have spoken to several people who have and we are told the film is excellent. We recommend that all of our readers and fellow researchers review the sources we researched in our series.

Peer review can be tough - but we could not think of a higher compliment than to commit a significant amount of time and resources to the study of another organization's work. If Puppycide Database Project believed that the sources we reviewed for Statistics are Misleading 100% of the Time were not worthy of study, we would not have studied them. We hope that others pursue our own Database, articles and analysis as comprehensively.

There is only so much that a single individual or organization can do to further a new field of research. Make no mistake, the study of police use of force toward animals is a brand new field of research - at least in the United States the Puppycide Database Project has been unable to locate any controlled studies on the topic. Even should we complete our database so that each and every police shooting of an animal is accounted for, any analysis of that data we provide will need to be reviewed thoroughly by other researchers. Use of force behavior specific to animal encounters deserves the same attention from criminologists as more general use of force studies.

Before we conclude this series, we want to address one last type of response we received about Statistics are Misleading. Several readers wrote to us to let us know that they did not trust any figures or statistics they encounter on matters of political concern. We received this feedback from a startling array of readers with different opinions on law enforcement policy. The sentiment was not specific to those with a middling interest in politics, either: determined, well known activists shared this view right along first-time internet commentators.

This response gave us pause; such a sentiment is reason for concern for researchers, journalists, pollsters, or anyone whose work depends on using numbers to communicate to members of the public. The point made to us was not that these readers were skeptical of statistics made to communicate matters of social policy - but they view such claims as completely lacking any credibility.

Such a response cannot be dismissed as the ramblings of the mathematically illiterate. Statistics reported in media sources routinely refuse to provide even public references or "show their work" in a manner that is consistent with the requirements of even a freshman research paper. Readers are assured to place their faith in the credibility of well-respected media organizations, but are left adrift when it comes to light that such organizations fail to perform substantive fact-checking, or if two or more organizations make competing factual claims. Hundreds of news organizations repeated the two statistics we reviewed for this series. Few if any of the mistakes we found in these statistics should have escaped fact-checking that simply consulted the same sources we did.

Our point is not to offer some vague fear-mongering about "The Media": many of our staff and volunteers are themselves journalists or freelancers for newspapers. The reasons why the level of public skepticism toward factual statistical claims are varied and complex - but they aren't crazy.

Statistics provide the foundation for essentially all modern scientific research. A great deal of this research does not need to pay any regard of the growing public skepticism we describe here. However, researchers can have no such luxury if there is to be any hope their studies will eventually influence public policy.

Our task at Puppycide Database Project cannot be successful if all we do is count the numbers we find - it's not enough to be correct. Our findings must be presented in a way that their objectivity is clear to readers. Mistakes and fundamental biases need to be addressed transparently and immediately. We can't stay contained in an ivory tower because our Project simply won't work with that approach.

With any luck, those who follow our research will meet us half way. By showing our work, clearly stating our sources and limitations as well as responding to criticism we can convince readers that there is nothing intrinsically dishonest about statistics.