Avoiding Bad Stats and the Benefits of Playing Trivia with Friends: PancakesCon 4

Editor's Notes

1. On knowing what normal looks like, this can be tricky, because it requires some expertise in a field, and familiarity with general stats in your field.For example, to know a stat on the number of unfilled jobs in cybersecurity seems off, you need a general understanding of how large the cybersecurity field is, and some awareness of how much hiring is going on in the public and private space, in your country and globally.You don't need to know all that to investigate a stat or claim though! Most of the data you need is available on the Internet!The Gell-Mann Amnesia Effect was something that Michael Crichton came up with - yeah, the guy that created Jurassic Park and ERHe used this term to describe the phenomenon of experts believing news articles written on topics outside of their fields of expertise, yet acknowledging that articles written in the same publication within their fields of expertise are error-ridden and full of misunderstandingThe lesson here, is that it's important to have a healthy dose of skepticism while consuming media, reports, papers, and research.
2. On knowing what normal looks like, this can be tricky, because it requires some expertise in a field, and familiarity with general stats in your field.For example, to know a stat on the number of unfilled jobs in cybersecurity seems off, you need a general understanding of how large the cybersecurity field is, and some awareness of how much hiring is going on in the public and private space, in your country and globally.You don't need to know all that to investigate a stat or claim though! Most of the data you need is available on the Internet!The Gell-Mann Amnesia Effect was something that Michael Crichton came up with - yeah, the guy that created Jurassic Park and ERHe used this term to describe the phenomenon of experts believing news articles written on topics outside of their fields of expertise, yet acknowledging that articles written in the same publication within their fields of expertise are error-ridden and full of misunderstandingThe lesson here, is that it's important to have a healthy dose of skepticism while consuming media, reports, papers, and research.
3. On knowing what normal looks like, this can be tricky, because it requires some expertise in a field, and familiarity with general stats in your field.For example, to know a stat on the number of unfilled jobs in cybersecurity seems off, you need a general understanding of how large the cybersecurity field is, and some awareness of how much hiring is going on in the public and private space, in your country and globally.You don't need to know all that to investigate a stat or claim though! Most of the data you need is available on the Internet!The Gell-Mann Amnesia Effect was something that Michael Crichton came up with - yeah, the guy that created Jurassic Park and ERHe used this term to describe the phenomenon of experts believing news articles written on topics outside of their fields of expertise, yet acknowledging that articles written in the same publication within their fields of expertise are error-ridden and full of misunderstandingThe lesson here, is that it's important to have a healthy dose of skepticism while consuming media, reports, papers, and research.
4. On knowing what normal looks like, this can be tricky, because it requires some expertise in a field, and familiarity with general stats in your field.For example, to know a stat on the number of unfilled jobs in cybersecurity seems off, you need a general understanding of how large the cybersecurity field is, and some awareness of how much hiring is going on in the public and private space, in your country and globally.You don't need to know all that to investigate a stat or claim though! Most of the data you need is available on the Internet!The Gell-Mann Amnesia Effect was something that Michael Crichton came up with - yeah, the guy that created Jurassic Park and ERHe used this term to describe the phenomenon of experts believing news articles written on topics outside of their fields of expertise, yet acknowledging that articles written in the same publication within their fields of expertise are error-ridden and full of misunderstandingThe lesson here, is that it's important to have a healthy dose of skepticism while consuming media, reports, papers, and research.
5. Let's take a closer look at one of these stats. First off, TRUST YOUR GUT. If it sounds like BS, it's worth investigating. In 2017, I set out to document every company that had ever been destroyed by a breach. How many did I find? Only 23 over a 20 year period. All small businesses; maybe 4 had over 100 employees, but all less than 500 Is it possible I missed some, or some didn't get reported? Sure! Is it possible I'm missing 150 PER YEAR? Probably not. This answers a more broad and general assumption in our industry: the assumption that security incidents MUST be ending companies Ramon Ray: founder and owner of smallbiztechnology.com and Smart Hustle. He was still using the stat as recently as late 2021, five years after NextGov interviewed him for their article debunking this claim. And that's at the core of what we're talking about here: there are a lot of folks out there that won't let the truth get in the way of a good sound byte
6. Let's take a closer look at one of these stats. First off, TRUST YOUR GUT. If it sounds like BS, it's worth investigating. In 2017, I set out to document every company that had ever been destroyed by a breach. How many did I find? Only 23 over a 20 year period. All small businesses; maybe 4 had over 100 employees, but all less than 500 Is it possible I missed some, or some didn't get reported? Sure! Is it possible I'm missing 150 PER YEAR? Probably not. This answers a more broad and general assumption in our industry: the assumption that security incidents MUST be ending companies Ramon Ray: founder and owner of smallbiztechnology.com and Smart Hustle. He was still using the stat as recently as late 2021, five years after NextGov interviewed him for their article debunking this claim. And that's at the core of what we're talking about here: there are a lot of folks out there that won't let the truth get in the way of a good sound byte
7. Let's take a closer look at one of these stats. First off, TRUST YOUR GUT. If it sounds like BS, it's worth investigating. In 2017, I set out to document every company that had ever been destroyed by a breach. How many did I find? Only 23 over a 20 year period. All small businesses; maybe 4 had over 100 employees, but all less than 500 Is it possible I missed some, or some didn't get reported? Sure! Is it possible I'm missing 150 PER YEAR? Probably not. This answers a more broad and general assumption in our industry: the assumption that security incidents MUST be ending companies Ramon Ray: founder and owner of smallbiztechnology.com and Smart Hustle. He was still using the stat as recently as late 2021, five years after NextGov interviewed him for their article debunking this claim. And that's at the core of what we're talking about here: there are a lot of folks out there that won't let the truth get in the way of a good sound byte
8. Let's take a closer look at one of these stats. First off, TRUST YOUR GUT. If it sounds like BS, it's worth investigating. In 2017, I set out to document every company that had ever been destroyed by a breach. How many did I find? Only 23 over a 20 year period. All small businesses; maybe 4 had over 100 employees, but all less than 500 Is it possible I missed some, or some didn't get reported? Sure! Is it possible I'm missing 150 PER YEAR? Probably not. This answers a more broad and general assumption in our industry: the assumption that security incidents MUST be ending companies Ramon Ray: founder and owner of smallbiztechnology.com and Smart Hustle. He was still using the stat as recently as late 2021, five years after NextGov interviewed him for their article debunking this claim. And that's at the core of what we're talking about here: there are a lot of folks out there that won't let the truth get in the way of a good sound byte
9. Let's take a closer look at one of these stats. First off, TRUST YOUR GUT. If it sounds like BS, it's worth investigating. In 2017, I set out to document every company that had ever been destroyed by a breach. How many did I find? Only 23 over a 20 year period. All small businesses; maybe 4 had over 100 employees, but all less than 500 Is it possible I missed some, or some didn't get reported? Sure! Is it possible I'm missing 150 PER YEAR? Probably not. This answers a more broad and general assumption in our industry: the assumption that security incidents MUST be ending companies Ramon Ray: founder and owner of smallbiztechnology.com and Smart Hustle. He was still using the stat as recently as late 2021, five years after NextGov interviewed him for their article debunking this claim. And that's at the core of what we're talking about here: there are a lot of folks out there that won't let the truth get in the way of a good sound byte
10. Let's take a closer look at one of these stats. First off, TRUST YOUR GUT. If it sounds like BS, it's worth investigating. In 2017, I set out to document every company that had ever been destroyed by a breach. How many did I find? Only 23 over a 20 year period. All small businesses; maybe 4 had over 100 employees, but all less than 500 Is it possible I missed some, or some didn't get reported? Sure! Is it possible I'm missing 150 PER YEAR? Probably not. This answers a more broad and general assumption in our industry: the assumption that security incidents MUST be ending companies Ramon Ray: founder and owner of smallbiztechnology.com and Smart Hustle. He was still using the stat as recently as late 2021, five years after NextGov interviewed him for their article debunking this claim. And that's at the core of what we're talking about here: there are a lot of folks out there that won't let the truth get in the way of a good sound byte
11. Let's take a closer look at one of these stats. First off, TRUST YOUR GUT. If it sounds like BS, it's worth investigating. In 2017, I set out to document every company that had ever been destroyed by a breach. How many did I find? Only 23 over a 20 year period. All small businesses; maybe 4 had over 100 employees, but all less than 500 Is it possible I missed some, or some didn't get reported? Sure! Is it possible I'm missing 150 PER YEAR? Probably not. This answers a more broad and general assumption in our industry: the assumption that security incidents MUST be ending companies Ramon Ray: founder and owner of smallbiztechnology.com and Smart Hustle. He was still using the stat as recently as late 2021, five years after NextGov interviewed him for their article debunking this claim. And that's at the core of what we're talking about here: there are a lot of folks out there that won't let the truth get in the way of a good sound byte
12. Let's take a closer look at one of these stats. First off, TRUST YOUR GUT. If it sounds like BS, it's worth investigating. In 2017, I set out to document every company that had ever been destroyed by a breach. How many did I find? Only 23 over a 20 year period. All small businesses; maybe 4 had over 100 employees, but all less than 500 Is it possible I missed some, or some didn't get reported? Sure! Is it possible I'm missing 150 PER YEAR? Probably not. This answers a more broad and general assumption in our industry: the assumption that security incidents MUST be ending companies Ramon Ray: founder and owner of smallbiztechnology.com and Smart Hustle. He was still using the stat as recently as late 2021, five years after NextGov interviewed him for their article debunking this claim. And that's at the core of what we're talking about here: there are a lot of folks out there that won't let the truth get in the way of a good sound byte
13. 99.5% of losses are unaccounted for - could we make a case that these are all indirect losses? I don't think so...
14. 99.5% of losses are unaccounted for - could we make a case that these are all indirect losses? I don't think so...
15. 99.5% of losses are unaccounted for - could we make a case that these are all indirect losses? I don't think so...
16. 99.5% of losses are unaccounted for - could we make a case that these are all indirect losses? I don't think so...
17. 99.5% of losses are unaccounted for - could we make a case that these are all indirect losses? I don't think so...
18. 99.5% of losses are unaccounted for - could we make a case that these are all indirect losses? I don't think so...
19. Even though the tweet engagement was almost non-existent Cylance held a grudge for years They mis-represented a Verizon DBIR stat You're required to get permission from VZ before using their stats, for exactly this reason They assumed the top of the graph = 100%, when it actually = ~35% I DMed some of the DBIR folks and had the raw data from the graph in my hands in under 30 minutes The danger here is that if you tell defenders that 90% of their problem is malware... there's a really good chance they're going to find a way to justify pouring 90% of their resources into addressing it! At the detriment of other areas that need budget and attention When myths and lies prevail, they can cause us to choose the wrong path
20. Even though the tweet engagement was almost non-existent Cylance held a grudge for years They mis-represented a Verizon DBIR stat You're required to get permission from VZ before using their stats, for exactly this reason They assumed the top of the graph = 100%, when it actually = ~35% I DMed some of the DBIR folks and had the raw data from the graph in my hands in under 30 minutes The danger here is that if you tell defenders that 90% of their problem is malware... there's a really good chance they're going to find a way to justify pouring 90% of their resources into addressing it! At the detriment of other areas that need budget and attention When myths and lies prevail, they can cause us to choose the wrong path