Watch this episode on YouTube.
Reasonably Accurate 馃馃 Transcript
Morning everybody. How are you doing today? Um, wanted to talk to you about measuring things, not like measuring things but measuring things that matter. Um And that's a famous, um, book by John Deer. Um, and it really hits point, uh, hits home on the key point of, are you measuring, do you have metrics that actually matter?
The reason why I want to talk about this is, you know, we're here in early Middish December. Um, and I'm starting to look at 2019, right? So I'm looking back at 2018. Um, what worked, what didn't work? Um So I'm analyzing the stats, um, and the analytics behind this show, um, behind my writing, the talks that I give the interviews, I give all the activities that I do over the course of the year.
I'm trying to figure out what had an impact, what I should focus on what I should change. What's my strategy for 2019 and to do that. Um, obviously I need data now throughout the year, I've been measuring things and actually, I've been doing it for years measuring the potential impact of each activity that I take.
So, uh around this time, I get to look back and say, ok, you know, what really mattered was I able to be effective. Um, you know, and I compare that to my list of stated goals that I have for myself at the beginning of the year.
Um And what was interesting this year is, you know, I did pretty well. I met the goals that I was looking for, um, or came close enough on, on a few key ones. Uh, there's three goals that I had that had no way to measure.
And in retrospect, you know, it was nice, an aspirational statement. But when I'm looking at the, um, at the summary for 2018, I look back at this one, there's no possible way I could ever have measured these goals. They don't make any sense, they're basically worthless because they're entirely subjective and there's no way to measure it.
So that brings me back to today's topic metrics because I have yet to meet a security team that has effective metrics. I've met tons of security teams that are measuring a huge amount of different things. But are they actually driving action or are they driving towards better measuring the, the effectiveness of, uh, you know, your efforts towards various outcomes more often than not?
No, they aren't. Now, that's not to say I have a list of metrics that will help you. Um I wish I did, but let's walk through an example. I was talking to um some folks who are running a sock. So a security operation center, this is where an enterprise pulls all their data into one set of teams in an operation center that are monitoring um and kick off incident response, incident analysis, all this kind of thing.
So basically when you know what hits the fan, they're the ones that figure out that it hits the fan and then go from there. Um The interesting thing is they normally security operations teams are absolutely overwhelmed with a volume of alerts. Um, and my talk at Aws reinvents I ballpark, the effectiveness of these teams that, you know, and a very wishful thinking of maybe 10% of the things that they handle are actually effective.
To be honest, I think that, uh, numbers probably far more, uh, close to 1%. So, you know, for every 100 incidents or 100 not incidents, but 100 events that they have to analyze, maybe one of them is actually worth their time in a security incident.
So this is not a highly effective way of doing things, but it is the way we do things. Um, and I think the reason why it's been that way for so long is that I've yet to come across a sock that measures what I think is a really obvious metric which is effectiveness of team or let me rephrase that not effectiveness of team, but percentage of events analyzed that led to incidents because for me that's, uh, you know, tells you whether you're measuring the right things or looking at the right things.
Um, that's part of it. Obviously it's more complicated than that. Maybe you just didn't get, uh, breached, maybe there were no security incidents. But then the question is raised as to why do you have so many events that you're analyzing? Um, so, you know, a little confusing but a pretty straightforward example of measuring, um, something that has an impact.
Um, interesting one, you know, if you look at uh, sort of the world of sports, um, any sport, any team sport, there's always a player on the team, there's always a role on the team for somebody who does what we call the intangibles and the intangibles because they don't show up in the basic metrics.
So, um, it's somebody on, uh, a hockey team or a basketball team that's not getting points, it's not getting assists. Um, in basketball, we're a little bit better about measuring. So maybe they get, um, rebounds, maybe they're getting steals. Um, but really, it's the person that makes the play that lets the team make the more important, big quotes around important play.
So it's this, you know, it's the person who gives the extra little bit of effort in the corner to dig the puck out so that you can go down the ice and score or that, you know, that person who saves the ball from going out of bounds, um, or forwards that header to the right guy who then passes it to the other guy who makes the play in football.
Um, real football, not like soccer for North America. Um, so the, uh, you know, there's this intangible and sports analytics is getting way, way better at measuring it, but sports have been going on for decades and they're just now getting to the point where they can give you an intangible.
So I'll give you a hockey reference just because it's winter and I'm Canadian. Um Eric Carlson got traded from the Ottawa Senators to San Jose and when he was with Ottawa, he was scoring over a pointed game. So he was either putting a, a goal or assisting on a goal.
Very tangible. Very like that guy is good kind of thing. When he moved to San Jose, he's on a much stronger team. He's with uh other key defenders and San Jose is a really strong analytics team and they said, no, no, he's contributing just as much, but he didn't get a point in like the 1st 20 games and they were like, well, how is he contributing just as much?
Well, because it's the other things they're measuring uh time on ice, uh go overall plus minus. Um They're measuring how effective their team is when he's a part of it. Um And that's what we need to do in the security world. We need to figure out what those intangible metrics are because the tangible ones we have are ridiculous.
Um, number of firewall events that, you know, things we've blocked on the firewall. I don't care about what you blocked on the firewall. I care about what got through. Um, same with, you know, uh, incident response, number of incidents. Uh, we responded to, ok, who, who cares, how was the impact of the business?
How many did you miss? How many incidents didn't you respond to? Right. And we really need to measure what matters. And I think that's a fundamental weakness in our security strategies. It was, it's a typical problem. Don't get me wrong. Um As demonstrated by the fact that I even set up goals for myself this year that were not, I'm not able to measure, which is just ridiculous.
So I'm trying to do better for 2019 to set up measurable goals to set up tangible quantifiable goals that get me closer to my desired outcome and my desired outcome is to help you understand security and privacy. So, um not only the impact of your own life but how um cybersecurity and privacy by design is the best way forward and how we can get there.
That's my goal. I want people to understand this stuff better. Um I want them to have a more complete understanding. I want them to, I want to demystify a lot of this. So I have tangible goals on audience reach on, you know, feedback on community participation and I'm still shaping those for 2019.
But I know that's where I want to make an impact and hopefully I'll be able to measure that. What do you think? What do you uh how do you approach measuring? Um What do you think about my goals for 2019? What should I be setting?
Uh Let me know online at Mark NC A for those of you in the vlog in the comments down below. And as always for podcast listeners and everybody else me at Mark N dot C A by email. Um I hope you're set up for fantastic uh day to day and I hope to talk to you uh online and on the show tomorrow and hear your thoughts on this really important subject.
Take care.
