Security Metrics 🗑🔥

13 minute read | Last updated 10-Dec-2018 |  Security

Watch the episode on YouTube

Join the discussion on LinkedIn

Share on Twitter

Bad Robot Transcript

Morning everybody. How you doing today? I wanted to talk to you about measuring things that like measuring things but measuring things that matter minutes a famous book by John door and it really hits point hits home on the key point. I have are you measuring the of metrics that actually matter the reason why I want to talk about this is you know, we’re here in early mittasch December and I’m starting to look at 2019, right? So I’m looking back in 2018.

And what worked what didn’t work ABS on analyzing the stats in the analytics behind the show behind my writing the talks that I give the interviews. I give all the activities that I do over the course of a year trying to figure out what had an impact strategy for 2019.

It’s do that. Obviously I need data. I’ve been measuring the measuring the potential impact of each activity. Time I get to look back and say okay and what really mattered was able to be effective, you know, when I compare that to my list of stated goals that I have for myself at the beginning of the year and it was interesting this year is the goals that I was looking for.

I’m working close enough on on a few key ones but there’s three goals that I had that had no way to measure and in retrospect it was nice to aspirational statement when I’m looking at the at the summary for 2018 of these goals. They don’t make any sense. They’re basically worthless because they’re entirely subjective and there’s no way to measure it back to today’s topic.

Metrics because I have yet to meet a security team that has effective metrics. I meant tons of security teams that are measuring a huge amount of different things. But are they actually driving a action or are they driving towards better measuring the effectiveness of you know, your efforts towards various outcomes of more often than not know.

They aren’t now that’s not to say I have a list of metrics that will help you. I wish I did but let’s walk through an example. I was talking to some folks who are running in a sock Social Security operation Center. This is where an Enterprise pulls all their data into 1 M of teams in an operation Center that are monitoring a and kick-off incident response incident analysis all this kind of thing.

So basically when you know, what hit the fan they’re the ones that figure out that it hits the fan and then go from there. The interesting thing is normally security operations teams are absolutely overwhelmed with a volume of alerts Adam and I talk it into bo3 and I ball park to the effectiveness of these teams that you A very wishful thinking of maybe 10% of the things that they handle are actually effective to be honest.

I think that number is probably far more close to 1% So, you know for every hundred incidents or incidences a hundred events that they are have to analyze maybe one of them is actually worth their time and a security incident. So it’s not a highly effective way of doing things but it is the way we do things and I think the reason why it’s been that way for so long is that I’ve yet to come across a sock that measures what I think is really obvious metric which is effectiveness of team or let me rephrase that.

Effectiveness of team but percentage of events analyze that led to incidents because for me that’s you don’t tells you whether you are measuring the right things are looking at the right things that part of it obviously it’s more complicated then maybe you just didn’t get breached. Maybe there were no security incidents, but then the question is raised as to why do you have so many events that you’re analyzing? What a pretty straightforward example of measuring I’m something that has an impact interesting one, you know, if you look at sort of the World of Sports sport any team sport, there’s always a player on the team.

There’s always a role on the team for somebody who does what we call the intangibles and the intangibles because they don’t show up in the basic metrics. So it’s somebody on a hockey team or basketball team is not getting points. That’s not getting assists in basketball were a little bit better but measuring so maybe they get rebounds maybe they’re getting steals.

But really it’s the person that makes the play that lets the team make the more important than quotes around important play, you know that person who saves the ball from going out of bounds I more for that head to the right guy who then passes it to the other guy who makes the play in football.

Real football not like soccer for North America. So the there’s this intangible in sports analytics is getting way way better at measuring it but Sportsman going on for decades and they’re just now getting to the point where they can give you an intangible. So I’ll give you a hockey reference is because it’s winter and I’m Canadian Erik Karlsson got traded from the Ottawa Senators to San Jose.

And when he was with OtterBox kiwi scoring over appointed game, so he was either put me in a goal or assisting on a goal very tangible very like that guy is good kind of thing. We move to San Jose. He’s on the fenders and San Jose is a really strong analytics team and they said no he’s contributing just as much but he didn’t get a point like the first 20 games and the other things they’re measuring time on Ice overall plus-minus and they’re measuring how effective their team is when he’s a part of it and that’s what we need to do in the security world.

We need to figure out what those intangible metrics are because Tangible ones we have are ridiculous number firewall events that we have things. We blocked on the firewall. I don’t care about what you blocked on the firewall care about what got through the same with you into response number of incidents.

We’ve responded to okay who who cares? How was their impact of the business? How many did you miss? I mean since didn’t you respond, right? And we really need to measure what matters and I think that’s a fundamental weakness in our security strategies. It was it’s a typical problem.

Don’t get me wrong as demonstrated by the fact that I even set of goals for myself this year that we’re not I’m not able to measure which is just ridiculous. So I’m trying to do better for 2019. I have to set up measurable goals to set of tangible quantifiable gold that get me closer to my desired outcome and my desire to help you understand security and privacy.

So I’m not only the impact of your own life, but how cyber security and privacy by Design is the best way forward and how we can get there. That’s my girl. I want people to understand this stuff better. I want them to have a more complete understanding with them.

I want to demystify a lot of this. So I have an audience reach feedback on community participation and I’m still shaking laws for 2019, but I know that’s where I want to make an impact and hopefully I’ll be able to measure that what do you think? What are you measuring when you think about my goals for 2019? What should I be setting? I let me know online at Markin CA Lucy in the blog in the comment down below and as always for podcast listeners.

Everybody else me at Mark and I. Ca by email. I hope your setup for fantastic day today, and I hope to talk to you online and on the show tomorrow and hear your thoughts on this really important subject. Take care. Morning everybody. How you doing today? I wanted to talk to you about measuring things that like measuring things but measuring things that matter minutes a famous book by John door and it really hits point hits home on the key point.

I have are you measuring the of metrics that actually matter the reason why I want to talk about this is you know, we’re here in early mittasch December and I’m starting to look at 2019, right? So I’m looking back in 2018. And what worked what didn’t work ABS on analyzing the stats in the analytics behind the show behind my writing the talks that I give the interviews.

I give all the activities that I do over the course of a year trying to figure out what had an impact strategy for 2019. It’s do that. Obviously I need data. I’ve been measuring the measuring the potential impact of each activity. Time I get to look back and say okay and what really mattered was able to be effective, you know, when I compare that to my list of stated goals that I have for myself at the beginning of the year and it was interesting this year is the goals that I was looking for.

I’m working close enough on on a few key ones but there’s three goals that I had that had no way to measure and in retrospect it was nice to aspirational statement when I’m looking at the at the summary for 2018 of these goals. They don’t make any sense. They’re basically worthless because they’re entirely subjective and there’s no way to measure it back to today’s topic.

Metrics because I have yet to meet a security team that has effective metrics. I meant tons of security teams that are measuring a huge amount of different things. But are they actually driving a action or are they driving towards better measuring the effectiveness of you know, your efforts towards various outcomes of more often than not know.

They aren’t now that’s not to say I have a list of metrics that will help you. I wish I did but let’s walk through an example. I was talking to some folks who are running in a sock Social Security operation Center. This is where an Enterprise pulls all their data into 1 M of teams in an operation Center that are monitoring a and kick-off incident response incident analysis all this kind of thing.

So basically when you know, what hit the fan they’re the ones that figure out that it hits the fan and then go from there. The interesting thing is normally security operations teams are absolutely overwhelmed with a volume of alerts Adam and I talk it into bo3 and I ball park to the effectiveness of these teams that you A very wishful thinking of maybe 10% of the things that they handle are actually effective to be honest.

I think that number is probably far more close to 1% So, you know for every hundred incidents or incidences a hundred events that they are have to analyze maybe one of them is actually worth their time and a security incident. So it’s not a highly effective way of doing things but it is the way we do things and I think the reason why it’s been that way for so long is that I’ve yet to come across a sock that measures what I think is really obvious metric which is effectiveness of team or let me rephrase that.

Effectiveness of team but percentage of events analyze that led to incidents because for me that’s you don’t tells you whether you are measuring the right things are looking at the right things that part of it obviously it’s more complicated then maybe you just didn’t get breached. Maybe there were no security incidents, but then the question is raised as to why do you have so many events that you’re analyzing? What a pretty straightforward example of measuring I’m something that has an impact interesting one, you know, if you look at sort of the World of Sports sport any team sport, there’s always a player on the team.

There’s always a role on the team for somebody who does what we call the intangibles and the intangibles because they don’t show up in the basic metrics. So it’s somebody on a hockey team or basketball team is not getting points. That’s not getting assists in basketball were a little bit better but measuring so maybe they get rebounds maybe they’re getting steals.

But really it’s the person that makes the play that lets the team make the more important than quotes around important play, you know that person who saves the ball from going out of bounds I more for that head to the right guy who then passes it to the other guy who makes the play in football.

Real football not like soccer for North America. So the there’s this intangible in sports analytics is getting way way better at measuring it but Sportsman going on for decades and they’re just now getting to the point where they can give you an intangible. So I’ll give you a hockey reference is because it’s winter and I’m Canadian Erik Karlsson got traded from the Ottawa Senators to San Jose.

And when he was with OtterBox kiwi scoring over appointed game, so he was either put me in a goal or assisting on a goal very tangible very like that guy is good kind of thing. We move to San Jose. He’s on the fenders and San Jose is a really strong analytics team and they said no he’s contributing just as much but he didn’t get a point like the first 20 games and the other things they’re measuring time on Ice overall plus-minus and they’re measuring how effective their team is when he’s a part of it and that’s what we need to do in the security world.

We need to figure out what those intangible metrics are because Tangible ones we have are ridiculous number firewall events that we have things. We blocked on the firewall. I don’t care about what you blocked on the firewall care about what got through the same with you into response number of incidents.

We’ve responded to okay who who cares? How was their impact of the business? How many did you miss? I mean since didn’t you respond, right? And we really need to measure what matters and I think that’s a fundamental weakness in our security strategies. It was it’s a typical problem.

Don’t get me wrong as demonstrated by the fact that I even set of goals for myself this year that we’re not I’m not able to measure which is just ridiculous. So I’m trying to do better for 2019. I have to set up measurable goals to set of tangible quantifiable gold that get me closer to my desired outcome and my desire to help you understand security and privacy.

So I’m not only the impact of your own life, but how cyber security and privacy by Design is the best way forward and how we can get there. That’s my girl. I want people to understand this stuff better. I want them to have a more complete understanding with them.

I want to demystify a lot of this. So I have an audience reach feedback on community participation and I’m still shaking laws for 2019, but I know that’s where I want to make an impact and hopefully I’ll be able to measure that what do you think? What are you measuring when you think about my goals for 2019? What should I be setting? I let me know online at Markin CA Lucy in the blog in the comment down below and as always for podcast listeners.

Everybody else me at Mark and I. Ca by email. I hope your setup for fantastic day today, and I hope to talk to you online and on the show tomorrow and hear your thoughts on this really important subject. Take care.