Watch this episode on YouTube.
Reasonably Accurate 馃馃 Transcript
Good morning, everyone. Hope you're having a great day set up for a successful Wednesday. Um, I'm marknca. I wanted to talk to you a bit about the um, report going around that the FBI in the US had grossly overestimated the amount of phones that they had um trouble accessing due to encryption.
Now, um regardless of your view on the politics around the situation, I think it's far simpler um to um put the misreported number of 7800 at simple oversight or lack of accurate data keeping. Um then at some sort of malicious argument, but it does raise a larger issue around accurate security statistics and making the argument for particular security controls because at the end of the day, that's what the FBI is doing.
Now, I'm on record as firmly disagreeing with the FBI stance that they need access to encrypted devices. In fact, with any law enforcement's um uh stance that they need backdoors into encryption because we have a hard enough time making secure systems without trying to put a back door into them, let alone trying to put a back door and then trusting a certain party with access to these devices because that's just never gonna work.
It's a key escrow system. There are number of reasons, any number of reasons why that won't work. I'll link to a great article down below written by a lot of um uh notable cryptographers in response to Ray Ozzie's um little uh dog and pony show around his system to solve the going dark problem.
Um But what really kind of bothered me about this um report around the FBI. Um you know, and the FBI is earnestly trying to do the job they're tasked with. Um And that's true of law enforcement and the RC MP here in Canada. And we as a society want our law enforcement arguing for every tool that they can to implement the laws um in the way that we as the society and community have asked for, right?
But it's our job then to balance that out by saying, look, here are the guidelines you need to stay between these lines. Um And that's the real problem here and that's actually the same problem within your organization um around a lack of data and metrics.
And that's really at the core of this issue is that we need accurate security metrics in order to make informed decisions if we don't have data and we aren't making data based decisions, you're basically guessing. Um This is one of my fundamental problems with any sort of risk assessment framework that people take them as gospel as quantitative when they're really qualitative frameworks.
So you'll see in a framework will be like, oh, the risk is five out of 10. Well, what created that number? How did you come up with five out of 10? Well, you know, it was a red and a medium and a blah, blah, blah and there's no real data behind it.
It's just supposition, it's just speculation. Um And that's not how you make informed decisions. That's not how computer science works. And remember, computer science and security are all actually science. We should be taking a scientific method. Yes, there is a lot of operationalized, get it out of the lab into the real world.
But at every chance we get, we should be collecting as much uh accurate data as we can and as reasonable as possible. Obviously, keeping in mind that GDPR is coming, I mean, you need to respect, we see user boundaries, but the challenge here with this FBI is not this misreported number because I genuinely think that's due to poor record keeping.
Um and sort of Guim information rather than any malicious intent. Um The challenge is in general, we don't have accurate statistics and we don't frame statistics properly. So, um if I was and this is a total hypothetical, so let's say let's do a group exercise here, hypothetically, if we are law enforcement, we're trying to make the argument that we need to um have access in a backdoor to devices.
What do we need? We need not just the number of devices that we can't access because the number doesn't mean anything. Oh, we can't access a million devices. Is that a lot? I mean, it sounds like a lot but compared to what you need to put that number in context.
So you need to say, well, we have currently uh a million devices out of 20 million cases that we try that involve digital devices that are inaccessible to us. And then they, they, you start to frame the city. Go well, hold on. That means one in 20 are inaccessible, but 19 and 20 are accessible.
Well, then in those cases where it's not, what are the potential crimes up for trial or there another act, any other angles, there's far more to this than just a number. And this holds true if we take this to the organization as well, the number of CIO s that I see reporting back to the CEO S or the CIO S or even the board with ridiculous stats like number of um you know, attempts stopped at the firewall.
Who cares? Um Is that any productive value to the business? Right? Is there anything that puts that in proper context? Hey, we stopped um you know, a billion attempts on the firewall this month. Ok. Out of how many connections made out of like, is that normal for an organization our size is that, you know what what, what puts that number into context instead of just throwing a number out there, what puts that number into context?
And that's really the challenge here is that I think in general, um not just the law enforcement case, but in the, in the computer science and the defender in the security team within the organization case, we've forgotten how to run stats, how to put an argument in context, how to gather data, to make that qualitative argument with evidence.
And that's really a key here. You need evidence. Um That's sort of my ramble, my rant for today. There's a ton more bubbling up here around that because if you do a search for security metrics, you're gonna be absolutely disappointed in the data that you see back in the post that you see back.
Um There's very little actionable, there's very little um intelligent metrics out there. There's a lot of vanity metrics. It's the same challenge in marketing like, hey, we got 1000 page views or 100,000 page views. OK. Who, who was viewing the page? Was it our target customer or was it our target audience?
Was it anybody we care about? Was it a bot like all this kind of stuff? You need metrics that makes sense. You need data that drives decisions. And that's really um sort of what piqued my uh my interest and started rolling this rant this morning.
Um I think it's important. I think it's an effort that's required. I think everybody should be doing um basic statistics. One on one, I think everybody should be reviewing how to make it evidence based and a data, data based argument because I think we're all better off when we stick to the facts as opposed to just guessing because then numbers get thrown around, numbers get exaggerated.
And then eventually you get in uh to a highly political situation where we need an honest and transparent and open here. We need one that's uh we need clear amount of evidence that we can have an active debate, not just in the US, but in globally how we want to handle this stuff because our borders are quickly becoming somewhat irrelevant when it comes to cybercrime and our technology comes from countries around the world.
We need to have a clear argument and you can only do that with data. I hope you're uh set up for a fantastic Wednesday. Hit me up online, marknca as always down in the comments below. Um Look forward to talking about this issue because it's all about discussion, interaction and seeing uh every side of the issue.
Um and the data to back up those positions. We'll talk to you tomorrow. Have a great day.
