Cloud 37 min read

#LetsTalkCloud: Misconfigurations & Scale

S01E02 - Host, Mark Nunnikhoven, interviews Head of Security, Cloud One Conformity at Trend Micro, Paul Hortop. Together they discuss building systems that fulfill your responsibilities in the cloud.


An icon representing a document where the bottom half of it has been drawn with a dotted outline, implying a copy This post was originally written for Trend Micro .

Reasonably Accurate 馃馃 Transcript

[00:00:00] Mark: Hey, everybody. Thanks for joining. This is our second episode of let's talk cloud. We fired this up last week, had Jeff Westfall and Fernando Cardozo from Trend Micro on board for a great conversation. Three of us were talking about their experiences from the trenches, sort of real world problems. We had a ton of feedback from folks, lots of great responses, and we want to keep that going.

[00:00:43] I can see it. Already on the live stream, we've got folks from Jordan, from Spain, from Panama, from the UAE, um, France, uh, Germany, uh, all over the world, uh, which is amazing. We very, very much, uh, appreciate that. Um, we are streaming across three platforms right now. So we [00:01:00] are, uh, on LinkedIn Live, we're on YouTube, uh, Events, and we're on Periscope slash Twitter.

[00:01:05] Um, please, uh, fire off your comments, um, there. Uh, you can hit us up on the hashtag. Let's talk cloud or respond directly on the streams. We've got a team watching that in order to fire off the comments my way so that we can interject them into the conversation. So please don't be shy. We want this to be very much community driven, community led.

[00:01:30] And that's the goal here is to really You know, we, we sat around the table and we realized trends got a global reach. We've been working actively in the cloud for years, but we haven't done a great job at sort of showing you that expertise and sharing and talking about the challenges that we see and not from a marketing perspective, not from a product perspective, like we have a ton of people who are doing real work in the cloud.

[00:01:52] And we've learned a lot of things. Um, a lot of the time, the really hard way, uh, and that has, uh, you know, we've lessons learned [00:02:00] moving forward. And so we wanted to start sharing that with you folks. So that's the genesis of the stream. So again, hit us up a hashtag. Let's talk cloud. Um, I see even more people joining Turkey, Chile, Columbia, Angola, Senegal, Belgium.

[00:02:12] Welcome everybody. Thanks for joining. Um, especially for those of you who are like super late at night right now. Um, crazy, but you know, Hey, that's awesome that you're joining. And so keep this conversation going. Speaking of global reach, um, our guest today, uh, is joining us from a new team at trend micro, uh, the cloud conformity team.

[00:02:32] They also have global reach a little bit smaller than a trend, but now they've expanded into the trend family. Um, and Paul is joining us from Australia. So, uh, Paul Hortop is our guest. today. He is the head of security of our cloud conformity unit. Um, and so Paul, let me just unmute you here so we make sure that we, uh, we can hear you folks.[00:03:00]

[00:03:45] There we go. Let's see. I think we had a no sound glitch. So we're back on there. You gotta love live streaming. Um, so we're back on board, uh, which is good. Um, but yeah, so cloud, uh, just really just quick summarizes as Paul was saying, he's got a ton of experience, um, not [00:04:00] just, uh, as the head of security, but also, um, helping guide the product and using the product.

[00:04:04] Um, constant, uh, you know, daily, which is, which is huge. Um, so let me just double check with the stream that we're back on line, um, for the audio and, uh, I think we should be good. Um, so Paul, first thing I want to talk to you about here because you are, um, in the trenches because you're using this every day and you are the customer advocate, as you said.

[00:04:27] So obviously you're talking to folks, you're dealing, uh, hearing their problems. Um, what teams do you see working in the cloud? So it's that there are a number

[00:04:39] Paul: of different teams. Uh, there are engineers, developers. security ops teams, dev ops teams, and they all have different challenges. So when I think about a developer, they're typically working on a ticket.

[00:04:54] They're trying to fix something, they're trying to get something created, get it up and going. And [00:05:00] it's very easy to click through a whole bunch of different settings on the different interfaces. And before you know it, you've created an EC2 instance, for example, that's open to the internet on SSH. And it's really hard, particularly, you know, when you're working as a, as a developer, you're trying to fix that ticket, you're working really quickly.

[00:05:19] And it's possible to end up creating a lot of infrastructure that you can't see. So, for example, If your platform is running on a beanstalk, there's all kinds of different bits and pieces there. So, for example, there's a load balancer, there's the auto scaling group. And it's really easy to end up with the load balancer being public facing.

[00:05:35] And you didn't even know that what you were creating created that. You, it's very easy to, to, to a new service and there's all kinds of insecurity in there. And also, um, your, your platform isn't necessarily going to be performant the way that it scales may not be the way that you want it to scale. So there's all of these different challenges there.

[00:05:55] And so that's the developers challenge. [00:06:00] And then you've got security teams and the security teams tend to approach things from a couple of different angles. So, The security operations, they've got to make sure that the platform is secure. They're typically working with a security information event management system.

[00:06:16] And they've got to get to a position of visibility really quickly where they can look across regions and across accounts and be able to see the moment someone creates an S3 bucket, for example, that is public facing. And if it shouldn't be, they've got to be able to identify that quickly and be able to remediate it quickly.

[00:06:33] And there's all those. Potential insecurities. They've got to be able to see the moment that someone signs in as a root user because that almost never should happen and get in that visibility of what's happening in real time is really important, but also they need to be able to get to a position where they can see the whole of their exposure across their virtual private cloud across the actual infrastructure that they're building and then the applications that are running on top of that.

[00:06:59] Uh, [00:07:00] infrastructure and then how that infrastructure is interacting with, say, for example, the Internet, because once you've got so pretty, we're looking at AWS, you've got the AWS infrastructure and how. Uh, the AWS console is telling you that your AWS infrastructure is working. But then if you think about a database or a server that's connected to the internet, there's a whole bunch of other net, network traffic that is going straight into that.

[00:07:26] That you, you have to make sure that your tooling covers, uh, both of those things. So that's the development engineering side of things. That's the security operations side of things. And then you've got compliance and audit teams, and they've got a slightly different job, but they'll typically take a standard.

[00:07:43] So it, it might be the ISO standard or the CIS baseline for, um, cloud security. And then they've got to be able to report across that. And some of the, the organizations that, um, I come across, I've got 500 plus accounts. [00:08:00] And so when you're trying to manage a challenge like that, you'll go in multi region, uh, hundreds and hundreds of accounts, and you've got to do a simple thing like just tell me every EC2 instance that is running a Microsoft server that is open on RDP, remote desktop protocol, and tell me that it's been patched against the latest vulnerability.

[00:08:22] Or just tell Give me some assurance that the password that it is using is really good if that's internet phasing. So for example, if, if that password is winter is coming, then yeah, it's, it's game over. Because the hackers now are really good at identifying these instances and just doing a shotgun attack against them.

[00:08:45] Mark: Yeah, so those are, those are some of

[00:08:46] Paul: the different groups that we see, uh, working and trying to secure that.

[00:08:53] Mark: Cloud deployments, lots to pack their unpack there. Um, I joke at the winter is [00:09:00] coming comment simply because I'm up here in Ottawa, Canada and winter showed up today and gave us about 15 centimeters of snow.

[00:09:07] Um, but you know, very, very accurate. So there's a, there's a ton of stuff there that you did. You mentioned. But the first thing that I wanted to ask right off the bat is, so you identified developers, security, compliance and audit. Are these people working together? Um, Or is it sort of independent efforts that they're just kind of going out there?

[00:09:23] Because especially if you have an organization, like you said, with 500 accounts and if every developer has the power to just kind of go launch cloud formation and there's a ton of new stuff out there, if they're not talking, that's a whole host of problems.

[00:09:37] Paul: It really varies. So there are some organizations and enterprises where you will meet head of security and head of DevOps.

[00:09:45] And when those two sit down together and say, okay. Here's our cloud, uh, posture at the moment. Yeah, we've got all of these issues. There's all of this technical debt. But working together, what can we agree on that we're going to fix [00:10:00] this week? And that just works fantastically. Then you get a real sync going because there is so much.

[00:10:05] Cloud is hard to do. No one is an expert on every service. There's new services every day. Everything is changing in this perpetual change. If you built a workload a year ago, that needs to be refactored. Now there'll be cheaper instances, better capabilities, new security things, nothing stays still. And no one is on top of all of this.

[00:10:27] And you know, The developer thing, you know, you're just fixing that ticket, like you may not know what infrastructure your ticket, your feature is running on, whether that should be updated, and hey, if that was created by someone else, why should you have to fix that stuff that someone else created? And you know, it's not not part of your ticket.

[00:10:48] So it works really well when we see those teams coming together. But there are still a lot of organizations where because of size, because of the challenge, The teams are split up and they're not necessarily talking [00:11:00] to each other in the way that they should be. But that's, that's really hard. It's improving, but there are still cases where people are broken down into those silos.

[00:11:08] And then they bump into each other.

[00:11:09] Mark: Yeah. Yeah, and you know, bumping into each other is probably the best case scenario. The question, you know, when those silos, what about the gaps between them? Right? And, and that causes a host of problems, because you mentioned S3 Buckets right off the start. Those are, you know, everyone who follows me knows that's sort of my pet peeve, Soapbox, because they start locked down, and then somebody makes a decision to unlock them, or implicitly unlocks it without actually making that decision.

[00:11:33] And when you've got these teams not talking to each other, It's, it's a nightmare.

[00:11:39] Paul: Yeah, it's really difficult and there is so much that that can happen that it is like that. So it's hard to get that visibility. And if you're trying to do something, just something as simple as tackle your infrastructure.

[00:11:53] And that typically starts from someone trying to control costs. And you've got to control costs, you've got to get a view [00:12:00] on your costs. And so it's great when that first starts, because that starts to give you, um, that global view. Like, it's one thing to be able to identify a server that's open to the internet on RDP.

[00:12:13] But if that's in a, um, fleet of EC2 servers, and there's 300 of them, And 20 different business groups responsible. Who do you send the email to saying, this server needs to be sorted out? So, in trying to get something in place like that, that tagging, so that you know, uh, who all of the infrastructure belongs to, and then that helps for those cross role teams.

[00:12:37] But at the beginning, it's really hard, and the typical journey that I see is a organization will start off with a proof of concept. Thank you very much. On for example AWS. Okay, that goes so well. It immediately turns into a production workload And then you roll forward two years. Yeah, they've got 150 [00:13:00] accounts that stuff in every region We've got sandbox accounts development accounts project accounts.

[00:13:06] The cost is totally out of control You know what you can't see within an account across regions, let alone across accounts. Your monitoring may not be up to speed. And you go, Oh my God, where do I start? And so the first thing that you've got to do at that stage is, um, be able to get that view across all of your infrastructure.

[00:13:27] But then you end up at the next problem, which is, Okay, so before I didn't know what was happening here. Now I know I've got a whole load of technical repairs. That's it. This exposure and you've got to come up with a plan to sort that out. But even that, you know, if you've got hundreds of thousands of failures and you're discovering things like, Oh, we've got, um, our main database in production and it's running on Microsoft and it's not encrypted and it's 15

[00:13:54] Background: terabytes

[00:13:55] Paul: and you've never taken an unencrypted database.

[00:13:59] And encrypted it. [00:14:00] Yep. Ah. So, it's so hard. And that's just one problem. And that, a typical enterprise, that could take them six months. By the time they make a copy of that database that they can play with in the development environment. Yeah. Where they can take a backup of it. Create a new instance. Make that new instance encrypted.

[00:14:24] Restore from that, uh, backup. Onto the new encrypted instance. Do that five times, work out, it's gonna take them 10 hours to do and then go and tell the CIO. We've never done this before, but we've practiced it in development five times. We think that we'll have an outage of 12 hours, but we're gonna have to get on.

[00:14:44] It's, it's tough. It's really hard to go through that. And those are the kind of challenges because, and if that team had just turned on the encryption flag on day one, they wouldn't have any of that pain. Yeah. And there's so many gotchas like that. So with my [00:15:00] SQL database, you can turn on auto minor update and say, that's just updating the whole time.

[00:15:06] You never end up getting behind. If you don't, you can end up in this scenario where you've been running your database for three years and it's unsupported. You've got to jump through six versions. You've got two weeks to do it. And about to do a product launch at the same time, because you weren't planning that outage.

[00:15:23] It's like, these are, these are the gotchas.

[00:15:26] Mark: Yeah. Yeah, and there's two things in what you said there that really jumped out at me. Well, three things. The first of all, you're very polite in your phrasing because I think a lot of the things I would say, this is just how it happens and it's horrible. We need to fix it.

[00:15:38] The mights and maybes. Very polite of you, Paul. Very polite. Um, but the two things you'd said, the one, the POC and then getting, uh, you know, going to directly to production. For the record, there should be a step in between there where you tear down the POC and then Evaluate your architecture, fix it and go to production, but that never happens.

[00:15:58] Um, and then the, you know, the not [00:16:00] knowing about something as simple as turning on a feature and that will keep you up to date. And I think that's, that's the downside of the cloud is that things go so fast and move so quickly and things like little tiny features like, you know, auto update. Aren't big announcements.

[00:16:14] They just happen and maybe you notice it in the docs or maybe you notice it in the console. Um, but most of us never use the console, so you'd never see it unless you found it in the documentation, and there's something that could be a lifesaver. Um, that, that never comes there. What are the things that, uh, happens, um, in the AWS hero community after re invent is everyone always asks us like, Hey, what's the cool new thing you found?

[00:16:35] And inevitably everyone I talked to is like, Oh, I found this one basically checkbox that I can rip out this whole weird hacky infrastructure I had built to now actually solve this. Just let AWS solve it because it's a checkbox now.

[00:16:49] Paul: There, there are so many of those I found out the other day and I just didn't know, and I'd been looking for it for ages.

[00:16:55] That if you're using AWS API gateway, you can immediately [00:17:00] just get a manifest of your API Yeah, it's like oh my god, that's so amazing and I'd just be walking around blindly going Oh, that's something that I really need to do and really sort and do it by hand But it was there the whole time and it's like there are so many of these cloud can be hard Yeah, and in particular if when you're focused on that one feature You You don't necessarily really understand all the infrastructure, you know, that thing of the beanstalk being consisting of EC2s and auto scaling groups and load balancers.

[00:17:33] You're a developer, you're coding, you don't necessarily understand what a load balancer is doing. You didn't go to load balancer school. It's, it's really hard. And we tend not to have, um, People looking after databases anymore. Developers tend to run databases, which is if if they have no background in running a database then they're going to be doing things like all logging in with the same username sharing it [00:18:00] amongst the whole development team and before you know it that same password is being used in production and everyone knows it and people leave the guys And then you end up with, oh, let's use the same password for every C2 in SSH.

[00:18:13] You start looking around an organization for all the SSH keys, all the AWS login credentials. It ends up all over the place unless, you know, you're really concentrating on that stuff from day one. But if you come in at the two year point, there's a lot of technical debt to work through. They're all built in.

[00:18:33] Mark: And the challenge is, you know, the challenge is understanding. So we're about to come up to AWS reinvent and we're going to get a ton of new toys, which is amazing. But then trying to keep, um, you know, up to speed on all that is almost impossible. Um, but the, the, the frustrating thing I find is, you know, all the cases you're mentioning, I've seen, I've talked to people who have them.

[00:18:52] But there are easy solutions, which means, you know, the gap isn't in the technology. The gap is in that, you know, keeping a pace or finding [00:19:00] out and being aware of these changes. There was a question from the audience I wanted to tackle. So I have a feeling I know what you're gonna say for one. So, but Andrew Brown was asking, he's the CEO of exampro based out of Toronto here in Canada.

[00:19:14] Um, fantastic content creator in his own right. He was asking what's, uh, third party security vendors are good at that visibility problem you were talking about. So seeing across multiple accounts. Now I have a feeling you may say us with cloud conformity. Um, but in addition to that What do you see some tools from AWS or open source tools that will help you track down those 500 accounts, see what's going on?

[00:19:38] What's your trick there beyond obviously checking out the free trial of Cloud Conformity at cloudconformity. com? Not that my marketing people would ever let me say that. Certainly.

[00:19:48] Paul: So there are a number of tools that do do that for you. And what we would always recommend is take some talks. So for example, when I was a customer, I looked at ten different tools and I did a paper [00:20:00] review of ten and then I deployed two.

[00:20:02] Take your worst account where you know stuff is really bad. The tool should find all of that. Take your best account that your best engineer has built and you think that it is totally rock solid. And particularly if it's been done by a third party provider that you paid a fortune to. And they've said this is top notch.

[00:20:19] And run the tool over that and see what you find. Uh huh. And at that stage as well you should also. See things that you just didn't have visibility of before that should or typically scare when a When a enterprise doesn't have tooling like this deployed, they'll run the tool and they'll go, Oh my God, we need to sort that out now.

[00:20:40] There is a database server that is facing the internet that shouldn't be. There are numerous S3 buckets that are facing the internet. All of these kind of things that can be really scary. And then there's another bit of it as well, and that is your tooling should enable the chief security officer to be able to say, when he goes to the board, since the [00:21:00] last time I came and spoke to you, we've not had a breach.

[00:21:02] Thank you. We don't have a breach now, because I'd know, because I'd get an alert to my mobile phone the moment there was an S3 bucket that was public. I would know about it within two minutes. That's the kind of confidence the Chief Security Officer of an organization that is running cloud, uh, architecture should have.

[00:21:20] I like the should have,

[00:21:22] Mark: and, you know, hopefully, and I mean the tools that, that's, like you say, I keep saying, the frustration part for me is that the tools are there. The tools are there to find that, you know, and one of the things I noticed some of the comments in the stream we're talking about, um, cause you've mentioned multiple times, like, you know, people with 150 accounts with 500 accounts, um, multi account is an actual security strategy, right?

[00:21:42] Like setting up accounts.

[00:21:44] Paul: It's a great segregation between accounts. If you're trying to protect things, you need to have those account boundaries. That's the best boundary that you can have in the cloud.

[00:21:53] Mark: Yeah. And especially in, you know, in all three of the clouds, it works similarly, but you can do cross account permission [00:22:00] roles so that you can protect things.

[00:22:01] Right. And then that way you get that immediate blast radius because you know that hard boundary. But then the flip side is you get this challenge of, Hey, we've got 500 accounts now. What's where, how do we know? Similarly with tagging, right? I, I laughed when it came out, but when AWS two or three years ago wrote a white paper only about how to tag things.

[00:22:23] I was like, what are they talking about? But that is one of the most useful resources.

[00:22:28] Paul: It is amazing. It is so good. You can use it for doing your identity and access management, uh, just identifying who owns a stack, uh, things like versions. So quite often when people are deploying into production, you see some funny practices there where yes, it's a blue green deployment, but they leave up the last three deployments in production because, because, and the reason for this is people are really scared in production, right?

[00:22:57] As a developer, logging into production and [00:23:00] doing something, that's scary stuff, and I totally get that. So, you do the new deployment, and you've got the previous deployment, that is the known good. Then you've got the one before that, that is your backup one. And then there's the two that you forgot about.

[00:23:12] And I have seen at least five versions being deployed in production. And if you are not putting a simple tag that says, so you'll have your application tag, and then you do a version tag, and that version could be 70423, but that allows you to pinpoint exactly what's there. And that just makes everything so much easier, from monitoring, to investigating issues, to security, so that you know, when you're going to review some infrastructure for best practice, You're only looking at the latest deployment and you can see that all the way through and it allows you to really neatly identify because we end up with multiple versions of an application.

[00:23:50] And if your tag just says corporate application, good luck. Yeah, yeah, you're not going to know where you are. And there are all these sort of [00:24:00] approaches to how to tackle doing things in the cloud that are different from the stuff that we were doing in data centers and the stuff that we were on premises.

[00:24:09] It's a new world and new challenges and these are the bits that are

[00:24:11] Mark: hard. Well, it's, it's a lot of these come down to scale, right? Like if you're, so in those 500 accounts, if everything's deploying, you know, two or three apps, um, trying to track this stuff down, if you don't have these good sort of hygiene practices like tagging, like clear usage for each of those accounts, how long, you know, you can't take advantage of this wonderful power of the cloud because you're chasing your own tail trying to find out like, Oh, okay, I know there's a problem with the app.

[00:24:38] But where is it? What account is it in? What version is it? What's it running? What's going on?

[00:24:44] Paul: You've got to move over to managing infrastructure as code or as templates. And then driving all the changes into that. So, uh, by doing that you avoid the whack a mole. So if you think, if you're running a production account, there's an [00:25:00] EC2 fleet, there's a whole bunch of servers that are open to the internet on SSH.

[00:25:05] And you don't want to see that anymore. If you just tell the developers to stop that and go and fix their current versions, next month, they'll just pop all back up again. But as soon as you start to adopt templates and ensure that all the changes go into the original templates that live alongside the infrastructure that they represent, and that that becomes your working rhythm, you agree that you're going to resolve something in the environment, The changes go into the templates.

[00:25:35] You don't fiddle with the infrastructure live. You make the changes to the infrastructure. The next deployment, those changes go out. You version all the templates that represent your infrastructure and you drive the changes there. It's a much more streamlined process for managing. You end up fixing things once and you work through that technical debt, and that means you're able to tackle.

[00:25:59] This [00:26:00] month, you tackle EC2s. Next month, you tackle RDS. And it goes on like that. And every time you uplift it, you're just improving your baseline. And you're going for that biggest improvement. And when you get that sync between security and compliance, and development operations, and you just go, What are we going to do this week that is going to make a difference?

[00:26:19] Okay, we're going to go to all the MySQL databases, and we're going to turn on Auto Miner Update. And just getting into that flow, and And I'm driving that into those templates and it, it just makes things so much more efficient if you fix things just once and they go away and then you're just looking for the edge cases and those edge cases are typically when, when you've got other deeper problems, you start to be able to see the wood from the trees.

[00:26:44] Mark: Yeah, yeah. And I mean, that's, it's funny if we, you know, the, these streams. So again, you know, audience hashtag let's talk cloud. If you want to get involved, um, one of the things I'm sure if we look back on the on demand, when you've said logging into production, I'm sure I made a face. I always make a face because I [00:27:00] 100 percent agree.

[00:27:00] It's one of those things where mentally people have to get out of that thought process of the old way of doing things. We now have essentially unlimited capacity, right? It's, it costs you a couple bucks to duplicate your, your production deployment to do those blue greens, right? You have unlimited capacity, so don't log in and fix things.

[00:27:19] Like people always freak out when I give a talk and I say, don't patch production. And they're like, what do you mean? And I'm like, well, never touch production. It should always be stable. If you identify an issue, go back into your infrastructure as code, fix it there and redeploy because now you've got consistency.

[00:27:34] So even if you mess up, you've got consistently messed up things, which are easier to fix.

[00:27:39] Paul: You're, you're at a known good and it's going from this kind of wild west. to enforcing your vision of how it should be. You're not going to fix everything. There are so many things that need fixing that can be non optimal.

[00:27:54] But if you go looking for the big things that make a big difference and that also makes things easier for the [00:28:00] developers because they're not having to go being told to go and sort something out that they sorted out last month and your infrastructure becomes. Better all the time because, um, it's the usual 80 20 thing where the 80 percent improvement will come in that first 20 percent of improving stuff.

[00:28:17] And then you're able to go after how you improve your application and the business that you're about. Because you're not in the business, really, of running cloud infrastructure. You're in the business of whatever your business is. And the more time your team's able to go back and focus that, That's gotta be the winning strategy.

[00:28:33] Mark: Perfect. Yeah. Well said. Very well said. Um, so a couple things, you or a couple of pop things have kind of popped up briefly in the conversation, but I want, I want to turn around and be a little more explicit in the focus. So one of the things you mentioned, if you're not keeping an eye on your applications, if you're kind of leaving them sit for a while, you may be missing out on, um, things like cost, right?

[00:28:53] So, cost, you know, the cloud, I saw a thing from RightScale a while ago. Um, that they did a survey and people were shocked at their [00:29:00] cost, uh, in the cloud. Um, and I find, I always find that frustrating because I think you've got better visibility with the granularity. And yes, it can be hard to figure out that granularity because all these little microtransactions.

[00:29:12] But, I find it's much easier for a business to align, you know, action A to cost B. And you get that direct, like, okay, I know what I'm spending on, I know what I'm getting out. Um, based on, you know, your experience as a user, as somebody who's, who's managing security, has that pay as you go approach adjusted how you think about your monitoring and security tooling at all?

[00:29:35] Um, what's the impact there?

[00:29:38] Paul: Certainly for the, the pay as you go thing, for me, means that you can experiment much more quickly. If you're not locked in to a year contract, Or you're buying a security product that's 150, 000. You can just turn it on and run it for a month. So for me GuardDuty is a great example of this.

[00:29:58] You just turn it on, run it for a [00:30:00] month, it costs you nothing to evaluate it. And by the way, we would always recommend using GuardDuty. But, in the old world, you would have had to have run a proof of concept, maybe a number of different products. You'd have had to have done all the commercial negotiations.

[00:30:16] It might have taken a year, and for a product like that, you'd have been paying 150, 000. You can turn it on, and it costs next to nothing. And it comes with some unique features, so, it's worth it. You know, GuardDuty is able to look at the VPC flow logs, the DNS lookups, it is also munching all of that data against third party threat information, and then sending you alerts on anything that it finds, so for me, that's a fantastic capability, it's really easy to turn on, and if your experience is bad, you can turn it off.

[00:30:49] Yeah. At the end of the trial, or at the end of month two and go, uh, that's actually more expensive and we've already got a different capability that takes care of that. That's fine. But what that allows you to do [00:31:00] is do that, have a look at stuff, if it doesn't work, you can actually do the real trial rather than that really formal, long process, which to me is too old and, you know.

[00:31:10] Not doing that anymore. I think the other thing that it does as well, is you don't need to involve sales. You can go to Marketplace, you can choose a product, deploy it, run it, and at no stage have you spoken to sales. And for a number of people, that is really attractive. You know, I'm totally capable of downloading a a set of rules for a web application firewall, setting them up, running them, and evaluating them.

[00:31:36] And I'll decide if they're I'm going to buy them. There's a lot of people who like to buy like that, and they can do it 24 7. Yeah, so, when I look at how Amazon revolutionized buying a book, I think the marketplace

[00:31:48] Mark: is revolutionizing how you buy software. For sure. Cause then also, you know, I find, um, you know, cause I used to, so I've been with trend for seven and a half years or so before that I was with the Canadian federal government.

[00:31:59] So [00:32:00] very traditional buying infrastructure. If we wanted to buy anything, we had the vendor come in and set up a POC. They would normally come and say, here's how you can test a whole bunch of stuff. Um, and I find that ability in the marketplace to try something and test it against my own criteria, right?

[00:32:15] As long as there's good documentation there so that I can, you know, I can get help if I need it. And that's changed how I, as a customer, am doing things because I go, You know what? This is my problem. This is my use case. I don't really care about the rest of the big bubble. I want to solve problem X. And I'm going to test against problem X.

[00:32:30] And if this doesn't do it, I'm going to move on. And at the end of the week, it cost me 10 in various fees to various vendors. But now I know, right? I have hands on experience with all these things. Whereas before, that was really, really difficult to pull off. Alright, Paul, I'm going to hit you with a couple of things here.

[00:32:46] I want, uh, I want to, this is, we tried this out on the first stream and it worked really, really well. I want rapid fire, sort of one, one, two word, uh, responses, uh, and then we'll, then we'll dig in a little bit more, uh, depending on what you [00:33:00] say. So no wrong answers, except, you know, all the wrong answers I don't agree with.

[00:33:04] Um, so, uh, compliance, useful or just a check mark?

[00:33:10] Paul: Oh, really useful. So. Yeah, yeah, yeah, I gotta hold you

[00:33:17] Mark: to short, cause then that makes it, we'll come back, we'll circle back, don't worry. Alright, uh, we're a year plus into GDPR, has it advanced security or has it isolated it more as an organization or an activity?

[00:33:29] Paul: Definitely advanced.

[00:33:31] Mark: Okay, uh, bucket permissions on S3, real issue or just headlines that are, uh, attention grabbing?

[00:33:39] Paul: Real

[00:33:40] Mark: issue. Okay, uh, PCI DSS? Uh, modern or outdated? Yeah, still

[00:33:49] Paul: relevant to those customers who have to use it

[00:33:53] Mark: Very safe answer my friend very safe answer, uh security automation lip service or really [00:34:00] real activity

[00:34:01] Paul: If you're not there You're struggling.

[00:34:04] You've got to get that.

[00:34:05] Mark: Okay last of the rapid fires audits overvote overblown panic or legitimate concern

[00:34:13] Paul: are legitimate

[00:34:14] Mark: Okay. Let's tie that with compliance and the audits. Um, so where, where, I mean, you live this as head of security, right? This is your bread and butter. You have to deal with that. Um, you know, auditors are real people.

[00:34:27] We give them a chance. Once you get to know them, they're nice, you know, initially, not so much. Um, but, uh, what, so what's your take compliance and auditing? What's the advantage to organizations?

[00:34:38] Paul: You don't know what you don't know. So if going back to that, you've had your pock, you've been running for two years.

[00:34:45] Another way of tackling that would be to run an audit across the infrastructure. So I would take something like the CIS baseline for cloud security. And if you take a scenario, and this is not uncommon, an enterprise has been told, the [00:35:00] security team, one of the big four auditing companies coming in a year's time, they're going to audit your cloud infrastructure.

[00:35:07] How do you prepare for that? And what we would advise in that scenario is to take a baseline, like the CIS baseline, and then apply that against your infrastructure. See where you have the failures on day one, and make that your baseline. And then every month for the next 12 months, you're going to tackle the top five issues.

[00:35:28] And when the auditors come in, you won't have done all of it. But you can tell a really good story about we've adopted this baseline

[00:35:35] Background: Mm

[00:35:35] Paul: hmm, and this is how we're working to uplift our infrastructure and show some of those things within the baseline Won't add a great deal of business value, but some of them really will, and some of them will identify some major issues.

[00:35:49] But you've got to align to the whole standards, the standards written for every type of company, every kind of environment, so that there are going to be some edge cases there. But generally speaking, again, [00:36:00] 80 20. I'd be happy if 80% of a baseline added real value and 20 didn't. Okay, that's fine. You've gotta start somewhere.

[00:36:07] And hey, they've come up and they've come up with that baseline. If you had to come up with your own control standard for internally, you know, you could spend a year working that out. Take someone

[00:36:18] Mark: else's. Yeah, fair. And that's, that's solid practical advice. You can tell you've lived it cause that's a reasonable pragmatic approach.

[00:36:25] Right. And you know, you keep using that word baseline. And I think that's a key one, right. Is that, that's the baseline you should be at. And then once you've achieved that, you should be looking at your specific threat model to move beyond it. Right. So maybe the, yeah, you know, cause everyone's got a unique scenario.

[00:36:40] And I do, we deal with this quite often when people ask about hacks and they're like, Oh, was it, you know, was it a nation state? And you're like, Oh, that's not the risk model you should be worrying about for your company, right? Most companies, standard cyber criminals. So let's defend based on your risk model.

[00:36:55] What's your secret sauce, right? So the example I always use, it's very [00:37:00] flippant, but it works well is, you know, you don't put a ton of effort into protecting the cafeteria menu, but you darn well better do your best to protect customer data and financial information, right? So what's important to you?

[00:37:12] What's important to your business? Um, and that's what you go after. So that baseline and then move forward. Um, what's your take on audit evidence? I find this really interesting. Um, so, let me just get you to give me a little spiel on audit evidence and then I'll ask you a little more depth.

[00:37:29] Paul: So, to me, when it comes to an audit, what I'd be looking for is a policy that says, this is what we do.

[00:37:37] They're not expecting to see a process. That is how you do the thing that you say that you're going to do. And then I'd expect to see the evidence of you doing that. And whatever that reasonable evidence is for that. And with automation, so for example with PCI DSS, you used to say, Okay, so, There are a hundred servers.

[00:37:55] We're going to go in five and you put that in as your evidence We selected five out of a hundred but [00:38:00] with automation you're able to check all of those servers and you're able to check them 24 7 so that will give you an advantage because if you're going out of compliance You'll know about it as early as possible You'll have the largest window to be able to fix it.

[00:38:13] You're also able to show the whole time that you are In compliance, which is unusual because in the olden days, how could you prove that? So it's really easy for anyone to say any compliance standard. Ah, at the time they got hacked, they weren't compliant. Who knows? Who could prove it? Whereas today you could actually say, actually, that workload was compliant with the standard and it was still hacked.

[00:38:34] So we did our best job, but yeah, and that could potentially happen. Um, when it comes to the evidence, if you're looking at something like AWS, Cloud infrastructure in general, you go looking for the lowest level API call and if the tooling that you use is using that and you're able to go and test one thing and then go and check it in CloudTrail and [00:39:00] be able to marry up the two and say, look, we're showing what that says there.

[00:39:04] Then once you've shown an auditor that for one thing, they're usually happy enough with what you're doing. That's, that's quite often a customer concern. But when the tire hits the road, I rarely see auditors challenging that because typically by that stage, um, the enterprise involved has done so much work on their security, their resilience, even making sure that their platform is optimized for cost.

[00:39:34] Because that's an important thing like you can't build a workload that's going to cost a fortune to run Because you're not going to be able to run it like that and that's not going to prove anything to the auditors And they also want to be able to see that it's resilient You know, it's it doesn't matter whether it's secure if it falls down every two minutes And what are you going to do if that AZ goes down or that region goes down and having that kind of approach in your planning is It's really useful.[00:40:00]

[00:40:00] Mark: Yeah, because I think unfortunately this week a lot of people are going to have to deal with that scenario because GCP, a bunch of the compute services went down, uh, Monday morning North American time, um, and that's, that's a great advice. Plan ahead for that. Show that there, uh, that disaster recovery, that resiliency.

[00:40:15] And one of the things that really jumped out at me in your answer there was that automation. Um, so the traditional way of handling an audit was, you know, you'd freak out on Friday because you remember the auditor showing up on Monday. And you'd scramble to get a whole bunch of evidence. But if you're, I see you've had this experience, um, if you're dealing with the cloud, if everything's infrastructure is code, I always try to advise people to think about that when you're doing deployments to have as part of that deployment.

[00:40:42] audit evidence that gets recorded at the time to say, you know, here's what's out there. Here's what's changed versus the last time. And if it's, if the system's taking care of it for you, that makes your job so much easier as far as monitoring that compliance and then proving it to the auditors and self auditing so that you get that continuous [00:41:00] compliance going, right?

[00:41:03] Paul: If any of your audit evidence or the way you're managing controls is in an Excel spreadsheet, you've got a fantastic opportunity to automate that. Yeah, you've got to go, go beyond that. You're staying in a very painful place if that's where you are.

[00:41:20] Mark: Yeah.

[00:41:20] Paul: And

[00:41:20] Mark: that's,

[00:41:21] Paul: that's a big opportunity.

[00:41:22] Mark: Yeah. And again, you're so positive.

[00:41:26] It's amazing to find somebody who's still doing like hands on security work. Who's positive. Normally we're, you know, like four days of beard, grumbly in the corner with a drink going, Oh, I can't believe this, but it's good. That's good. That means, you know, you're making progress. Um, let me ask you then. So, cause you are so positive.

[00:41:42] Um, obviously, uh, you know, you have, Uh, buy in with the people you're working with, right? Because it is your head of security of cloud conformity. Um, you've gotten the buy in from the people and you're implementing this stuff, right? We're a cloud

[00:41:56] Paul: security company. We were born in the cloud. So [00:42:00] yeah, this is our bread and butter.

[00:42:01] We do this every day and everyone passionately believes in it. We've got a company who are incredibly passionate about cloud security and what our product does and helping people. And there is such a, an opportunity. So, you know, within the cloud shared responsibility model, there's a lot that's back on the customer side.

[00:42:19] That's hard. And we are seeing it, our opportunity is making that as easy as possible for, for everyone. Yeah, so for example, our knowledge base is public facing. There's over 500 AWS best practices anyone can look at. And it tells you why it's bad, how to check for it using the console and command line interface.

[00:42:37] How to fix it using command line interface in the console. You don't need, you know, to buy our tool or anything like that. And that's out there the whole time. That's how passionate we are about making the cloud easier to secure.

[00:42:50] Mark: Yeah. And, and that's a fantastic, uh, resource. I just dropped the link in the LinkedIn chat.

[00:42:54] We'll put it on on Twitter as well. Cause I love it as well. The KB is very straightforward. Like here's how you do it in the [00:43:00] browser. Here's how you do it in the CLI. Awesome. Um, we had a comment, uh, from Andrew, uh, who I know well, he's a great guy. Um, he made, you know, a, a, a joke remark, but said, you know, if, if your audit evidence is an Airtable over Excel, it's okay.

[00:43:14] And, uh, I would say, you know, as much as he's joking, because Airtable's the new web based Excel, I would say if your output is in Airtable, that's a great way to visualize what you're automating in the backend and feed that into the Airtable API to get a human readable feedback. You know, red light, green light, you're good.

[00:43:33] Um, so, you know, as much as Andrew was joking, uh, air tables, great, uh, quick site is great for that in AWS as well. If you dump your evidence into an S3 bucket, visualize it in quick site. And then I find the strategy there is really good to have for the team's hands on very granular dashboards. When you get higher up the level to the execs you get simpler and simpler You know think like going from like really high art tools to like crayons at the top [00:44:00] Because that's their level of concern.

[00:44:02] Are we good or not?

[00:44:04] Paul: Just showing a simple percentage figure on where you are with security where you are with cost optimization Where you are with your workloads being performant It makes it really simple to be able to compare account to account, team to team, and we find that people find that a really easy way of viewing things when it's executives looking at their posture.

[00:44:26] Mark: Absolutely.

[00:44:27] Paul: And if it's automated, then it's unarguable. Right. And then when you make changes, then when you improve it, it really is improving.

[00:44:34] Mark: Yeah. And you're moving to quantifiable, uh, right, as opposed to the, you know, this is always frustrates me with threat models where it's like, Oh, it's high. Based on what?

[00:44:44] Whereas, you know, having evidence based security is really fantastic, right? Um, so this has been a great conversation, Paul. I really appreciate it. I do have one final question for you. I thought you'd mind. Yeah, it's more of a personal one and it's not a bad one. Um, but what's, [00:45:00] what's your favorite cloud service?

[00:45:01] So, you know, something specific within AWS or GCP or Azure. What do you like personally?

[00:45:08] Paul: It's really easy. We're serverless, so for us, it's Lambda. We can fix pretty well anything with a Lambda. They're just, they're so cool. It's the future, and we love it. So, we're really passionate about serverless. AWS Lambda, awesome.

[00:45:23] Mark: Fantastic, and I think, knowing how much that's been sort of through the AWS Summit series since the spring has been getting talked about more and more, I'm pretty sure we're in for a lot of cool new toys to help us manage Lambdas in Lambda itself, uh, come reInvent. Um, you'll be at reInvent this year?

[00:45:40] Yes, looking forward to it. It's going to be crazy. 65, 000 plus people. Um, so Cloud Conformity is a new part of the Trend Micro family. We'll be at the Trend booth, um, in there. But, uh, so you were at reInvent last year. Do you have any tips for attendees at reInvent at all based [00:46:00] on your experience last year?

[00:46:01] Paul: I, I would say don't get too hung up about getting to every single presentation. You can always catch up with those afterwards. So those are going to be recorded. Uh, for me, it's the interaction with the AWS people, with the people from other companies, with customers, all of those things. That's the value of reInvent.

[00:46:20] The event itself meeting people and spending the time there. That's fantastic. That's great advice. Yeah, and this year There's a whole bunch of new events. I know talking to Annie and Jill who run reinvent for AWS And they've the pub crawl has kind of gone away. There's still some pub events, but they brought in board game night Movie night a whole bunch more sort of social networking events To get people together because yeah, everything is up on the AWS YouTube channel within a week or two.

[00:46:46] Um, and you know, there's tons of stuff going on on Twitter, uh, which we've got our Twitter handles up on the, uh, on the banner below on the Chiron below. Um, but yeah, that's great advice. Uh, also bring a good pair of shoes cause you're going to be walking a lot. It's a [00:47:00] big area. It's huge. It is crazy and it's gotten bigger.

[00:47:04] Uh, so with that, Paul, I think we're going to wrap up this stream. I really appreciate you taking the time. This has been a fantastic conversation. Thank you to the audience. We've had a lot of great talk, um, around the hashtag. Let's talk cloud. This was episode two of many to come. You can see online or on the screen right now, uh, Paul's Twitter, uh, follow him, uh, on Twitter, uh, reach out, uh, you know, tell him what you thought of this, uh, ask him any questions, all that kind of stuff.

[00:47:27] Always good. I'm just randomly roping you in for extra work at this point, Paul. Thanks, Mark. Yeah, well, you know, hey, it's Welcome to the team. So with that, thank you very much, everybody. We appreciate it. Uh, remember, uh, keep this conversation going. We will monitor the comments and reply on LinkedIn, on YouTube, on Twitter.

[00:47:46] Um, and we'll see you next week for the next episode. Thanks a lot.

Read next