This post was originally written for
Trend Micro .Reasonably Accurate 馃馃 Transcript
[00:01:13] Mark: Perfect. All right.
Good morning, everybody. Uh, good afternoon, good evening from around the world. We very much appreciate you guys joining, uh, us for the fourth episode of, uh, Let's Talk Cloud. Um, just a quick little reminder on a logistics.
Uh, we are streaming live on LinkedIn, as YouTube, as well as on Twitter. Um, please feel free to join in the conversation, ask your questions on any of those platform. You can use the, um, #letstalkcloud as well, um, if you wanna expand that conversation out.
[00:04:14] My name is Mark Nunnikhoven. I'm the, uh, Vice President of Cloud Research here at Trend Micro. Um, and this series has really just been about kind of, uh, helping you guys, uh, learn about challenges, um, mainly security focused, but in general, uh, with building in the cloud, with modern application development.
We've talked about- uh, in this series so far, we've talked about, you know, real world problems from the trenches. Um, we've talked about, um, containers. Uh, you know, we're talking about all po- all sorts of different things.
[00:04:43] So today, what we are talking about, um, is open source risks. Uh, and to do that we have brought in, uh, one of the top experts around the world. Uh, joining us today us, uh-
[00:05:41] [silence]
[00:05:41] Brian: ... level. Uh, as my story will show, there are a lot of things in your program that are not yours and you just use it. So, uh, um, I'm most active in the Java space.
So, I'm a Java, uh, developer by trade, but I did a lot of other things. Um, and now I'm a developer advocate for a, um, for a security company and I'm very glad to be here.
[00:06:06] Mark: Fantastic. Thank you for, uh, for that. And just, uh, for those of you on the stream, yes, the audio cut out there for a second. We are back now. Um, Brian was just giving us a little of his, uh, background. He's a developer advocate for Snyk.
Um, very active in the development community, as well. You can see from his profile, um, if you check him out on social, he gives talks around the world, uh, helping, uh, us developers understand the challenges of building in the modern age and pulling in, uh, open source and, uh, all of that. So, fantastic.
[00:06:35] Um, and today's a little bit different than the streams. Normally on these streams you're gonna see, uh, you know, myself and a couple guests and we just go straight conversation for, you know, 20, 30 minutes or something like that.
But um, we've got a real treat for you today. Brian is actually going to, uh, give us, uh, one of, uh, his presentations that have been really well received, um, at a number of fantastic developer events.
[00:06:56] Um, so we're gonna cut now to Brian, who's gonna share out his screen, um, and we are gonna, uh, watch, uh, this presentation. If you guys have questions during it, hit us up on LinkedIn Live, on YouTube, or on Twitter.
Um, and I will act as sort of the proxy for the audience, um, and, uh, ask that question if it makes sense in the moment or will queue them up for the end, and we're gonna have a little talk at the end. Uh, so Brian, I'm gonna switch over, uh, to you, if you wanna take it away, um, with Stranger Danger.
[00:07:24] Brian: Uh, will start sharing the screen now. And you probably see my slides, because I will start with a few slides and afterwards, I will do some light hacking as well, because we are gonna talk about finding security vulnerabilities in your code, and necessarily they are not yours. That's what the Stranger Danger says.
[00:07:44] So, let me... Well, you had an introduction. I'm a developer advocate for Snyk. I do a lot of community stuff for some Java user groups and a user group- uh, online user group, uh, that is specifically there for AppSec.
So, AppSec all around the world, so, uh, that means application security, but also container security. So, if you wanna know more, please, uh, join MyDevSecOps as well for more, uh, uh, info.
[00:08:11] But let's get, let's get to it, because we are gonna start with DevOps. And we tend to like DevOps. And I like DevOps too, but the question is why? Because what we do with DevOps is we try to cut away walls.
We want to make sure that the walls between your original Dev silo and Ops silo is not there anymore. And eventually, we want to make sure that we deliver faster. We want adapt to the market needs and we want to make sure that if we have a new idea, we do not silo it anymore and we build it as a Dev community, throw it over that wall, and Ops will make sure it works. It's more of the thing like, if you build it, you own it.
So, you need to take ownership as a developer, and as an Ops person you need to be aware of what is built.
[00:09:05] But doing that and getting faster and delivering faster gives us some problems in the security space. That's why this new term is coming up, DevSecOps. And that Sec is in the middle, because well, security is either in- it's, it's, it's not... Uh, it- it's still a siloed thing nowadays, but it should be part of both Dev and Ops.
And to make people more away of that security part in that whole thing, uh, this term was invented. Although, I think security, say in a couple of years from now, should be a first-class citizen in a developer's and in operations people's mind.
[00:09:48] So, but what, what do- what, what do we try to do with DevSecOps? Because we invented something new, we invented a new word, we invented a new way of working, but we- we're trying to solve problems, right?
So, the first thing is what I already said. We're trying to speed up development. Like, a few years ago when I was working at a- one of the larger banks from the Netherlands, I, I, I live in the Netherlands and I worked for one of the major banks over there, and we'd released practically three times a year, and that was about a decade ago.
[00:10:19] Nowadays, when for instance you work in eCommerce, or well, it doesn't matter actu- actually what company you work for, we're speeding this up. And it might be that you deliver things and ship things multiple times a day.
But the point is, we're focusing on new features. We're focusing, we're focusing on stuff getting out, because we want to be ahead of our competitors. That doesn't necessarily mean that security is, is on our mind. Security is not- we do not have the security mindset, right?
[00:10:51] And because, uh, uh, of that, uh, thing, there is a lack of focus, and not only on, on the, the Ops side, but also on the developer's side. And the main thing is, because in these companies, even though you do DevOps, at the end there is that security, so you- se- se- security part, security counterpart that you work with. And for now, from a, from a developer's point of view, there are basically the people that say no.
[00:11:20] I mean, we're trying to build things, we're trying to build new features, we're trying to ship them as fast as possible, and at the end, somebody from the security depart- department says, "No, because well, it's not safe enough."
And then we have to go all the way back and try again. So, there will be a little bit of tension between developers and the security co- counterpart, and we want to take that away. Because in the end, we have the same goal. The goals to make sure that we deliver stuff, but in the end, we do not want to get customer data out, because we know what can happen, right?
[00:11:54] If customer data gets out and PII data gets out, we have these big headlines, uh, in the newspaper or on every, uh, uh, uh, uh, news channel. And well, you could say that it's good marketing, but some people will, will, will disagree on that. Because if you look at Equifax, I mean, if I asked people, "Do you know Equifax?"
They know Equifax because of the breach they had. If I asked people, "Did, did you know about Equifax, especially outside of the U.S., uh, did you know about Equifax like five years ago?" Everybody would say, "No. Who is that?" Except if you're in the U.S. already.
[00:12:32] But the problem with Equifax was, they, they didn't do anything wrong particularly. And I don't want to focus on the company, I wanna focus on the breach they had. Because what happened is that they were using an, an outdated package.
They were using the outdated Apache Struts Library. And at a certain point, somebody found a vulnerability. The vulnerability was disclosed and there was already a new package. But the point is, if you are not aware of that and you're still using that old vulnerable package in production, you are vulnerable without knowing it.
[00:13:11] And I mean, if we look at the problems with Apache Struts 2, the point when it came out, when it was published, it was only a few days that people were attacking and, and, and looking into that thing.
But once you're in and somebody doesn't know that you're using that library, you can snoop around, you can dig around, you can literally take your time to see what is out there. What can I, what can I use, what can I see?
[00:13:42] And in the end, at this point for, for Equifax, the problem was that over 140 million records were exposed. And I wanna show you what happened, because that hack was not trivial.
So, I will cut this down and I will go into an application over here. This is a Java application that is deployed on Heroku. So, it's a live application on a Cloud instance.
[00:14:08] I will go to the terminal, because I will show you what the problem is over here. First of all, somebody found out that we could exploit Apache Struts 2 with a certain header.
And I will show you the header over here. It's not actually a header. It's a content type. And that content type is very fuzzy, as you can see. Actually, if you look at this content type, it's an illegal content type.
[00:14:32] And what we do in this content type, because it's an illegal content type, the flow of a request using this content type goes into an exceptional mode. And in that exceptional flow, we're trying to utilize, um, a domain specific language, or a- uh, uh, we were trying to util- utilize old GML, object grab naviga- navigational language, which, uh, which is part of that Apache Struts, uh, instance.
[00:14:59] And with that expression language, we could not only, uh, manipulate objects in runtime, but we could also create new objects. And that's what we see here. If we see here, we accessed the process builder in Java.
And what we do with the process builder is we set- we say, "Start up a new process." And that process is a backed process. And with a backed process, we just feed it a command.
[00:15:25] So basically what we do, we starting- we're starting up a new process, probably a new trend, and we feed an arbitrary command. And if this works, we can execute arbitrary code. And that is in general what happened to that, um, uh, to Equifax or to, uh- that, that is the problem with that Apache Struts Library.
[00:15:46] And to show you that, I will show you this command over here. I will copy/paste it to Nitro and then explain to you what happens. Because what we do is, we use this header and we, uh, we make a code, uh, code request to, in this case, my, my instance on, uh, on Heroku.
And we feed the nth command. It basically asked me to show me the environment variables on my, uh, cloud instance. By running this, you will see that in the end, I will have, over here I will have my environment variables on this instance on the cloud.
[00:16:23] And I've seen what my Java, uh, op- options are, but I also can see what my- where are my- where the location of my Java home is. And this is just one thing we can do, but we can execute any arbitrary command using this header. And it's not the problem, like, finding that header
But once this is found and published, everybody can use it and it is just- people just blindly start to fire on your website, on somebody else's website. And if you're in that crossfire and people, people find- found out that you are using that library, you're just a victim and people will go in. That's basically what happened with, uh, with the Apache Struts Library.
[00:17:04] We have to, uh, remind this one, because I will come back to that, uh, to that Java Home Version in, uh, in another hack later. But I will go back to my slides now to explain to you, um, a little more. Well, we did the hacking part, but let's get to your application.
If this is your application, say this big, big yellow ball, and make comparison to the binary... If, if you put this binary into, well, a container or, or, or a cloud instance or a, or a web server, it doesn't even matter, but how much of this application is actually your code? Basically this.
[00:17:41] I mean, there is a small portion that is your code and a large portion that is framework. Because we depend heavily on frameworks, third-party libraries, and so forth, because we do not want to take care of all the boiler plate stuff.
We do not want to create yet another crap repository. We do not want to create another REST endpoint. No, we want to focus on business value, and that is a good thing in my opinion. But it also means that we are depending on other people's code, basically blindly.
[00:18:13] And if you ask a general developer, "Do you do things like code reviews?" They probably say yes. But how well are you aware of what kind of dependencies you are using? I mean, we have a server as an example over here. It's 19 lines of code.
It's no Java code. It has two direct dependencies. One of them is AWS SDK. That means dependencies depending on other dependencies. So, in total we have 19 dependencies.
[00:18:44] And if I would ask you just a second to think about how many lines of code would I put into production, you'd probably say, "Well, a few thousand maybe." But it will end up in almost 200,000 lines of code. The majority of that part, again, is not yours and we trust on it blindly. Same with Spring and Java.
This application was a 222 line, uh, line code, uh, uh, code example that we put in production. It had five direct dependencies, which end up in 54 dependencies in total. And in the end, with all the libraries and all the stuff we put- we, we put in to and, and we auto wire in this, uh, Spring application, we ended almost with half a million lines of code.
[00:19:33] And then the point is, are you actually using these lines of code, and do you know if you're, you're using these lines of code? There are- there is a lot of fuss going on. People might say, "Well, half of this you are not using." But are you sure?
I mean, for Java for instance, we have the reflection API. Maybe there is some, some by- some, some byproduct manipulation. I don't know. But are you sure you're not using this, and do you want a potential thing that you're not using, do you want that vulnerability in your code?
[00:20:03] Well, open source has exploded in, in, in the last decade, but the point is, if an open source library is there, it's written by maintainers. Just like any developer can, uh, uh, can, uh, commit to and can help maintain sort of packages, it is not safe, nor it's unsafe.
It's in between just like your own code. And if there's one vulnerability in an open source package and many people are using it, if I found that vulnerability, I have a lot of potential victims.
[00:20:36] So, we say that open source is amazing and we think open source is amazing, but we have to take care, we have to take responsibility, and we have to know what we're using, right? We created a report at the beginning of this year. We called it the State of Open Source Security to look into this.
And if we look at the amount of packages that are out there, we see that- we know that NPM and Maven Central, so basically the Note and the Java community or ecosystems are the biggest ones. And they grew massively last year.
[00:21:08] NPM grew enormous and there are now over a million packages available. But even for Maven Central, there, uh, uh, there was an increase of over 100,000 packages between January 2018 and January 20, uh, 2019. We're working on new reports, so new, uh, new stats will be, uh, will be out, uh, shortly at, uh, I think end of, uh, end of Q1 in 2020.
But just to give you an example, and with the number of vulnerabilities that were disclosed each year, it is very scary to see that that Mav- the Maven Central and, uh, NMP have a large growth between 2017 and 2018.
[00:21:49] The point is, in most cases it's not the dependency you put in. Again, if you take a look at NMP and Maven Central, the two biggest ones, the direct dependencies are not the problem. But because dependencies are depending on other dependencies, it can be, like, three layers deep or something, where the problem actually is.
And you should know, or do you know which packages, your package that you just imported, uh, are- is depending on? So, three layers deep. Do you still know what you're using in your system?
[00:22:19] And then we asked, who is responsi- who is responsible for security? Well, if we see this task, you see that it's not just one person or one group. It is a split responsibility between most of the time security teams, operations, and developers.
But as we are there, I mean, we can see that developers do have a fair share and a lot of people think that developers should take awareness and they should take responsibility for this.
[00:22:53] Well, we can ask any further, like, "Are you confident about your own security?" 63% said, "Well, kind of." But again, if we compare that to your peers, how do you review- uh, do you review their stuff? But do you use their- uh, do you use the dependencies blindly or are you actually looking into that?
[00:23:15] I will go into a few, um, few more things. Um, and I will, um, and I will go into a few, a few hacks as well. So first of all, you see that I have an application over here and it is- it's a simple note application. It is a to-do app.
So, I can do things like buy flowers. Nothing more, nothing less. It's a simple to-do app based on Note. And I have an about page over here. You see I'm a very good UX Designer and it is, um, it is the, it is the best, best to-do app ever, right?
[00:23:49] So [laughs], but not to make too many jokes. If we want to, if we want to hack this, because we're- I know that this app is using, uh, a library called BST Library. And BST Library has a vulnerability, uh, which is a path reversal vulnerability.
So, I will go back into my, uh, into my Bash. And um, I will go and, and do that path reversal. Normally, I won't do this in a, in a browser, because the browser will take care of a lot of things, will normalize a few of the things. And if I do curl commands, these, these things are raw. These commands are raw and I can exactly point out what I want to put in.
[00:24:27] So, if I curl something like this, it was local host 3001, public, and it was about that HTML. And you see I will get, get my plain HTML out of it. Normally if you do path reversal, you will probably do something like, uh, like this. A ../../ and you do that a certain amount of times, because if you're in the root and you do ../ in the root, you're in the root.
[00:25:00] So, we're trying to get out of that sandbox called my web container or ye- uh, my, my public folder or whatever we want to do. And if I do this, I will get an HTML back again. It lets me find out this is just my, my homepage.
Because BST Library is a real library, and the problem is... Or it's not a problem. It actually looks at the fact that I tried to do path reversal. But we found out that it does blacklisting. It blacklists that ../ and we found out that you wanna ../, you block this and we redirect you to the homepage.
[00:25:39] So, we have to be a little smarter. And I will tell on forehand that things like escaping will not work, right? If I do this, I still have that same, same thing. So, what does work is things like this, and it's just URL and coding.
So, if I put this one in and I do it a certain amount of times, I will get the direct restructure of my local machine in this case. And if I will do the same and I will go to, for instance, the past WD file, I will have access to that file.
[00:26:20] The point is, this, this exploit may not be very, very heavy. It may not be very severe, because you can read. But it's an entry point, because now we can read.
And of course we shouldn't be using- we shouldn't be running an application as root, but it's always- it's not- it- it's never just one problem. It's always a chain of problems that we're, we're, we're handling with.
[00:26:44] And if we can do this, we can look at a certain, uh, uh, certain things, and use that, use that to start an investigation and maybe find new things. Maybe we can find property files or if I do things like, uh, like this, I just... I'm not going outside of the whole box, but I'm just trying to get outside of the public folder.
And now I will see I have access to my package.json. So, if I do this and I say package.json, and now I have access to what kind of libraries I'm using. And for instance, I'm using this library, the MS Library. Well, we put it in there on purpose of course, because that MS Library also had a vulnerability.
[00:27:35] Normally I would, uh, I would ask people, like, "How, how much do you like regular expressions?" And most people do like to write regular expressions, but not to read the regular expressions, nor to debug other people's regular expressions.
Because if we have a regular expression like, uh, like this, and we want to match that to, for instance, this string. Of course we know that this works, because well, it's a list of- uh, it's a string of only containing the letter A. But we do not know if it uses the plus or the star or the star to halfway and then the plus. It depends on the, on the, on the regular expression entered, right?
[00:28:16] But what we know about regular expressions is that if it doesn't match, it goes into backtrack- into some sort of backtracking mode. And it tries every single pattern that is available or that is valid in that regular expression to match it against this string. Right?
So, what we're trying to do is, we know that a note application works on only one thread. And if we occupy that thread with the load of an- a very large regular expression that goes into backtracking mode, we just make sure it's not available anymore. And we create a regular expression denial of service, as we call it.
[00:28:59] And imagine that you're, that you're, uh... Well, it's almost Black Friday. Say you are a, a eCommerce platform and you were running on one note and I can just feed a regular expression that goes into backtracking mode and you're not available on Black Friday. Then they- then your customers would probably go to you- to your competitor, right?
[00:29:19] So, let's try this one. I'm going back to here and, um... Well, I'll put up one of my, one of my examples. The first example is just to show you, um, how this works, because I'm using a common line, uh, HTTP server called HT-PI.
And by doing this, I will, I will say, "Call mom in 20 minutes," and I will fill it up in this form. So doing this, you will see it comes back and we will have something like, "Call mom in 20 M." So, it does something with the 20 and the minutes.
[00:29:55] So, let's do the next one. What we're trying to do here is, we're creating a regular expression, or we're using a regular, regular expression. We create a, a list of 60,000 fives and try to match that to minutes. And again, put that in the form.
By doing this, you will see that it still works, but it takes a little bit longer. And if we look at the results, we will see that it matches the enormous list of fives with minutes to build up into infinity days. Well, that's kind of hard to do, but fair.
[00:30:38] The question is, how do we get into that backtracking mode? The answer to that is quite simple. Just make sure it doesn't match. So, let's get to the word minutes, and just substitute the S for a Zed, for a Z I have to say. I'm Dutch and I misspell a lot of things in English. Well, honest mistake, right? But if I execute this, you will see it hangs.
If I go to my control over here and I will push the enter a few times, you, you see it hangs. And it hangs until all the possibilities in that regular expression are evaluated and tested against the string. Then it will free up the space for new requests and it will come in.
[00:31:28] If I add another zero to that 60,000 we will, we will be sitting here all day waiting until it will free up the memory. And remember, this is just one instance and it's very easy if your, um, one or two instances, uh, will just run this code once or twice or three times.
But if you're on the cloud, it's even worse. I mean, I just run this, this, this, this call and one of your, um, one of your instances is free. So another one probably pops up.
[00:31:58] If I continue that, it's not a denial of service, uh, but a denial of your Christmas bonus maybe. At least, it costs you a lot of money, and that's the whole point of it. Well, it's easy, as a Java developer, I love to make fun of Note. But at the Ja- in the Java space, we have a, we have a lot of problems over such.
[00:32:19] Let's go to that, that app I just showed you that had that exploit with, uh, Apache Struts. Let me sign up on this one. I will sign up. I will, uh, sign up with my email address.
I will use the password, uh, one, two, three, four, five, six, because I work at a security company is that is a very secure password. And I will sign up. No, no, no, don't, don't, don't do that.
[00:32:41] So again, we make to-do lists, because, well, my inspiration was not that good, good point- at, at the point of, of creating this. But what I can do is, I can create to-dos. Fair.
But say I want to create it and do it like this, E Pi. I will put it somewhere in 9070, because that sounds reasonable and I will give it a high priority. Let's create that one.
[00:33:02] What we see is that, that Pi character is transformed into a ASCII representation. In the code here, every title, every label goes into, uh, the native to ASCII function, which is part of- which is natively part of the JDK. Keep that one in mind, because it looks like I'm switching contexts, but we're going- coming back to this one.
[00:33:30] The second part, the second thing I can do on this application, I can upload files. And I can upload ZIP files. And a ZIP file will be using a library to unzip it and put the unzipped files in my public folder. But, and there is a but over here, if I am over here and, uh, I have a ZIP file prepared over here, and say my ZIP file looks like this, you see that the first part of my ZIP file is a text file, the GoDoc text.
But the second one includes a path reversal. So, ../..//app/.jdk/bin, which is the path of the JDK on my Heroku instance and I point it to the native to ASCII file. Basically what's inside of native to ASCII file is just an echo, "Ha-ha-ha-ha, gotcha."
[00:34:31] So, I just tried to override with this ZIP file. I'm trying to override native stuff in the JDK by just unzipping this file. Let's see if that works. So we're going into the thing. I'm looking for, uh, a demo... Yep. It's over here.
And then I have this ZIP file. I open it and I upload it. What we see here that's in my public folder, we see the GoDoc text. So, we know that the unzip operation worked.
[00:35:09] But there can be two ways. Or it deliberately... Well, ignored that other thing and it's a legal ZIP file, or it actually executed it. If I create another to-do, and I will say just like this I will give it a due date again in 9070, it doesn't matter. And I create it, I will see that I have overwritten the native to ASCII, uh, operation, native to ASCII file inside my Heroku JDK.
[00:35:47] The point here is that this library, like any other- like, like a lot of other libraries, didn't do well on inspecting the input. In Java, there is the problem that there is not a native, um, uh, uh, there is not a native library that can zip and unzip, but we do have the building blocks.
That means there are a lot of third party libraries that can do this for you, but if they don't look closely at the path and just use the path and connect the file link to it and executed it without tracking if it does anything like path reversal, we have this problem. And you see it even works on a Heroku cloud instance.
[00:36:27] So, let's get back to my slides, because that is all fun and stuff, that, that, that, that whole hacking thing, but there is a solution on it. But the problem is the solution is not that obvious. And it's not just, "Hey, just use my tool and you're done," because the problem is more.
The problem is three ways. First of all, we have to look at team culture. Second, we have to look at process, and third on tooling.
[00:36:59] Well, let's look, let's look into that, that whole team culture thing. Team culture is, we're working with a lot of different people with a lot of different opinions and a lot of different, uh, say, way of, of looking at the world. A developer looks at the world that- as they want to, they want to create stuff, right? Uh, but if you look at Ops people, they just want to maintain stuff.
They don't care how you build it or how fast or what's released. Managers probably look at what is the revenue, what is return on investment. And security people just want to make sure that data is not breached, or your data is not out there.
[00:37:35] But if we come together and use that, hold that SecOps mind thought- mind, uh, mind thought and, and put that in, and if we as developers make sure that we accept that, that, that security thing as a first class citizen, just as we do with scalability and, and, and maintainability, then the point at the end of your development cycle that, that security, uh, uh, person says, "No, you cannot go to production," would be slimmer, because you're already thinking of it from the beginning.
[00:38:09] If we're looking at, at process, we want you, um, we want to make sure that we do not add new process. We just want to make sure that we, we have things like fast feedback and, and, and, and, uh, uh, automating things, because if we add new process, like, "Okay, make sure that you do an actual pen test before you are, uh, releasing to production," you might do that two or three times, and then you're done and, and you go back to, to your old habits.
[00:38:37] And then there is tooling. The tooling needs to support both that process that you're already there, and that team culture. We need to make sure that, that fa- that fast feedback and automating is, is- are the key things, uh, in your, um, in your tooling, that your tooling can support that.
So, choosing the right tooling is... Well, it's very- uh, it's, it's a, it's a very, it's a very hard thing, because not every tool will fit your team culture or process.
[00:39:05] We want to say that we have to create a DevSecOps pipeline. And the DevSecOps pipeline works from the beginning, but on, on the total left side it's you coding on your local machine and then on the total right side, it is your production environment.
If you can scan your code during development with, say for instance, a, a, uh, common line plugin or, or a plugin in your IBE, why not do it over there before you, uh, uh, before you introduce new vulnerabilities or new dependencies in your repository.
[00:39:40] That would be great, right? Because if you're using a repository or if you're the- a dependency, uh, that is already vulnerable, you'd be want ver- I mean, a very early stage and you move onto a newer version and it might prevent you from, from, from doing a lot of rework.
That is your repository. You have to scan your repository on... Say for instance, if you connect your repository to a, to a tool that scans for you, because well, vulnerabilities will be disclosed everyday, and it can take some time before a vulnerability is actually found.
[00:40:13] So, by scanning it automatically and making sure that you, um, you get alerted by that saves you a lot of time and you will get act- actively pinged on that. And if you integrate it for is- with your Jira or whatever, uh, scrum or Kanban board you use, then you can even use it as a story to- and, and use prior authorization on that.
[00:40:34] It's prac- practically impossible for a lot of companies to go to zero vulnerabilities. But we need to make sure that as we have a point that we are comfortable with the number of the- number of, uh, vulnerabilities, we do not get worse, right?
So why not test every pull request? Every incoming pull request, make sure that there's not new- that you do not add new vulnerabilities in automatically before you, uh, merge them, that we have that CICD pipeline, or CI pipeline, depends on, uh, as in what you do.
[00:41:09] Why not scan it over there? Most people already do, I know. Like, if there is a security scanning, it's over there and we use it as a gatekeeper. We still need that gatekeeper, because if we go to production, we want to have as la- we want to make sure that the, that the number of, of vulnerabilities are as, are as low as possible.
And it's not only your application, but also for instance your docker containers. Or if you build a docker container, uh, is it your application part or is, is it your operating system part that you build your docker container- or, or in- that you build your application on top of?
[00:41:44] I mean, normally a docker container consists of a base image, plus your stuff. Maybe there's some problem in the base image, as well. Maybe you don't need such a big base image. Maybe we can test that during your continuous integration pipeline.
But having all- having done all that, then you go to production and maybe people, uh, think at that point we're done. And that is a big mistake, because that's basically what happened to Equifax.
[00:42:12] If you're in production you still need to scan. Why not, for instance, take a snapshot of your point- of that point in time and scan that on a daily basis? For instance, if you work with something that's in production and it might be there for weeks, months, even years, I know that a lot of developers have, have to maintain stuff that is there for years and do not dare to touch it.
But if that's the case, even if it's there for a few days, why not scan it on a regular basis and get alerted when there is a problem found or there is a new fix found or there is a patch found for, for a certain vulnerability so you can actively react on that?
[00:42:56] Well, that basically is the story, because we have to shift security left, and with left we mean as close... Let me go to the previous one, to the previous one. We have to shift security left, and as left as possible. So from your local machine and in every stage of your pipeline, you should be able to use the right tooling to help you out, uh, fixing security vulnerabilities like your dependencies, but also that part of your own code, like static analysis.
And there are a lot of tools that can help you, but you have to find out which is the right tool that will not be very intrusive into your process.
[00:43:34] And that is about my story for today, Mark. So, I will quit, uh, sharing my, uh, my screen. I'm very good at sharing screens.
[00:43:45] Mark: Let's see.
[00:43:45] Brian: Stop sharing.
[00:43:45] Mark: There we go.
[00:43:46] Brian: All right.
[00:43:48] Mark: Perfect. All right. That was fantastic. Um, the- I think for me, one of the things that is always great about a presentation like that is that when you go into those live, um, exploits and you're doing that live hacking, how simple it is.
Right? Like, yes, you're putting complicated stuff, but I mean, you know, you're sending one command and you're in, or a couple commands and you're in.
[00:44:08] It's not this... You know, people I think have this perception that hacking is super complicated all the time and it takes all of this crazy amount of computing resources and knowledge, and here you are firing off, you know, one terminal command, uh, and getting back, you know, the overwriting files or getting back information from the core system that you want, right?
[00:44:30] Brian: Yeah. And, and it's not that you have to find it out yourself, because there is... If, uh, if somebody already found out what the sh- what the, uh, header is you need to use and publish it on the internet, then every, every child can use it just by copy/pasting the commands like I did today.
And then it is very easy. And that's the whole point. It's not discovering it. Discovering it is hard, and that, that, that takes a lot of, takes a lot of engineering work and a lot of, uh, backwards engineering work to, to figure that out. But once that point is out, once that vulnerability is, is, is found and it's, and it's, and it's out in the open, then, then the, then the actual problem gets-
[00:45:12] Mark: Yeah, and that's what we see with our data from, like, uh, our sensors around the world, is inevitably, you know... And I mean, it was in your chart, right? Like, the March 24th, as soon as the vulnerability is publicly announced and people can look at the pat- what the patch is fixing, you also know what you can do to get around the un-patched version.
So, you see that massive spike in, "Well, now I can just copy/paste the text and spam anything on the internet, hoping it works."
[00:45:36] Brian: Exactly, exactly. And uh, uh, that's, that's what- that's why we want to try to not just put out the point, like, "Okay, there is a vulnerability," but in the most essential way, we want to also give you a fix. And then it's actionable for a developer, in that you can actually do something about it.
Because saying something is vulnerable is useful, but not- it, it... A, a developer or operations or DevOps teams, it doesn't matter how you look at it, want to solve things. They do, do not- don't like it.
[00:46:08] And the, the, the same, the same is for open source, uh, maintainers. If we point something out or somebody points something out, they want to fix it as soon as possible if they are capable of doing that with the manpower available, of course.
But nobody wants to be vulnerable, but you have to be aware of that. And if you're not aware of the fact that it's already a patch out, well, that is, that is your responsibility basically-
[00:46:32] Mark: Yeah.
[00:46:32] Brian: ... as maintainer.
[00:46:34] Mark: And that's a huge issue, right? Especially with a... I think one of the-
[00:46:37] Brian: Yeah.
[00:46:37] Mark: ... things that was really surprising, I think for the audience was when you had your slide showing, you know, here's your 244 lines of code, and that turns into 460,000 lines when you push it out to production.
Keeping track of those, I think that's, you know, more than fixing the problem, knowing you have a problem I think is a bigger problem.
[00:46:58] Brian: Sure, sure. And it's getting bigger and bigger if you see the, the, the old monoliths we need to, we need to maintain, uh, uh, well, in, in, in a number of companies.
And you see that over time the, the amount of dependencies, they grow. And actually, nobody knows anymore why this dependency was in.
[00:47:15] Mark: Mm-hmm [affirmative].
[00:47:16] Brian: And then you have to, have to argue, like, like, "Should we, should we keep this in? Shouldn't we keep this in? And for what reason should we?" And security is one reason, but also things like maintainability.
Do we actually know what happened? Uh, can we, can we fix any bugs in there? So, there are a lot of reasons to keep track of your dependencies.
[00:47:35] Mark: Yeah. And one of the things that you mentioned, um, in your presentation that kind of stuck out at me was, um, when you were saying, you know, "You've got all these dependencies and, you know, the code may not be being used." Um, do you know if you're using it? Um, you know, is that a problem?
You know, and there's... I mean, obviously complexity adds to, to operation challenges, as well as maintainability, but my security hat, you know, my spidey-sense kind of trickles out and goes like, "Even if you're not using it, having it there means that somebody else could."
[00:48:01] So you know, when you were doing those past- uh, path reversal attacks, just as easy, if you've left something in and somebody can inject code into your application, maybe they're taking advantage of all that unused code, like the reflection API, to do some nasty stuff to your app.
[00:48:17] Brian: Exactly, exactly. And you... By, by having this in, and even, even if you know you're not using it, you're giving, like, your fellow developers the chance to use it anyway, because-
[00:48:27] Mark: Mm-hmm [affirmative].
[00:48:27] Brian: ... it's like, "Okay, this package is already in." So, it's probably safe to use it. And I won't- will not... I do not want to introduce another package and it's already in there. So, why not use it?
[00:48:37] Mark: Yeah.
[00:48:37] Brian: So, then you just have to keep track of, what is my path and if it's actually, actually used or not? So that means yet another point of scanning.
So, I would suggest to, like, lower the amount of dependencies as much as possible. And if there is a vulnerability- vul- vulnerable direct or transmitted dependency, see if there's a fix.
[00:49:00] Mark: Yeah. That's good-
[00:49:01] Brian: That's an easy one.
[00:49:02] Mark: That's-
[00:49:02] Brian: If there's-
[00:49:02] Mark: That's-
[00:49:02] Brian: If there's no fi- if, if there's, if there's no fix, then, then we, we have, we have something more to do, so then, then we need to, like, shift things or so. But if there is a fix, what's, what's easier than just, just-
[00:49:15] Mark: Yeah.
[00:49:16] Brian: ... using the newer version?
[00:49:17] Mark: Yeah, which totally makes sense. And you got through that entire answer without trashing Node.js, which is impressive. Uh, 'cause, like, if you want-
[00:49:24] Brian: [laughs]
[00:49:25] Mark: Just, it's all dependencies all the time, right? I mean, they just pull in so, so much, and I think, um, you know, this leads into one of the questions I really wanted to ask you, because I know you're active in the developer community.
You're talking to people around the world. Is there a really... I mean, I know you're doing a lot of work to raise awareness, but do you find that people you're talking to are already aware of this issue? Do they think about dependencies more than just, "Oh, I have to keep this up-to-date to make sure that my code works"?
Are they thinking about any of this in a security context and the risk context?
[00:49:56] Brian: Um, correct answer would say not enough.
[00:49:59] Mark: Okay.
[00:50:00] Brian: Uh, it's starting to grow. It's starting to grow now, and people all right, uh, uh... Like, developers, uh, if they- if, if they see these kind of things, if they see, uh, uh, what's in the news and how, how important that is, they, they have a feeling for it. Because I think a developer don't want to write vulnerable code.
That's not- that's nobody's intention. So, so if they see these, these kind of examples, like, "Oh, yeah, we need to take care of that," but then there's that whole thing of, uh, team or company culture. Am I allowed to use this? Or, uh, how should we- how should I convince myself security, uh, uh, uh...
Or how should I convince my fellow players to use this? How should I convince my, my security manager that it's not all about policies, but also about, well, these kind of things. And I think it's- the awareness is starting to grow, but we still have a lot of work to do.
[00:50:53] Mark: Yeah. And, you know, so this kinda rolls into, into another question I wanted to ask you. Um, you know, uh, you- if you look at the OWASP Top 10, it really hasn't changed in a decade.
They've moved things around and they've renamed a couple things, um, but does this... What does that say to you, that we haven't really changed the top 10 vulnerabilities for web apps in a decade?
[00:51:15] Brian: Um, what that, what that says to me is that, I wouldn't say we're making the same mistakes over and over again, but the same type of mistakes will come in different forms. For instance, uh, validating your inputs.
[00:51:27] Mark: Mm-hmm [affirmative].
[00:51:28] Brian: We all probably know that we have to fill out the user input, but this one, the, the ZIP file I showed you, it's also an input. And how do we validate that?
So, it's the same thing in a different, in a, well, in, in, in a different setting. And again, I think the, the whole problem is, is that most companies choose to, to work by, "How much should I invest in something instead... And, and, and how much do I get in return?"
[00:51:57] And that's a difficult question with security, because what you- what is your return on investment in adding more resources into security? It's not getting hacked and how do you measure that. And that is a difficult, uh, dis- dis- uh, discussion, and I think there- there- there's another thing, like, uh, frameworks are changing constantly.
Um, and, well, also the- they are getting new developers in and, and are people aware enough that these things happen? And if there is enough pressure on, for instance, junior developers?
[00:52:33] Mark: Yep.
[00:52:34] Brian: Are they aware that they just created a, a, a SQL injec- a, a, a, a possible SQL injection problem? I don't know. And, but it's also like, do we as senior or as more, more experienced developers, do we- in code reviews, do we look at these security problems, or do we look at code styling or-
[00:52:54] Mark: [laughs]
[00:52:54] Brian: ... preferences or more- uh, uh, more... I will not answer that question, but I think the audience will, will, will know what I'm talking about, because code reviews, that- that's a whole, whole topic we can, we can get into and I can do a whole talk on that.
But I think it's a, it's a combination of all these, these, these things. Uh, simply because security is not, not the first thing we think of when we are developing.
[00:53:18] Mark: Yeah. And I, I think-
[00:53:19] Brian: And I hope to raise that awareness a bit.
[00:53:20] Mark: Yeah, and I think you've been doing a great job on that But a great example of what you're talking about there is, you know, path reversal, you know, when I'm thinking of it with my developer hat on, I'm going, "Okay, somebody's gonna mess up the path and I wanna make sure that no matter what, that, uh, application grabs the right file."
So if they just give me the file name, you know, I'm going to assume it's in the current working directory. If they give me a partial path, I wanna resolve it. Like, you can totally understand the development process of going, "No matter what you give me, I wanna make sure that I can find that file if it exists."
[00:53:50] The security flip side of that is, you've just now created most likely a path reversal, uh, vulnerability, because the developers try to make sure that that's a smooth experience so you don't sit there and go, "Well, I didn't add the full path," or, you know, "I didn't start with the right root," or something like that.
Um, and I think that, that kind of speaks to that culture, is that we don't build security in. You mentioned it in the intro of your talk, is that security's very siloed still. It's not part of the, uh, you know, sort of the fabric of developing. It's just starting to creep in.
[00:54:19] Brian: Yeah. And I also think that, for instance, if we need to review our code, or if I... Like, what happens typically, we have a sprint. And at the end of the sprint, I need to deliver.
So, a day before the deadline I ask somebody I worked on for two weeks like, "Can you please review my code?" Is that fair? That's the first question.
[00:54:41] And the second question is, who do, who do I ask to review my code? Is that a senior a peer, is that a junior developer? Is that... If, if- and if my stuff is something that... Well, leans a lot on security. That security is an important part of that part I'm working on.
Who- shouldn't I be asking somebody who is keen on security to review my code? So, specifically ask certain people, "Can you review my code?" Uh, we do that with things like, uh, uh, um, making sure that we have the program as fast as possible, so opt- like, things like optimization and performance, we ask sort of people to look at it.
But maybe we should do the same with security, because well, it's not, it's not only a hot topic, it's, it's, it's an actual problem.
[00:55:27] Mark: Yeah. Yeah, and then the, the, you know, fantastic idea, the challenge there then gets being to the expertise, right? Who's got the expertise to do that kind of review-
[00:55:35] Brian: Yeah.
[00:55:35] Mark: ... to spot those issues. Um, but, you know, we're- I think- I feel like development in, in general, as a practice, as a craft, is making progress. Um, but Brian, we're at a point now where we, in this- in the stream, that we always try to do this.
We do a little rapid fire. So, I'm gonna ask you a couple questions. Um, some of them a little controversial, but I want just a, a, a single-
[00:55:56] Brian: Mm-hmm [affirmative].
[00:55:56] Mark: ... short little yes or no kind of answer.
[00:55:58] Brian: [laughs]
[00:55:58] Mark: Um, and then we'll, then we'll circle back. Don't worry, I'll give you opportunity and we'll circle back.
[00:56:02] Brian: Okay.
[00:56:03] Mark: Um, nothing bad. Don't worry.
[00:56:04] Brian: Sure.
[00:56:04] Mark: It's not gotcha. It's all good. Um, though you may get a little booed at the next Dev, uh, meetup or community thing you're at, because this, this could be inflammatory.
Though I won't ask tabs versus spaces. Um, so, uh, building, uh, AWS Lambdas in functional Java, horrible shoehorn or a legitimate approach?
[00:56:25] Brian: Legitimate approach.
[00:56:26] Mark: Okay. Uh, Jira, corporate standard or baggage?
[00:56:33] Brian: Corporate standard.
[00:56:34] Mark: Okay. Uh, DevSecOps, good term, marketing push, or sadly needed reminder, uh, to people?
[00:56:44] Brian: The last one, sadly needed reminder to people.
[00:56:47] Mark: Okay. Uh, containers, stopping Devs from going serverless or a legitimate approach on their own?
[00:56:55] Brian: Legitimate app- app- approach on their own.
[00:56:57] Mark: Okay, okay. Um, so DevOps, uh, just giving developers more access to production or is it actually changing, uh, culture?
[00:57:09] Brian: [laughs] Oh, that's controversial. Um [laughs], neither, but that's not the answer you want to hear. Uh-
[00:57:18] Mark: Well, no, that's the, that's the end of the rapid fire, so let's dive into that one. What do you, what do you think?
[00:57:23] Brian: Well, I think DevOps, the, the point is there is, there is no one way to- if you ask two companies, "What is DevOps?"
[00:57:31] Mark: Mm-hmm [affirmative].
[00:57:31] Brian: They both say different things. So, we sh- we first have to agree on, what is DevOps? And what is the idea of DevOps? Because in some ways, it actually [laughs]... I've been in a company where DevOps was used to just cut down the number of people.
[00:57:46] Mark: Yep, yep.
[00:57:46] Brian: Yeah, we just take- we, we just, we just- we, we just take a few operation- uh, uh, a guy- uh, guys who are developers that are in the operations side and we take them out. You Devs have to do it, it, it, it yourself and we now do DevOps, and using it as an excuse.
So, it depends on how you, how you implement DevOps. And I think the, the, the, the meaning- uh, the, the, the term if you, um, actu- if you build it, you should own it. That is a good one. He- you eat your own dog food.
[00:58:13] Mark: Yep.
[00:58:13] Brian: I'm, I'm all for that, because if I deploy something on Friday, then it's not my, my, my Ops peer who should be called during the weekend. It's me.
[00:58:25] Mark: Yeah.
[00:58:26] Brian: Right?
[00:58:27] Mark: Yeah, fair.
[00:58:27] Brian: So, I believe- I, I, I believe that point. If- I should be aware of that everything I put in production, I, I took all effort possible that it's working. And if it's not working, I should fix it, because I built it and maybe I need some help from some peers, but I know what, what- I know what, what, what was in my mind when I, when I was building it. So, I believe that, uh, the general practice is a bit different.
[00:58:53] Mark: Yep.
[00:58:53] Brian: That... Well, people use- pe- pe- people, people use it in so- in sometimes in different, different ways. But I believe that you [00:59:00] build it, you own it, is the core thing of DevOps.
[00:59:03] Mark: I-
[00:59:03] Brian: As a team. Not as a person, but as a team.
[00:59:05] Mark: Yeah. Love it, love it. And uh, on that DevOps note actually, um, the Unicorn Project, which is the latest book from Gene Kim and the team, who started off with the Phoenix Project all that, actually dropped today.
Uh, it's queued up on my Kindle, ready to read. Um, we'll see where they're going, because they-
[00:59:20] Brian: Oh, wow.
[00:59:21] Mark: Yeah, because again, it's, it's... I think, you know, the cultural change is the hardest part of all this, right? Like, we can stand up new tools, you can build new pipelines, you can say, "Hey, we're using, um, you know, this new framework.
We're gonna put everything into Slack for ChatOps and all this stuff," but it's that mental mode of, you know, like you said, no more Friday, like, "Hey, I got it done. Have fun on the weekend, Brian, dealing with that." Um, that's [laughs] got a- that takes awhile to change, right?
[00:59:45] Brian: It is, it is. And it's- it, it also takes awhile to change that. Uh, maybe she- maybe we should even force people, because if... Now, if, if- the, the, the excuse, like, "No, that is not my responsibility," now it is your responsibility.
So, you have to take care of that. And if you're not willing to do that, maybe we should find a different, different engineer that can do it, because well, I want to have a good product in general and not-
[01:00:10] Mark: Yep.
[01:00:11] Brian: ... people blaming each other.
[01:00:12] Mark: Yeah, and I think people make different choices. So, we, we underwent a huge, uh, cultural shift in the last four years, um, on the Trend Micro side in the engineering team here in Ottawa, um, to go to this more of a model of, you know, you build it, you own it.
And it was interesting from the outside to see the different choices they started to make. Instead of trying to use the latest, shiniest, coolest thing they could find, they started asking those questions, like, "Wait a minute. If I deploy this thing and it's version, you know, .001 and it's just, you know, I found it on GitHub and it was good," um, and then they go, "Wait a minute, I'm responsible now.
I'm gonna get called, uh, you know, at 2:00 in the morning when this blows up," and they started making more- you know, tried to balance features for stability. Um, which I think is a- is that goal, right? It's that, that, "I built it-
[01:00:56] Brian: Yep.
[01:00:56] Mark: ... I own this." And it takes a time, but it's worth it. Um, so one, one big question here. Uh, and this is one I've some- I've asked every one of my guests so far. Um, what is your favorite cloud service right now?
So, one part of AWS or a service in GCP or in Azure or something like that. What are you having fun with and, you know, finding as your favorite from a, just from a personal experience side?
[01:01:20] Brian: To be, to be honest, I don't have a real favorite. I, I did some personal stuff with, uh, with AWS. We're working, you know, in, in, in- within Snyk, we're working with, uh, with Google Cloud.
[01:01:32] Mark: Mm-hmm [affirmative].
[01:01:32] Brian: Um, and it depends on what, it depends on what you want. If I want to automate things for, for my, for my, uh, for my Amazon Alexa, then you have to go to, to it- to their Cloud, of course, and to their, their Lambda functions.
But to be fr- to be honest, um, both are, are... Well, I've, I've had- I have had experience with these two and we are using Heroku, uh, uh, as, as well. Um, it depends.
[01:01:59] Mark: Sure.
[01:02:00] Brian: And that's, that's my all honest, honest... On, on this question, it depends on your, on your, on your goal and on your context basically.
[01:02:06] Mark: Yeah.
[01:02:06] Brian: But uh, well, I, I think both have their pros and cons.
[01:02:09] Mark: Fair enough. Fair enough. Um, so good. On that note, um, next week is AWS Reinvent, uh, 2019. Um, and I heard that you're coming, right?
[01:02:21] Brian: Yes, I'm there.
[01:02:22] Mark: And is this your first one?
[01:02:25] Brian: Yes, this is my first one.
[01:02:27] Mark: Okay. Uh, so what are you expecting? Uh, you know, I've been there since the start, so I have a, I have a totally different view on this. So, you know, you, you... Obviously AWS is, is huge.
Uh, Reinvent is crazy. As a newcomer to Reinvent, what are you, what are you expecting for the week, or what are you most looking forward to?
[01:02:45] Brian: Um, I expect a whole lot of people, a whole lot of craziness, and I try to keep my expectations low for the sake of, I have absolutely no clue what to expect, to be, to be fair.
Because with so many people around, I- the only thing I really hope is that it is some part still, uh, that the developer is the central point. And uh, because that's what, that's what I advocate. I love to work with developers. I'm- I- although my title says developer advocate, I'm still developing, like, 50% of my time.
[01:03:20] Mark: Mm-hmm [affirmative].
[01:03:20] Brian: Uh, so I can relate to these people, uh, the best, and I can talk to these- and I feel their problems the best. So, I hope that is still the majority of that conference, even though it's so big.
But um, I'm, I'm just there and, and trying to, to actually enjoy it. That's, that's what I want. That's, that's what I want to do there.
[01:03:39] Mark: Yeah, and I think, I think you'll be pleasantly surprised. Despite the fact that there's over 50,000 people going this year, there's still that feeling of, um, you know, everybody's there to build, everybody's there to learn about all this new stuff.
[01:03:52] Brian: Yeah.
[01:03:52] Mark: Um, it's not quite like the first one when we coding in the halls and building stuff as we go, um, but there's still that sort of, that vibe.
Um, you know, and I'd be remiss, uh, to, uh, fail to take this opportunity to say that you will actually be in the Trend Micro booth a little bit. Um, Booth 2820-
[01:04:08] Brian: Yep.
[01:04:08] Mark: ... on, on the floor, doing some demos of, uh, the cool tech that Snyk has got, um, as well as some of our partnership. Um, also, we are gonna be going, uh, live again from Reinvent, I think, uh, during the week as well.
We'll be posting that schedule on Trend Micro's social channels, um, to let you know, because the nice thing about, uh, Reinvent is it is very, very social. Lots of stuff going on Twitter, lots of stuff going on in person. Um, as a long time attendee, I will give you, the first-time attendee, a couple quick tips here. Um-
[01:04:37] Brian: All right.
[01:04:38] Mark: Bring sneakers. Good sneakers and good socks. You will be walking an insane amount. Um, one- so not last year, but the year before, I never left one of the buildings, the Venetian. I stayed in the Venetian the whole time.
Somehow I still managed to walk 12 to 15 kilometers a day. It is crazy. Um, so good sneakers. Uh, water bottle, which is part of the swag. When you register, you'll get a water bottle. Stay hydrated. Uh, protein bars or granola bars in your backpack is important, because the lineups are long for food.
[01:05:07] Um, and, uh, eyedrops and maybe lip chap, because your in a casino the whole time. Um, and, uh, it's still lots of smoke. Um, but yeah, sneakers are critical. Uh, if you make the mistake of not wearing good footwear, it is- it's gonna be a long week.
And battery- uh, phone charger or a battery is also very important, um, because you can imagine with 50,000 geeks around, outlets are not really that plentiful. So, uh, keeping the phone charged is kind of tricky. Um, but that, uh-
[01:05:34] Brian: Okay.
[01:05:34] Mark: I think it's gonna be a fantastic week. Um, you guys, uh, who are on the stream, you can follow us, um, @trendmicro, @snyksec. Uh, everybody's gonna be super active on Twitter, trying to share out. If you haven't already started watching the announcements for Reinvent, um, catch up on the AWS blog.
They've already announced, I think, uh, 100 and some different features or services, which is nuts. We've not even started the show yet.
[01:05:56] Uh, but Brian, this has been fantastic. I think the audience has learned a lot, I've learned a lot. Um, really, really appreciate you coming on today. Um, thank you very much for taking the time to join us.
[01:06:07] Brian: Oh, you're more than welcome. And uh, well, I'll see you next week [laughs].
[01:06:11] Mark: Yeah, we'll see you in Vegas. Thanks.
[01:06:13] Brian: All right.
