Amazon S3 was one of the first AWS services and it continues to be the cornerstone of the AWS Cloud today. This stream covers the basics of the service and how to ensure that your data is only accessed in the ways that you expect.
Reasonably Accurate 馃馃 Transcript
Good morning everybody. How are you doing today? I know it鈥檚 Monday always tricky to kind of get motivated to do something on a Monday you鈥檙e dealing with the massive amount of stuff in your inbox and you鈥檙e trying to get a handle on your week. And I am at the AWS Summit in Toronto on Thursday, which is why I moved things rather than doing Tuesday Thursday stream.
I moved it over to hear to do Monday Wednesday and made you something on Friday, but I appreciate you jumping in and as I鈥檝e already commented out to Richard in the comments these are archived. So if you can鈥檛 check catch the whole stream live here and you can watch it on LinkedIn.
I posted to my website and YouTube as well. And so I definitely more I contact the final course. I鈥檓 looking at the link I put in for my sight and I put in the devil not the actual link. My site is not at localhost. That鈥檚 a nice little hack Mark end.
CA that being said maybe. Crossing reference vulnerability there with the Local Host 2019 AWS reinvent the latest content there to let me actually just even remove that comment. So there you can get it down at the bottom three today. This is the oldest of eight of your services at the one that kind of kicked it off and S3 and and easy to wear the first ones that were launched a little over thirteen years ago now crazy absolutely crazy.
It is the one that most people think they understand yet where a lot of the problems come but there鈥檚 a huge amount of power here in Amazon S3, and we鈥檙e going to talk about that. So what we鈥檙e going to go to around at 10:30 today. So book 26 minutes 25 and change remaining and as always we are streaming live to LinkedIn if you have comments fire them up on the stream here.
I am monitoring it up. Stop flow down pretty recently. If you鈥檙e watching this on the replay at later on leave a comment down below. I鈥檓 on the YouTube video and I will read all those are applied all those and that helps me set up happy content for future. I鈥檓 streams.
The next train is going to be on Wednesday and we鈥檙e going to talk about two Services Amazon, Athena and Amazon quicksight, which actually linked to S3 really powerful really cool stuff. But today has three, so let鈥檚 flip over to on the screen here and There we go. Let me get that one browser window open always resize it to make it a little easier for y鈥檃ll to see it.
Okay, so I鈥檓 going to move myself over in the Stream so you don鈥檛 have to worry about me get my timer over here when I make this a little bit smaller. I think cool. Alright, there we go. Amazon S3 have a 3鈥檚 has very simply for a simple storage service right? There were just calls it at 3.
Everybody knows what it is. It is a storage system. Basically, it鈥檚 a big directory up in the sky very very simply put very easy to understand in this case. Actually if we flip over over to my website just for a second because I know this is here if we鈥檙e looking in the shared-responsibility model which we did a whole stream on last week at S3 fits very firmly in this SAS or abstract level server.
Which means we are responsible for the date of that. We put into it and configuring the service and that鈥檚 what we鈥檙e going to touch on today. And you may think this is really basic stuff and a lot of it is but that鈥檚 the whole point of this stream obvious build up to renew its cover some of the basics covered some of the complicated stuff and we鈥檙e going to open your eyes I think to a few things in S3 and maybe you鈥檒l understand some of the challenges around it and because he has unfortunately gotten famous private number of data breaches that happen to people or storing data in S3.
That鈥檚 not S3 salt. That鈥檚 just people failing to understand this model and failing to understand how S3 actually works Miss configuration is the number one Cloud threat number one Cloud prep by far and because we have an insane amount of power, right if we look at all of the services available to us.
So this is the console you can see all of these Services now, how are you as an individual Builder supposed to understand all of the ins-and-outs of all these Price is Right. It鈥檚 really really difficult because there鈥檚 so much power and our fingertips part of the reason why we鈥檙e doing this stream.
We鈥檙e going to talk about that self. Let鈥檚 Dive Right into this. I am going to show you are a high-level a picture real quick of there we go. So there are three things we need to worry about we鈥檙e going to deal with two right off the bat and but I wanted to introduce this concept as we move forward and get if you have questions fire them off here on the LinkedIn stream and we will address them as we go so you can see on the screen.
We鈥檝e got a slide out very very simple and Amazon us we were going to worry about three objects today. The first is literally an object in common parlance. This is a file and the name of that file is called a key. Okay also have a bucket the bucket is where you put everything in Salinas through the first thing you do is you create a bucket and then you just start piling objects in there.
If you want you can use a bucket policy which defines who can shove thing. Is in that bucket who can take things out of that bucket, but right now we鈥檙e going to start with just the basics with objects and we have a bucket. So let鈥檚 flip back over to our browser and we are going to actually create a new bucket now.
We did this actually really really quick in our first stream on Lando. We鈥檙e going to redo it now. So if I click a new bucket now, we鈥檙e going to get the wizard to help us through this. And the first thing we need to do is give it a bucket name.
This is the name of the bucket is the way we鈥檙e going to be able to out refer to it in case or common parlance that we need to pick the region now, you鈥檒l notice in almost all the regions are available at Hong Kong is disabled because they鈥檙e not in China saying with the trainers.
I haven鈥檛 turned it on yet, but I can pick in this case. I鈥檓 going to pick North East Us East 1 North Virginia Zone. There鈥檚 a concept of regions right u.s. East one u.s. West one was to Canada. These are regions in the Region鈥檚 there are availability zones, and we do not need to pick an availability zone for S3.
Because S3 spends all of the availability Zone in a given region. That鈥檚 a really nice thing for durability. We鈥檙e going to talk about that in a few seconds. So here I can copy settings from an existing bucket not going to do that cuz I want to walk you guys through how these settings are set but we first name road to reinvent the name of our Market.
We鈥檙e going to leave that in u.s. East we want to keep all versions of all the objects in the same bucket know you can so S3 has a burgeoning system, which means if you put sample. Txt in and then you updated and upload sample. Txt. Again, you can save successive versions, which is really handy to roll back.
So think like an iCloud Drive Dropbox box.net they all have versioning built-in and extremely handy and depending on the kind of data that you鈥檙e dealing with for us. We鈥檙e not going to turn it on so you don鈥檛 want it and save login. Request so you can log all the requests your bucket this logs all the time, but gets in the boats and everything everyday activity on your bucket that can be handy.
Especially if you鈥檙e running a website out of your bucket in our case. We don鈥檛 want to tags common feature in AWS. Maybe we鈥檒l tackle that in a future stream but tags basically let you put on identifiers keys and values to make sure that you make sense of the resources in your own so far in a big company in here in a shared account you could say a key is team and then the value is, you know, Mark, right so we know Mark鈥檚 team has this bucket in our case.
We鈥檙e not going to worry about it. And then you have a finer grain object level logging. So the S3 a server access logging of top is served chunky through the course down here. You鈥檝e got some really granular logging and fills up your logs really really quick if you have an active bucket, but you got that option and then you can also check box for automatically encrypting your data when it鈥檚 stored in S3 sees me now this year.
Is a server-side key. Which means you do not manage the key and that鈥檚 still solid it still adds an additional layer of protection if you want and it鈥檚 really as easy as just checking that box. Now we click next get this to our set our permissions. Now, we鈥檙e going to spend quite a bit of time on permissions here if you guys have questions on permissions and let me know when the chat here in LinkedIn or after the fact in YouTube chat down below but very very simple.
This screen is one of my all-time favorites and the reason being is that at S3 is commonly misconfigured and people end up leaking a bunch of data out of it you Chris Vickery up guard fantastic creatures or Chris is have he has made his lunch on S3 bucket and Miss configurations for the last few years.
In fact, I鈥檝e actually written a few articles on over on the a cloud Guru blog around S3 security. I鈥檓 simply because people seem to have a challenge with this right and very very simply S3 bucket start lock down your explicit choices open them up. Okay. So I鈥檝e written this this article II which actual put the link to in the comments right now S3 security post.
And then there鈥檚 actually another S3 security post that I written. That鈥檚 a little bit more detailed. Second of three security post typing I didn鈥檛 hear for you all in the LinkedIn comments as you鈥檝e got those links if you want to see him this one I think summarizes it nicely just put everything in a bucket and Omega public.
How do you not make them public? Let鈥檚 flip back to the console. You can see here. This is a relatively new screen has been in place for about 18 months and one easy easy. Click block all public access. Block all public access. Here鈥檚 my number one tip my top tip.
I guess I鈥檒l give it lots of tattoos my top tip when it comes to S3 buckets. If you鈥檙e going to have public S3 buckets try to keep them in a completely separate account. You do not pay for a divorce account to pay for AWS account activity. So setting up a whole separate account for public buckets is not going to cost you anything.
Now, you can create complicated permissions that allow you to write from one account into the other but not read which is even better. But you can have that sort of Separation that that firewall of going till you know, here鈥檚 the line everything on this side of the line is private everything out in this account is public and that鈥檚 a really easy way to make sure you kind of don鈥檛 shoot yourself in the foot.
Here鈥檚 another way is when you鈥檙e setting Your bucket block all public access until this is unchecked through a specific API call or coming into the console to remove this nothing in this bucket can be public. That鈥檚 great. That鈥檚 what we want. We鈥檙e going to override everything else with an explicit block now underneath here.
There鈥檚 a ton of options right soon as they block public access to the buckets for anyting is through new list. So keep existing, but don鈥檛 add any new don鈥檛 let any any control us do it and block Public Access through new public bug has policy nine times out of ten.
You just want to leave this check to block all public access. . We鈥檙e going to do that cuz we鈥檙e not making a website. We鈥檙e going to block Public Access. Where do you cook next now? We can go even finer and set review now. We鈥檙e going to see okay road to reinvent you at least we鈥檙e disabled versioning at logging disabled tags.
We鈥檝e got server-side encryption on and we鈥檙e blocking all public access. Click create the bucket you see real quick. We鈥檝e created a bucket. Okay. So now we have our bucket in place. If we click there. We are going to come into the bucket you鈥檒l see we have nothing in our bucket we have yet to put an object into the bucket we can do that by clicking on the upload button is going to go to the upload dialogue this case.
We鈥檙e going to add a file we鈥檙e going to go to a desktop and we are going to add this file in which is just a simple image file to gives us an ISO summary. We can add a bunch of files here to do a group upload. You can鈥檛 go bigger than 160 gig in one鈥檚 upload through the web browser.
You have to use another two. Go to do that and not many people are going to hit that limit. But you know a lot of time you鈥檙e not using the web interface to access S3. Anyway, so we click next and here we go to a setting permissions allow permissions again, we鈥檝e already said our bucket permission saying don鈥檛 let anything be public now, we can have individual permissions within our object so we can say in this case.
I鈥檓 logged in with the AWS account for a account call data PS4 the demo account and it has read permissions as well as read write permissions on the specific object. So can read all objects that I can read and write those I could add more users here. I could add another account.
You could say Okay, Joe鈥檚 allowed to access this Francine鈥檚 allowed to access this you could create quite a nuanced set of permission for this specific file. Now the way that works as we already have two layers in place. We鈥檝e got a bucket policy up top is a bigger filter and then the object policy.
You鈥檝e got a pass through The bucket policy to ever hit the object. So if you鈥檙e blocked at the bucket and allowed at the object that鈥檚 going to conflict denies always be allows. So you want to make sure that they鈥檝e got that sort of filtering through text and we pick our tier now for Tears.
You are going to allow so just a real clarify cuz that was going to my head when you have the bucket policy and the object policy in our bucket policy. We鈥檝e said no Public Access. So that鈥檚 going to deny any on offend ocated user. So anybody who hasn鈥檛 logged in won鈥檛 be able to access it and then down to the object level where we say Francine and Joe and Mark can access this object.
So an authenticated user what they Fred gets through that bucket policy because he鈥檚 not a public user. He鈥檚 a he鈥檚 authenticated we know who he is and then he goes the object policy and doesn鈥檛 have permission. So he鈥檚 not allowed now if we said nobody but Fred is allowed to come through this bucket policy explicitly denying everybody that would stop everybody else.
Right? So there鈥檚 a little bit of nuance there, but you get the point bucket policy first and then the object policy, but now we need to pick our storage class know this has a direct impact on your bill. So standard tier is the most expensive Vegetarian maybe a little bit more expensive now.
I鈥檓 to start but the idea here is basically how often are you using this stuff? How much durability do you need? So for standard that鈥檚 normal standard S3. It goes across 3 availability zones. There is a huge amount of redundancy. I鈥檓 just going to flip over the actual detail here.
So if we look at general purpose as 3 standard 11:9 of durability, which means it鈥檚 highly unlikely that your object will be changed or anyting flipped as far as bits show resilience against the entire AZ going down because you鈥檙e across multiple availability zones availability. That means your ability to get back 90 by 4/9 which is pretty solid.
Right and that is the standard way that you get Storage in at 3 and you can go serve reduce your access from there. So one鈥檚 own I say is going to give you can only into one鈥檚 own instead of 3, you鈥檙e going to keep it for at least 30 days at Glacier goes to Cold Storage in Glacier Glacier deep archive is even colder storage intelligent tiering is a machine is a new level of storage.
It is an intelligent model. We鈥檙e basically says I鈥檓 going to start in standard and then if you don鈥檛 use this object, like if you don鈥檛 nobody is accessing and I鈥檓 going to put it down to less frequent access and if nobody鈥檚 using it then I鈥檓 going to push the glacier if still nobody鈥檚 using and we can push it too deep archive because the as you go down this chart it鈥檚 less money to store.
So Glacier and deep deep archive Glacier are really inexpensive the store they cost a ton of money to pull out though. So the ideas for long-term stuff that you have to legally keep but you probably don鈥檛 want to access daily Glacier and deep archive are great for stuff. You don鈥檛 really care about the So naive makes a lot of sense and most of time you鈥檙e going to just sit and standard having the costs are not at that big.
So it鈥檚 you know, it鈥檚 a fraction of pennies and things like that. So we鈥檙e going to keep in standard. We鈥檙e going to upload our next and then we go to our review that鈥檚 good. And we upload it and all of a sudden we have a file in S3. It鈥檚 called an object.
You鈥檒l notice that it is sitting right now in just a root of that bucket really interesting thing is that there is no directory structure in S3, even though there鈥檚 literally a button that calls create folder. The reason being is that this object has a name that name is called the key.
So if you look at this the key for our file right now is the filename right ACG. It鈥檚 an ad for the course. I did on mastering the well architected framework. At least this is screenshot and you鈥檒l see we have a URL if I hit this URL right now.
Access denied why is the access denied because I鈥檓 not affect ocated on this URL on my Fanta created in the console. I鈥檓 not in the URL was blocked all public access. But if we hadn鈥檛 blocked all public access, you can actually give out your else for your S3 objects.
So this case the key here if we come back to our key Point pun intended couch fee at there are no file folders, but if I create a file folder, let鈥檚 call this test and then save it and now I鈥檓 going to upload the exact same file. I鈥檓 just going to Breeze through this because we鈥檝e already talked about it.
I鈥檓 going through the same dialogue. I鈥檓 keeping all the default settings blah blah blah blah blah and this is now uploaded now if I click on this file, you鈥檙e going to see the keys changed the filename still the same. I literally uploaded the same father didn鈥檛 change anything, but the key is now test / file name and that鈥檚 because their file folders are alive.
I file folders are a lioness 3 and they are away that clients have de facto. Visualize the de-facto way to visualize files to make it easier to understand. So if we come back to the root of our bucket, you鈥檙e going to see we still have a folder and then we have the file in the route and we have a file in the folder the what the reason is to set up in the keys is that anytime the key start with the same prefix and a slash it just makes more sense for us to be able to manage it but it interesting little bit of a history trivia.
There are no file folders. It鈥檚 all a giant giant lie. So if we click back to our key, you鈥檒l see that we鈥檝e got these these the high-level information. So when it was modified the storage class is currently sitting in which is standard its eyes its name. We can also adjust some of the properties here so we can change the storage class we can add metadata to it.
So if you want to do key value pears and add additional information of these files that鈥檚 handy for automation. We can also lock this object so that nobody can delete it and we can add more tags to it as well course we can go to permission. This will allow us to add complicated permissions.
You can see here right now with these are the access object for owner would you me and we can add Public Access even though it says we can鈥檛 right now because we鈥檝e been blocked which is great. So anything we add under here doesn鈥檛 apply because we blocked it but we can add other accounts we can add other users or we can pull our records out of S3, but that鈥檚 really simply put how S3 works very easy object in an object which has a key which is the name into a bucket.
I鈥檓 going to flip it back to the slide tournament cuz I want to talk in depth about permissions so we know we have an object. We know we have a bucket we know we have a bucket policy. We鈥檝e seen a little bit about the bucket policy. I meant to get if you guys have questions just fire him off in the LinkedIn comments.
I am honoring those actively so I and I expect some questions on this park is it gets kind of weird. So we saw a bit of the bucket policy already, which is When you have a bucket new crate, you can say no public don鈥檛 let anything in this be public and while we saw that in a nice little dialogue what was actually happening behind the scenes was the console created a bucket policy for us in that bucket policy said no public you can edit that bucket policy and say public with the following exceptions.
You could remove the public Walker altogether. You can Define how this bucket is used at the idea of a bucket policy. You鈥檙e taking a set of permissions and you鈥檙e applying it to the bucket itself and that can say check to make sure I know who鈥檚 accessing me that can see everybody鈥檚 allowed to look at this image file, but not this image file.
There鈥檚 a rather complex set of things that you can do makes sense. Now, this was the original way to do policies. I鈥檓 in permissions in S3 because the newer way didn鈥檛 exist so bucket policies apply to the bucket and then anything that goes in the bucket can have additional object policies on top of it.
The other way primary way to apply for missions in S3 is using AWS IAM the identity and access management tool. So that allows you to assign permissions to users or to roll. So I am a user of markers user a roll goes to an object. So do a AC to server morning stance is the parlance in a day or two container somewhere over to a Lambda function rolls get assigned to things.
Okay. So you can think of the bucket policy is really a role assigned to that bucket, but it鈥檚 managed on its own and that鈥檚 the problem. That鈥檚 where people get messed up really a bucket policy is just a permanent. I am roll assigned to that bucket, but it鈥檚 managed in the S3 area of AWS all your other permissions are managed.
Yes. I am. This is where you can say, you know what marks allowed to access this bucket, but only objects that fall under the test folder rights do anything with the prefix test slash marks allowed to access. That you can do that fine grain stuff assigned to the person or the role in I am and that鈥檚 really where you should be a signing the vast majority of your permissions stick to one place.
Don鈥檛 do both both ends up being confusing do what? It鈥檚 Monday. I want to melt your brain a bit because why not brains are for melting. That鈥檚 what we鈥檙e going to go with for Monday morning. There are two more additional ways to actually apply permissions in AWS. I鈥檓 coming back to the how to secure an S3 bucket article a while ago.
So we talked about I am policies we talked about bucket policies. There鈥檚 also a thing called an access control list that gets applied and query string authentication or URL based access so you can give somebody a URL and say you can post to this URL or you can get from this URL and it actually goes directly into S3.
There are scenarios where you want to use that however, I strongly as strongly strongly so so strongly recommend that you keep your access all in I am Just do it easier simple. That鈥檚 what I am. Therefore. It鈥檚 there for keeping all of your permissions in one place and give you a lot of great tools as a lot of great ways to audit check your permissions try to keep them in.
I am as much as possible. If you have to go to the bucket because remember the bucket think of the bucket policy as a role in I am assigned to the bucket permanently. That鈥檚 fine. That works, too. Try to avoid Access Control lists try to avoid query string in url base stuff because you鈥檙e just going to get lost and if you have to create a new account or new buckets and try not to mix these various ways on the same bucket because there is no simple way to view all of the permissions assigned to a bucket and everything in that bucket.
Okay, that鈥檚 why you know, we talked with the start y Mis configurations of the number one problem. There鈥檚 too many options. Now those options trust me. There are times when one of these will save your butt. And when you鈥檙e in that scenario, you really want that option, but do not mix and match she just because you can doesn鈥檛 mean you should a good rule for life in general, but try to separate these by bucket.
If not by account. And again that public thing supercritical to go buy account just to save your you-know-what so that in a nutshell is S3. Now. I鈥檝e got two minutes left on one of the cover something really really simply so you noticed my website here. This is were looking at Mark end.
CA and this is the page. I鈥檓 tracking all the stuff we鈥檙e doing around reinvents. You see here. Hey, you know next to him is right now that鈥檚 what we鈥檙e doing today. And then on Wednesday, we鈥檙e doing 10 a.m. We鈥檙e going to talk about Athena and quicksight and I posted all the stuff we talked about already right now.
This is actually being served out of an S3 bucket. S3 is a full web server super cool. What we鈥檙e going to do is flip back the S3 here and we鈥檙e going to create road to reinvent. Now we鈥檙e going to click next we鈥檙e going to leave all of the bucket options as is we are then going to set permissions.
We鈥檙e going to remove blocking Public Access and we鈥檙e going to create this bucket. Now the reason being in the reason why we called this. Com is because you can actually assign your Bucket to be lineup of the domain name. Okay that were clicked on the properties of the bucket.
We鈥檙e going to go to static web hosting and we are going to say use this bucket to host a website indexed. Lhd males are default are. HTML. There鈥檚 a bunch of options here for for setting up your website. We鈥檙e simply going to cook save then we鈥檙e going to go back to the week or this bucket and we鈥檙e going to upload a file and upload that exact same file that we offloaded time and time again, when we get to the permissions, what we are going to do is leave that will do in the 2nd standard upload them.
Just looking at the time here. Yeah, it鈥檚 been uploaded. So if we go now and check you鈥檒l see the URLs different. That鈥檚 fine. But if we click on the URL. We still have access denied just because the bucket is set to host a website doesn鈥檛 mean that鈥檚 going to work.
So what we need to actually do is go to permissions. Everybody can read this object big morning. Hey, this is public again. Don鈥檛 do this in the same account create a second account to do it. We鈥檝e a dummy demo account that we鈥檙e looking at today, but now that we鈥檝e assigned this to be a public if we just refresh this we should see that everyone has the read access again.
We come back to our URL. There we go. Now this your LS3 on amazon.com name of our buckets the key name gives us access to this picture. Okay, you can set up an entire website there. That鈥檚 in fact what I鈥檝e done with my personal website and marketing. CA that鈥檚 in its own separate account and being hosted so that I don鈥檛 make the mistake of doing something like my personal photo archive being exposed.
But S3 has the ability to serve websites directly course, you want to make them fast, you can put them in like a cloud front in front of them and put that in a nutshell. We are at our time block here and let me switch back to me that it was S3 your buckets.
You object you put an object in a bucket. You have multiple ways of setting permissions stick to one of them. If you can stick to either I am permissions for everything or bucket policy for everything. If you need to have a private bucket or a public bucket put in a different account Accounts cost you nothing usage cost you so you鈥檙e not going to pay any more to put it in a and it鈥檚 separate account.
I鈥檓 going to pay the exact same thing. So that鈥檚 a good strategy operationally. So you don鈥檛 have to shoot yourself in the foot by putting something sensitive into a bucket of configured for public because we鈥檝e seen that time and time and time again over the last few years where people have made a simple mistake taking a bucket that starts lockdown and unfortunately adding additional permissions because of the different ways to add permissions, they get tripped up and expose customer records are sensitive data and that is a bad the very bad thing.
And this has been a quick introduction into S3. We鈥檙e going to build on Wednesday by looking at Athena and looking at Kwik site. These are tools that help you access the data in S3 or visualize the data in S3, really cool kind of hidden services to my favorite actually are off.
Overlooked and shouldn鈥檛 be so let me know what you think in the comments below. Thanks for joining the stream the next one as you saw is on Wednesday at 10 a.m. Eastern again. Normally don鈥檛 do Monday money is going too hard and but because I鈥檓 at the AWS Summit in Toronto on Thursday and made sense to do this here.
So I hit me up. Let me know. Thanks for joining the stream really appreciate it. Have a great day, and we will talk to you soon. Good morning, everybody. How you doing today? I know it鈥檚 Monday always tricky to kind of get motivated to do something on a Monday you鈥檙e dealing with the massive amount of stuff in your inbox and you鈥檙e trying to get a handle on your week.
And I am at the AWS Summit in Toronto on Thursday, which is why I moved things rather than doing Tuesday Thursday stream. I moved it over to hear to do Monday Wednesday and made you something on Friday, but I appreciate you jumping in and as I鈥檝e already commented out to Richard in the comments these are archived.
So if you can鈥檛 check catch the whole stream live here and you can watch it on LinkedIn. I posted to my website and YouTube as well. And so I definitely more I contact the final course. I鈥檓 looking at the link I put in for my sight and I put in the devil not the actual link.
My site is not at localhost. That鈥檚 a nice little hack Mark end. CA that being said maybe. Crossing reference vulnerability there with the Local Host 2019 AWS reinvent the latest content there to let me actually just even remove that comment. So there you can get it down at the bottom three today.
This is the oldest of eight of your services at the one that kind of kicked it off and S3 and and easy to wear the first ones that were launched a little over thirteen years ago now crazy absolutely crazy. It is the one that most people think they understand yet where a lot of the problems come but there鈥檚 a huge amount of power here in Amazon S3, and we鈥檙e going to talk about that.
So what we鈥檙e going to go to around at 10:30 today. So book 26 minutes 25 and change remaining and as always we are streaming live to LinkedIn if you have comments fire them up on the stream here. I am monitoring it up. Stop flow down pretty recently. If you鈥檙e watching this on the replay at later on leave a comment down below.
I鈥檓 on the YouTube video and I will read all those are applied all those and that helps me set up happy content for future. I鈥檓 streams. The next train is going to be on Wednesday and we鈥檙e going to talk about two Services Amazon, Athena and Amazon quicksight, which actually linked to S3 really powerful really cool stuff.
But today has three, so let鈥檚 flip over to on the screen here and There we go. Let me get that one browser window open always resize it to make it a little easier for y鈥檃ll to see it. Okay, so I鈥檓 going to move myself over in the Stream so you don鈥檛 have to worry about me get my timer over here when I make this a little bit smaller.
I think cool. Alright, there we go. Amazon S3 have a 3鈥檚 has very simply for a simple storage service right? There were just calls it at 3. Everybody knows what it is. It is a storage system. Basically, it鈥檚 a big directory up in the sky very very simply put very easy to understand in this case.
Actually if we flip over over to my website just for a second because I know this is here if we鈥檙e looking in the shared-responsibility model which we did a whole stream on last week at S3 fits very firmly in this SAS or abstract level server. Which means we are responsible for the date of that.
We put into it and configuring the service and that鈥檚 what we鈥檙e going to touch on today. And you may think this is really basic stuff and a lot of it is but that鈥檚 the whole point of this stream obvious build up to renew its cover some of the basics covered some of the complicated stuff and we鈥檙e going to open your eyes I think to a few things in S3 and maybe you鈥檒l understand some of the challenges around it and because he has unfortunately gotten famous private number of data breaches that happen to people or storing data in S3.
That鈥檚 not S3 salt. That鈥檚 just people failing to understand this model and failing to understand how S3 actually works Miss configuration is the number one Cloud threat number one Cloud prep by far and because we have an insane amount of power, right if we look at all of the services available to us.
So this is the console you can see all of these Services now, how are you as an individual Builder supposed to understand all of the ins-and-outs of all these Price is Right. It鈥檚 really really difficult because there鈥檚 so much power and our fingertips part of the reason why we鈥檙e doing this stream.
We鈥檙e going to talk about that self. Let鈥檚 Dive Right into this. I am going to show you are a high-level a picture real quick of there we go. So there are three things we need to worry about we鈥檙e going to deal with two right off the bat and but I wanted to introduce this concept as we move forward and get if you have questions fire them off here on the LinkedIn stream and we will address them as we go so you can see on the screen.
We鈥檝e got a slide out very very simple and Amazon us we were going to worry about three objects today. The first is literally an object in common parlance. This is a file and the name of that file is called a key. Okay also have a bucket the bucket is where you put everything in Salinas through the first thing you do is you create a bucket and then you just start piling objects in there.
If you want you can use a bucket policy which defines who can shove thing. Is in that bucket who can take things out of that bucket, but right now we鈥檙e going to start with just the basics with objects and we have a bucket. So let鈥檚 flip back over to our browser and we are going to actually create a new bucket now.
We did this actually really really quick in our first stream on Lando. We鈥檙e going to redo it now. So if I click a new bucket now, we鈥檙e going to get the wizard to help us through this. And the first thing we need to do is give it a bucket name.
This is the name of the bucket is the way we鈥檙e going to be able to out refer to it in case or common parlance that we need to pick the region now, you鈥檒l notice in almost all the regions are available at Hong Kong is disabled because they鈥檙e not in China saying with the trainers.
I haven鈥檛 turned it on yet, but I can pick in this case. I鈥檓 going to pick North East Us East 1 North Virginia Zone. There鈥檚 a concept of regions right u.s. East one u.s. West one was to Canada. These are regions in the Region鈥檚 there are availability zones, and we do not need to pick an availability zone for S3.
Because S3 spends all of the availability Zone in a given region. That鈥檚 a really nice thing for durability. We鈥檙e going to talk about that in a few seconds. So here I can copy settings from an existing bucket not going to do that cuz I want to walk you guys through how these settings are set but we first name road to reinvent the name of our Market.
We鈥檙e going to leave that in u.s. East we want to keep all versions of all the objects in the same bucket know you can so S3 has a burgeoning system, which means if you put sample. Txt in and then you updated and upload sample. Txt. Again, you can save successive versions, which is really handy to roll back.
So think like an iCloud Drive Dropbox box.net they all have versioning built-in and extremely handy and depending on the kind of data that you鈥檙e dealing with for us. We鈥檙e not going to turn it on so you don鈥檛 want it and save login. Request so you can log all the requests your bucket this logs all the time, but gets in the boats and everything everyday activity on your bucket that can be handy.
Especially if you鈥檙e running a website out of your bucket in our case. We don鈥檛 want to tags common feature in AWS. Maybe we鈥檒l tackle that in a future stream but tags basically let you put on identifiers keys and values to make sure that you make sense of the resources in your own so far in a big company in here in a shared account you could say a key is team and then the value is, you know, Mark, right so we know Mark鈥檚 team has this bucket in our case.
We鈥檙e not going to worry about it. And then you have a finer grain object level logging. So the S3 a server access logging of top is served chunky through the course down here. You鈥檝e got some really granular logging and fills up your logs really really quick if you have an active bucket, but you got that option and then you can also check box for automatically encrypting your data when it鈥檚 stored in S3 sees me now this year.
Is a server-side key. Which means you do not manage the key and that鈥檚 still solid it still adds an additional layer of protection if you want and it鈥檚 really as easy as just checking that box. Now we click next get this to our set our permissions. Now, we鈥檙e going to spend quite a bit of time on permissions here if you guys have questions on permissions and let me know when the chat here in LinkedIn or after the fact in YouTube chat down below but very very simple.
This screen is one of my all-time favorites and the reason being is that at S3 is commonly misconfigured and people end up leaking a bunch of data out of it you Chris Vickery up guard fantastic creatures or Chris is have he has made his lunch on S3 bucket and Miss configurations for the last few years.
In fact, I鈥檝e actually written a few articles on over on the a cloud Guru blog around S3 security. I鈥檓 simply because people seem to have a challenge with this right and very very simply S3 bucket start lock down your explicit choices open them up. Okay. So I鈥檝e written this this article II which actual put the link to in the comments right now S3 security post.
And then there鈥檚 actually another S3 security post that I written. That鈥檚 a little bit more detailed. Second of three security post typing I didn鈥檛 hear for you all in the LinkedIn comments as you鈥檝e got those links if you want to see him this one I think summarizes it nicely just put everything in a bucket and Omega public.
How do you not make them public? Let鈥檚 flip back to the console. You can see here. This is a relatively new screen has been in place for about 18 months and one easy easy. Click block all public access. Block all public access. Here鈥檚 my number one tip my top tip.
I guess I鈥檒l give it lots of tattoos my top tip when it comes to S3 buckets. If you鈥檙e going to have public S3 buckets try to keep them in a completely separate account. You do not pay for a divorce account to pay for AWS account activity. So setting up a whole separate account for public buckets is not going to cost you anything.
Now, you can create complicated permissions that allow you to write from one account into the other but not read which is even better. But you can have that sort of Separation that that firewall of going till you know, here鈥檚 the line everything on this side of the line is private everything out in this account is public and that鈥檚 a really easy way to make sure you kind of don鈥檛 shoot yourself in the foot.
Here鈥檚 another way is when you鈥檙e setting Your bucket block all public access until this is unchecked through a specific API call or coming into the console to remove this nothing in this bucket can be public. That鈥檚 great. That鈥檚 what we want. We鈥檙e going to override everything else with an explicit block now underneath here.
There鈥檚 a ton of options right soon as they block public access to the buckets for anyting is through new list. So keep existing, but don鈥檛 add any new don鈥檛 let any any control us do it and block Public Access through new public bug has policy nine times out of ten.
You just want to leave this check to block all public access. . We鈥檙e going to do that cuz we鈥檙e not making a website. We鈥檙e going to block Public Access. Where do you cook next now? We can go even finer and set review now. We鈥檙e going to see okay road to reinvent you at least we鈥檙e disabled versioning at logging disabled tags.
We鈥檝e got server-side encryption on and we鈥檙e blocking all public access. Click create the bucket you see real quick. We鈥檝e created a bucket. Okay. So now we have our bucket in place. If we click there. We are going to come into the bucket you鈥檒l see we have nothing in our bucket we have yet to put an object into the bucket we can do that by clicking on the upload button is going to go to the upload dialogue this case.
We鈥檙e going to add a file we鈥檙e going to go to a desktop and we are going to add this file in which is just a simple image file to gives us an ISO summary. We can add a bunch of files here to do a group upload. You can鈥檛 go bigger than 160 gig in one鈥檚 upload through the web browser.
You have to use another two. Go to do that and not many people are going to hit that limit. But you know a lot of time you鈥檙e not using the web interface to access S3. Anyway, so we click next and here we go to a setting permissions allow permissions again, we鈥檝e already said our bucket permission saying don鈥檛 let anything be public now, we can have individual permissions within our object so we can say in this case.
I鈥檓 logged in with the AWS account for a account call data PS4 the demo account and it has read permissions as well as read write permissions on the specific object. So can read all objects that I can read and write those I could add more users here. I could add another account.
You could say Okay, Joe鈥檚 allowed to access this Francine鈥檚 allowed to access this you could create quite a nuanced set of permission for this specific file. Now the way that works as we already have two layers in place. We鈥檝e got a bucket policy up top is a bigger filter and then the object policy.
You鈥檝e got a pass through The bucket policy to ever hit the object. So if you鈥檙e blocked at the bucket and allowed at the object that鈥檚 going to conflict denies always be allows. So you want to make sure that they鈥檝e got that sort of filtering through text and we pick our tier now for Tears.
You are going to allow so just a real clarify cuz that was going to my head when you have the bucket policy and the object policy in our bucket policy. We鈥檝e said no Public Access. So that鈥檚 going to deny any on offend ocated user. So anybody who hasn鈥檛 logged in won鈥檛 be able to access it and then down to the object level where we say Francine and Joe and Mark can access this object.
So an authenticated user what they Fred gets through that bucket policy because he鈥檚 not a public user. He鈥檚 a he鈥檚 authenticated we know who he is and then he goes the object policy and doesn鈥檛 have permission. So he鈥檚 not allowed now if we said nobody but Fred is allowed to come through this bucket policy explicitly denying everybody that would stop everybody else.
Right? So there鈥檚 a little bit of nuance there, but you get the point bucket policy first and then the object policy, but now we need to pick our storage class know this has a direct impact on your bill. So standard tier is the most expensive Vegetarian maybe a little bit more expensive now.
I鈥檓 to start but the idea here is basically how often are you using this stuff? How much durability do you need? So for standard that鈥檚 normal standard S3. It goes across 3 availability zones. There is a huge amount of redundancy. I鈥檓 just going to flip over the actual detail here.
So if we look at general purpose as 3 standard 11:9 of durability, which means it鈥檚 highly unlikely that your object will be changed or anyting flipped as far as bits show resilience against the entire AZ going down because you鈥檙e across multiple availability zones availability. That means your ability to get back 90 by 4/9 which is pretty solid.
Right and that is the standard way that you get Storage in at 3 and you can go serve reduce your access from there. So one鈥檚 own I say is going to give you can only into one鈥檚 own instead of 3, you鈥檙e going to keep it for at least 30 days at Glacier goes to Cold Storage in Glacier Glacier deep archive is even colder storage intelligent tiering is a machine is a new level of storage.
It is an intelligent model. We鈥檙e basically says I鈥檓 going to start in standard and then if you don鈥檛 use this object, like if you don鈥檛 nobody is accessing and I鈥檓 going to put it down to less frequent access and if nobody鈥檚 using it then I鈥檓 going to push the glacier if still nobody鈥檚 using and we can push it too deep archive because the as you go down this chart it鈥檚 less money to store.
So Glacier and deep deep archive Glacier are really inexpensive the store they cost a ton of money to pull out though. So the ideas for long-term stuff that you have to legally keep but you probably don鈥檛 want to access daily Glacier and deep archive are great for stuff. You don鈥檛 really care about the So naive makes a lot of sense and most of time you鈥檙e going to just sit and standard having the costs are not at that big.
So it鈥檚 you know, it鈥檚 a fraction of pennies and things like that. So we鈥檙e going to keep in standard. We鈥檙e going to upload our next and then we go to our review that鈥檚 good. And we upload it and all of a sudden we have a file in S3. It鈥檚 called an object.
You鈥檒l notice that it is sitting right now in just a root of that bucket really interesting thing is that there is no directory structure in S3, even though there鈥檚 literally a button that calls create folder. The reason being is that this object has a name that name is called the key.
So if you look at this the key for our file right now is the filename right ACG. It鈥檚 an ad for the course. I did on mastering the well architected framework. At least this is screenshot and you鈥檒l see we have a URL if I hit this URL right now.
Access denied why is the access denied because I鈥檓 not affect ocated on this URL on my Fanta created in the console. I鈥檓 not in the URL was blocked all public access. But if we hadn鈥檛 blocked all public access, you can actually give out your else for your S3 objects.
So this case the key here if we come back to our key Point pun intended couch fee at there are no file folders, but if I create a file folder, let鈥檚 call this test and then save it and now I鈥檓 going to upload the exact same file. I鈥檓 just going to Breeze through this because we鈥檝e already talked about it.
I鈥檓 going through the same dialogue. I鈥檓 keeping all the default settings blah blah blah blah blah and this is now uploaded now if I click on this file, you鈥檙e going to see the keys changed the filename still the same. I literally uploaded the same father didn鈥檛 change anything, but the key is now test / file name and that鈥檚 because their file folders are alive.
I file folders are a lioness 3 and they are away that clients have de facto. Visualize the de-facto way to visualize files to make it easier to understand. So if we come back to the root of our bucket, you鈥檙e going to see we still have a folder and then we have the file in the route and we have a file in the folder the what the reason is to set up in the keys is that anytime the key start with the same prefix and a slash it just makes more sense for us to be able to manage it but it interesting little bit of a history trivia.
There are no file folders. It鈥檚 all a giant giant lie. So if we click back to our key, you鈥檒l see that we鈥檝e got these these the high-level information. So when it was modified the storage class is currently sitting in which is standard its eyes its name. We can also adjust some of the properties here so we can change the storage class we can add metadata to it.
So if you want to do key value pears and add additional information of these files that鈥檚 handy for automation. We can also lock this object so that nobody can delete it and we can add more tags to it as well course we can go to permission. This will allow us to add complicated permissions.
You can see here right now with these are the access object for owner would you me and we can add Public Access even though it says we can鈥檛 right now because we鈥檝e been blocked which is great. So anything we add under here doesn鈥檛 apply because we blocked it but we can add other accounts we can add other users or we can pull our records out of S3, but that鈥檚 really simply put how S3 works very easy object in an object which has a key which is the name into a bucket.
I鈥檓 going to flip it back to the slide tournament cuz I want to talk in depth about permissions so we know we have an object. We know we have a bucket we know we have a bucket policy. We鈥檝e seen a little bit about the bucket policy. I meant to get if you guys have questions just fire him off in the LinkedIn comments.
I am honoring those actively so I and I expect some questions on this park is it gets kind of weird. So we saw a bit of the bucket policy already, which is When you have a bucket new crate, you can say no public don鈥檛 let anything in this be public and while we saw that in a nice little dialogue what was actually happening behind the scenes was the console created a bucket policy for us in that bucket policy said no public you can edit that bucket policy and say public with the following exceptions.
You could remove the public Walker altogether. You can Define how this bucket is used at the idea of a bucket policy. You鈥檙e taking a set of permissions and you鈥檙e applying it to the bucket itself and that can say check to make sure I know who鈥檚 accessing me that can see everybody鈥檚 allowed to look at this image file, but not this image file.
There鈥檚 a rather complex set of things that you can do makes sense. Now, this was the original way to do policies. I鈥檓 in permissions in S3 because the newer way didn鈥檛 exist so bucket policies apply to the bucket and then anything that goes in the bucket can have additional object policies on top of it.
The other way primary way to apply for missions in S3 is using AWS IAM the identity and access management tool. So that allows you to assign permissions to users or to roll. So I am a user of markers user a roll goes to an object. So do a AC to server morning stance is the parlance in a day or two container somewhere over to a Lambda function rolls get assigned to things.
Okay. So you can think of the bucket policy is really a role assigned to that bucket, but it鈥檚 managed on its own and that鈥檚 the problem. That鈥檚 where people get messed up really a bucket policy is just a permanent. I am roll assigned to that bucket, but it鈥檚 managed in the S3 area of AWS all your other permissions are managed.
Yes. I am. This is where you can say, you know what marks allowed to access this bucket, but only objects that fall under the test folder rights do anything with the prefix test slash marks allowed to access. That you can do that fine grain stuff assigned to the person or the role in I am and that鈥檚 really where you should be a signing the vast majority of your permissions stick to one place.
Don鈥檛 do both both ends up being confusing do what? It鈥檚 Monday. I want to melt your brain a bit because why not brains are for melting. That鈥檚 what we鈥檙e going to go with for Monday morning. There are two more additional ways to actually apply permissions in AWS. I鈥檓 coming back to the how to secure an S3 bucket article a while ago.
So we talked about I am policies we talked about bucket policies. There鈥檚 also a thing called an access control list that gets applied and query string authentication or URL based access so you can give somebody a URL and say you can post to this URL or you can get from this URL and it actually goes directly into S3.
There are scenarios where you want to use that however, I strongly as strongly strongly so so strongly recommend that you keep your access all in I am Just do it easier simple. That鈥檚 what I am. Therefore. It鈥檚 there for keeping all of your permissions in one place and give you a lot of great tools as a lot of great ways to audit check your permissions try to keep them in.
I am as much as possible. If you have to go to the bucket because remember the bucket think of the bucket policy as a role in I am assigned to the bucket permanently. That鈥檚 fine. That works, too. Try to avoid Access Control lists try to avoid query string in url base stuff because you鈥檙e just going to get lost and if you have to create a new account or new buckets and try not to mix these various ways on the same bucket because there is no simple way to view all of the permissions assigned to a bucket and everything in that bucket.
Okay, that鈥檚 why you know, we talked with the start y Mis configurations of the number one problem. There鈥檚 too many options. Now those options trust me. There are times when one of these will save your butt. And when you鈥檙e in that scenario, you really want that option, but do not mix and match she just because you can doesn鈥檛 mean you should a good rule for life in general, but try to separate these by bucket.
If not by account. And again that public thing supercritical to go buy account just to save your you-know-what so that in a nutshell is S3. Now. I鈥檝e got two minutes left on one of the cover something really really simply so you noticed my website here. This is were looking at Mark end.
CA and this is the page. I鈥檓 tracking all the stuff we鈥檙e doing around reinvents. You see here. Hey, you know next to him is right now that鈥檚 what we鈥檙e doing today. And then on Wednesday, we鈥檙e doing 10 a.m. We鈥檙e going to talk about Athena and quicksight and I posted all the stuff we talked about already right now.
This is actually being served out of an S3 bucket. S3 is a full web server super cool. What we鈥檙e going to do is flip back the S3 here and we鈥檙e going to create road to reinvent. Now we鈥檙e going to click next we鈥檙e going to leave all of the bucket options as is we are then going to set permissions.
We鈥檙e going to remove blocking Public Access and we鈥檙e going to create this bucket. Now the reason being in the reason why we called this. Com is because you can actually assign your Bucket to be lineup of the domain name. Okay that were clicked on the properties of the bucket.
We鈥檙e going to go to static web hosting and we are going to say use this bucket to host a website indexed. Lhd males are default are. HTML. There鈥檚 a bunch of options here for for setting up your website. We鈥檙e simply going to cook save then we鈥檙e going to go back to the week or this bucket and we鈥檙e going to upload a file and upload that exact same file that we offloaded time and time again, when we get to the permissions, what we are going to do is leave that will do in the 2nd standard upload them.
Just looking at the time here. Yeah, it鈥檚 been uploaded. So if we go now and check you鈥檒l see the URLs different. That鈥檚 fine. But if we click on the URL. We still have access denied just because the bucket is set to host a website doesn鈥檛 mean that鈥檚 going to work.
So what we need to actually do is go to permissions. Everybody can read this object big morning. Hey, this is public again. Don鈥檛 do this in the same account create a second account to do it. We鈥檝e a dummy demo account that we鈥檙e looking at today, but now that we鈥檝e assigned this to be a public if we just refresh this we should see that everyone has the read access again.
We come back to our URL. There we go. Now this your LS3 on amazon.com name of our buckets the key name gives us access to this picture. Okay, you can set up an entire website there. That鈥檚 in fact what I鈥檝e done with my personal website and marketing. CA that鈥檚 in its own separate account and being hosted so that I don鈥檛 make the mistake of doing something like my personal photo archive being exposed.
But S3 has the ability to serve websites directly course, you want to make them fast, you can put them in like a cloud front in front of them and put that in a nutshell. We are at our time block here and let me switch back to me that it was S3 your buckets.
You object you put an object in a bucket. You have multiple ways of setting permissions stick to one of them. If you can stick to either I am permissions for everything or bucket policy for everything. If you need to have a private bucket or a public bucket put in a different account Accounts cost you nothing usage cost you so you鈥檙e not going to pay any more to put it in a and it鈥檚 separate account.
I鈥檓 going to pay the exact same thing. So that鈥檚 a good strategy operationally. So you don鈥檛 have to shoot yourself in the foot by putting something sensitive into a bucket of configured for public because we鈥檝e seen that time and time and time again over the last few years where people have made a simple mistake taking a bucket that starts lockdown and unfortunately adding additional permissions because of the different ways to add permissions, they get tripped up and expose customer records are sensitive data and that is a bad the very bad thing.
And this has been a quick introduction into S3. We鈥檙e going to build on Wednesday by looking at Athena and looking at Kwik site. These are tools that help you access the data in S3 or visualize the data in S3, really cool kind of hidden services to my favorite actually are off.
Overlooked and shouldn鈥檛 be so let me know what you think in the comments below. Thanks for joining the stream the next one as you saw is on Wednesday at 10 a.m. Eastern again. Normally don鈥檛 do Monday money is going too hard and but because I鈥檓 at the AWS Summit in Toronto on Thursday and made sense to do this here.
So I hit me up. Let me know. Thanks for joining the stream really appreciate it. Have a great day, and we will talk to you soon.
