Security Cloud Courses About
imgs/hero.webp

Amazon S3 Glacier Re:Freezer

AWS Labs has a lot of open source code up on GitHub. Amazon S3 Glacier Re:Freezer is one of those projects.

It’s got a simple premise: get the data out of Amazon S3 Glacier and back into Amazon S3.

That sounds simple but the solution is surprisingly nuanced. I call out a few more details in the Twitter thread below…

Tweet 1/10 ๐Ÿ‘‡ Next tweet

let's take a look at the Amazon S3 Glacier Re:Freezer project from @awscloud Labs

the project is available at https://github.com/awslabs/amazon-s3-glacier-refreezer

๐Ÿงตโ˜๏ธ #cloud #devops

Tweet 2/10 ๐Ÿ‘‡ Next tweet ๐Ÿ‘† Start

this thread can be see unrolled at https://t.co/4d3MidmVDs

the previous thread about AWS Perspective is up at https://markn.ca/2021/aws-perspective/

๐Ÿงตโ˜๏ธ #cloud #devops

Tweet 3/10 ๐Ÿ‘‡ Next tweet ๐Ÿ‘† Start

the goal of this project is simple:

get data out of @awscloud S3 Glacier and back into an Amazon S3 bucket

that shouldn’t be too hard…

๐Ÿคฃ๐Ÿคฃ๐Ÿคฃ

๐Ÿงตโ˜๏ธ #cloud #devops

Tweet 4/10 ๐Ÿ‘‡ Next tweet ๐Ÿ‘† Start

I laugh because this is the solution architecture ๐Ÿ‘‡

๐Ÿงตโ˜๏ธ #cloud #devops

Tweet 5/10 ๐Ÿ‘‡ Next tweet ๐Ÿ‘† Start

think about that for a minute. it builds out resources in SEVEN different @awscloud services just to move data from one service to another

worse, those two services are intertwined already!

if there was ever a solution crying out to be in product, it’s this

๐Ÿงตโ˜๏ธ #cloud #devops

Tweet 6/10 ๐Ÿ‘‡ Next tweet ๐Ÿ‘† Start

ok, rant aside (for now), this is a very useful solution. Glacier is the long term, cold storage (https://aws.amazon.com/s3/glacier/) where writing is easy and reading is slow and expensive

Glacier is the “just in case” place for storage.

๐Ÿงตโ˜๏ธ #cloud #devop

Tweet 7/10 ๐Ÿ‘‡ Next tweet ๐Ÿ‘† Start

Glacier is a great service because it's super low storage costs (https://aws.amazon.com/s3/pricing/) make it easy to keep things "available"

you’re charge to get data out and for those times you actually need that data, it’s usually worth it

๐Ÿงตโ˜๏ธ #cloud #devop

Tweet 8/10 ๐Ÿ‘‡ Next tweet ๐Ÿ‘† Start

you would think that restoration process is automation in a slick manner. it isn't natively

this solution does a nice job of smoothing out this process

simply:

  • point at the Glacier Vault
  • pick your retrieval tier
  • target bucket
  • target storage class

๐Ÿงตโ˜๏ธ #cloud #devop

Tweet 9/10 ๐Ÿ‘‡ Next tweet ๐Ÿ‘† Start

the complexity in the architecture ๐Ÿ‘† is because the data retrieval from @awscloud Glacier is glacially slow

like a herd of ๐Ÿข๐Ÿข๐Ÿข who aren’t sure which way they are supposed to be going type of slow

๐Ÿงตโ˜๏ธ #cloud #devop

Tweet 10/10 ๐Ÿ‘‡ Next tweet ๐Ÿ‘† Start

all of the different components in the design make sure that the data gets to the target bucket with a high level of reliability

should this be part of the core service offering ? yes

but it’s not, so use this code

/๐Ÿงตโ˜๏ธ #cloud #devop