Originally presented at Cloud Security London meetup on 06-May-2020. You can view the details on the Meetup page.
You can view the slides as a PDF and watch the talk on Twitch.
The Unicorn Project by Gene Kim is the fourth book in this landmark series.
The Phoenix Project introduce the concept of "DevOps" to the world, showing work IT solutions could be delivered in a much more collaborative manner.
The DevOps Handbook builds on the success of The Phoenix Project but provides a much more pragmatic approach to the problem. It's chalk full of potential solutions and tactics.
Finally, Accelerate helps to make the business case for change. It highlights the advantages of the DevOps philosophy as it applies to the business.
Roughly summarized:
- Development and Operations should work together (The Phoenix Project)
- Here are the tactics and playbooks to help (The DevOps Handbook)
- Here's data to help support the cultural transformation (Accelerate)
- Development need better tooling & support (The Unicorn Project)
There are more than enough roadblocks in the way of development productivity. The biggest challenges:
- A lack of understanding of what needs to be in place to deliver desired outcomes
- Getting data to where it can be used most effectively
- Opposition to cultural change
Some of these roadblocks are within the team, with the CIO's organization, or the wider organization. Where a roadblock is will dictate how you can respond and remove it.
The book introduces the concept of the "five ideals". These ideals are what development teams should be working towards/for in order to deliver more to their customers.
- Locality and simplicity
- Focus, flow, and joy
- Improvement of daily work
- Psychological safety
- Customer focus
The book follows it's heroine, Maxine.
She's a top notch engineer who isn't afraid to ask hard, or really basic questions. That's a valuable skill that we'll see more of throughout the book.
The setup for the narrative of the book is very straightforward and will probably seem familiar for some.
There's a big outage that Maxine handles and save the company from a far worse fate. No good deed goes unpunished so instead of acknowledging organizational failure, Maxine is blamed.
She is cast out into the realm of "special projects" and her journey there ends up transforming a key part of the entire organization...not without more than a few challenges along the way.
For the purposes of this talk, we're going to explore Maxine's perspective. We're also going to introduce a new character, William. New as in "doesn't appear in the book at all" level of new.
He's entirely made up and will stand in for the security team. The actual security team members in the book aren't exactly inspiration. Very accurate but not inspiring.
So, we've got William and we're going to ensure that he does better than the security folks in the book.
Locality and Simplicity
A typical organizational workflow is an exercise in jumping through hoops. In order to deploy to production, you need access to the code repo or repos. Those repos have stakeholders, you need to check with them too.
Same with access to various systems, licenses for software/services, resources like compute, storage, etc and customers. Remember them?
Bottom line? At every step there are multiple people that need to be involved. That usually means established processes and a lot of negotiation.
This is NOT local and NOT simple.
The ideal flow is one where Maxine and the team have the required tooling readily available. They also have the authority to use it to best serve the customer.
This also provides a direct line to customers. Listening to them and working to resolve their concerns directly.
Stakeholders are still involved but at a much higher level. In this scenario, the stakeholders have authorized and enabled Maxine and team to accomplish the goal at a high level. "You're a pro, get it done", is a very nice approach.
As a security team, you can help development by getting out of their way. If security is a gatekeeper and needs to approve multiple steps, that's a frustrating roadblock for development.
DON'T.
Your role on the security team is to help educate and enable development. You can't do everything on your own (remember that pesky security skills gap problem?).
Making sure that development can use any security tools (like container scanning, vulnerability scanners, etc) as a self-service tool is critical to success. It's time for security to adopt the cloud model: API first.
This ideals applies to security as well.
Centralizing access to logs and their analysis is a great place to start. You're not going to be able to pull in all of the logs from your deployments to a central data store. That's just creating a massive data management problem. Leave the data where it is and centralize access, way simpler.
The same goes for audit access. You want to monitor (a/k/a read-only) what's going on out there. Using roles and cross-account credentials are key to success here. Set these up early as a standard part of your new account setup.
Finally, realize that you can't control everything. Set up guardrails to make sure that teams don't run into trouble. Remember, help don't block.
Focus, Flow, and Joy
It's probably a little weird to talk about "joy" when it comes to development. Anyone who's ever debugged anything is probably scratching their head right now.
This ideal is all about making the lives of developers better. Steps include;
- Use tools that make solving problems easier
- Focus on solving the business problem
- Leverage platforms for immediacy and fast feedback
Looking at the now classic "DevOps" flow (left hand side is development, right hand side is operations), security needs to adopt a different approach for each side.
In development, provides automated tools for development teams. Things like container scanning, vulnerability assessments, and others should all be just an API call away for teams. *Remember the results should be returned in a machine readable format too!
On the operations/production side, security should focus on immutability. Stability is everyone's friend.
Automate. Everything. No exceptions.
Improvement of Daily Work
* Sometimes, it's just easier to admit things up front. Security is really bad at improving our daily work.
Development general uses two flows to improve their daily work. The first is the "innovation flywheel".
Have an idea, run an experiment, get feedback. Repeat.
The second is more nuanced, it's the idea of the "andon cord"
When an issue is first seen, work should stop in order to address the issue right there and then. This avoids the creation and accumulation of technical debt.
The security team can help development use these flows by focusing on education and creating self-service tools. Teams can localize and improve on their work if they don't constantly have to ask questions of others and loop them in.
In order to improve our own work, security teams should adopt the concept of the "andon cord". This will help reduce technical and security debt. It also creates a fantastic entry point for automation. Every time someone pulls the cord, an automation should (probably) be the result.
Psychological Safety
In order to thrive, development needs a culture where;
- It's ok to make a mistake
- There's no fear of reprisal
- It's normal to discuss problems openly
Security can help this culture by;
- Not assigning blame (especially during an incident)
- Teaching and supporting a culture of learning
- Trusting and enabling development teams...and yes, verifying
Security teams need the same things thrive, a culture where;
- It's ok to make a mistake
- There's no fear of reprisal
- It's normal to discuss problems openly
Customer Focus
A customer focus starts with a focus on activities that a core to the business. IT in general has a ton of work that is seen as necessary to get to the real work. That work should be aggressively removed.
"Does this matter to our customer?" is a strong guiding light. If you can't draw a direct line between something and a customer, get rid of it (and "rid" can be stop doing or delegate to a cloud service provider)
Security needs to adopt the same principles. If it's not part of the core business, move that work along to another provider or simplify things in order to remove that work.
Keys To Succes
The five ideals apply equally to security and development. They can help make a team more productive, happier with their work, and drive business success.
You need to work to focus security practice on;
- Educate development about security concerns
- Provide self-service/API drive security tools to other teams
- Improve your daily work through relentless automation