Security is a service business...one star

I'm confident that most of security professionals will answer this in one of three ways;

"I don't know."

"I don't think they are."

"No."

None of those are great answers to the question.

Nope.

...organizationally 😉

When was the last time you designed a process for your team?

No, I don't mean writing down an playbook (though you should be doing that). I mean working through the steps of a systematic effort in order to design a process that works for your team and your customers.

Have you ever done that?

Let's start with first principles. There's always a reason why things end up in their current state and there's a lot we can learn from that history.

Why do most security teams organize the same way? Is that the best approach? Or just something we ended up with over time due to external factors?

This all started with endpoints.

Acknowledging that there was risk with our desktops (yes, desktops), organizations started to have folks assigned to managing these systems.

Not like we do today, but the first steps were there. Organizing the OS and its updates, anti-virus software, and other steps to help protect the business.

The real nucleus of what we know of as the security team came to be with network controls. Rolling out firewalls, then intrusion prevention, and other controls around the perimeter was enough work that dedicated teams were required.

No more—well, less—side of desk work. We now started to see teams responsibly for the castle wall protecting the "inside" of the business.

As connectivity expanded, we get closer to today. Teams are dealing with endpoint, network, and cloud controls.

While each of these areas contribute to defence in depth, we also approach them based on the security team's level of responsibility or influence.

Endpoint controls are still very much in the "OK, if it doesn't impact anything" bucket. Security teams tread lightly here, so as not to lose trust with the rest of the business.

Network controls are easier to roll out because they are typically entirely within the security team's purview, or at most involve a small handful of infrastructure teams.

Deploying security controls in the cloud can be more direct. WIth all resources available via an API, connecting to systems, monitoring them, and gaining visibility are more straightforward than ever.

But there's more to security than just these three areas. We've expanded to risk practices, compliance activities, and proactive work like threat hunting.

Security teams in medium-sized enterprises, are likely to scale to have one or two—or more—dedicated resources to each of these areas. Larger organizations can even get to the point where they have dedicated teams for each of these areas.

But one thing that tends to hold true—even for the smallest of teams—is that we organize our teams based on function.

This is Francine, she is responsible for our risk practice. Jo takes care of compliance. Etc.

Function structures tend to exhibit these properties:

• They allocate resources based on their functions
• Information flows up and down easily (or least by default)
• Decisions tend to stay within each of the functions
• Individuals in each function will develop deep expertise in that area over time
• Explicit workflows are required to break silos

And it's this last point that is the source of most of our challenges.

I don't think this structure is conducive to workflows that will meet your goals. Or the goals of your customers.

Worse, I don't think that we have the time/energy/awareness to step back and examine the link between our team structure and our workflows.

Simply put, we are too busy doing the work to understand how our approach to the work is making it harder for everyone.

In this section, the audience is asked to—and politely does—participate in a group activity. They say each of the letters as they appear on screen.

A B C D

E F G

H I J K

Stop.

I'm not sure why y'all are doing it this way. Let's restart.

(In person, the audience almost always nails this part. They are saying each of the letters in English at the same time and nailing the beginnings of the song as well.)

A B C D

Stop.

I ask the audience, "Why are you saying it that way?"

They are confused. I then repeat the beginning of the alphabet in Dutch. The letter sounds are very different than the English ones.

The point of this callout is that I had very different expectations for the activity. Expectations that didn't line up with the audience's assumptions.

On the same page & language, we restart.

A B C D

E F G

I ask the audience, "How many vowels have we said so far?"

This breaks the flow of the recitation and song. It's an unexpected question, even though it's simple one to answer.

We restart for the 3rd time.

A B C D

E F G

H I J K

LMNO P

Inevitably, a North American audience will say L, M, N, and O as "elemenopee"

It's a fun call out and it runs counter to the previous pacing, but it aligns with the song.

The point of this is that it's an unspoken change that everyone just gets. They go along with it because of the ingrained cultural elements, not because they talked about it beforehand.

Everyone in the audience (or enough that the point is made) recites the alphabet using, "The ABC Song" instead of just saying each letter in turn.

There wasn't a discussion or agreement to do this. Outside of the subtle hinting in the visuals, it's just what everyone defaults to.

It's a cultural expectation. It's "the way we've always done it".

It's a direct parallel to a lot of activities in organizations and frequently the security team (me as the speaker in this case) is unaware of that expectation!

For fun, the audience gets to repeat the whole song without interruption.

A fantastic amount will also—always—add the bonus line, "Now I know my A B Cs, next time won't you sing with me?"

For a bonus, unreleased tangent, it's pointed out that most folks also can't repeat a segment of the alphabet without starting at A and ending up in a close approximation of the song as well. Human brains are weird!

Everyone knew the song. You default to it, because you learned it and practiced it a lot as a child.

It's a shared experience that reinforces the original experience and understanding.

I restarted the group 4 times. Each time to clarify something for me or to force the group to confirm to my expectations and requirements.

That's a generally frustrating experience. While trying to fulfill my needs, I cost the group time and enjoyment.

...pausing to let that sink in...

Teams generally work well (enough) together.

Don't be the one who disrupts that.

Don't be the one who disrupts that to serve your own needs...even if those needs will help serve the group!

Let's pivot to an even more frustrating topic. But it's a topic that we can actually learn a lot from and relate to as a group.

When they first rolled out, self-checkouts were hailed as technological advancement, a time saver, and an overall benefit to both the business and the customer.

There were some discussions about the balance of those benefits, but outside of the "old man yells at cloud" segment, there wasn't a lot of negativity...at first.

I bring up self-checkouts because I'd like to share a story to help illustrate my overall point of the importance of explicit service design. To help understand how we can all be more effective security practitioners, I'd like to talk to you about my local pharmacy...

Before rolling out self-checkouts about 18 months ago, my pharmacy had six checkout lines.

Each one of the checkouts was staffed. In peak times, they had six employees running the six checkout lines.

If we put ourselves in the owners shoes, the six checkouts—running at a theoretically maximum—would require about $0.59/sale in overhead.

We get to that number by looking at the number of sales each line can process during an hour and the cost to serve that line.

When the pharmacy deployed their self-checkouts, they made a couple of slight adjustments to the traffic flow.

The two middle lanes now were product shelves for those impulse buys. The back wall now housed 3 self-checkouts and so did the left-most checkout line.

The right-most checkout line was kept as a staffed line to help address any customer issues. This employee was also responsible for helping any self-service customers who encounter issues.

Now, when we adjust for the extra time it takes for self-service, the overhead drops significantly for the store.

They are pushing through less sales (120 vs 180), but at 25% of the overhead.

Given the average sale at a pharmacy these days, this probably isn't a great business move. However, the back end costs for employees are going to be significantly higher than maintaining the self-checkout systems.

The self-checkouts also don't have scheduling issues. They are always available and you don't need to try and predict demand. There's a consistency there that simplifies operations.

The problem—ok, a problem—the store encountered quickly was that four of the six self-checkouts weren't seeing much use.

The reason was simple, customers weren't seeing them!

The product displays which were thought to be a clever way to re-purpose the previously staffed checkouts, were interfering with the view of the self-checkouts.

Customers were queueing up like they used to for the staffed checkouts and not taking advantage of the additional self-checkout capacity.

When we look at the throughput from this challenge. The overhead is half of the full service approach, not a quarter of it.

That's a huge impact to the expected savings. This is a problem that needs to be solved.

The solution the pharmacy came up with was to remove the obstructions. This makes perfect sense and really opened up the area.

While it removed the ability to convert the impulse buyers, it made it a lot easier to see the entire set of checkout options.

But there was a problem...

A significant percentage of the customers for the pharmacy are seniors. Seniors who were not having anything to do with the self-checkouts.

When presented with the suite of options, the seniors overwhelmingly selected the full service option. To the point where they were queueing up when almost all—if not all—of the self-checkouts were open.

This reduced the checkout throughput of the store dramatically.

Any guests on how the store "solved" this challenge?

To address this issue, the store put up a new half wall. They physically blocked the direct access to the full service checkout.

The positive (?) aspect to this solution is that it helped to shape the queue. Instead of blocking traffic to the main shopping aisles, the queue now formed in the checkout area.

However, this block reduced the visibility of the full service checkout. The customers who wanted to use it had to now go out of their way to queue up for it...if they saw their preferred option at all.

This also doubled their walk for the workflow. They now had to walk to the queue, move to the full service checkout and then walk past all of the self-checkouts (again) to leave the store.

This is not a good solution and customers complained. To help address this, the store added an additional staff member to help guide more people to the self-checkouts.

In isolation, each of these decisions makes sense. Given problem X, solution Y is a reasonable approach. But, when you examine the overall workflow, the entire problem space, you see how ridiculous these steps are.

From the business perspective, the numbers are better. Overhead is down.

But what about customer satisfaction? This is much harder to measure. Anecdotally, as a customer, I can tell you it's down. How much will that impact their bottom line? I'm not sure.

For our purposes, the key takeaway is that even though the steps taken to address each issue were logical and moved towards the state goal, the result isn't what was intended.

It's not just my experience or this pharmacy, self-checkout has not been an amazing solution.

Through multiple iterations of the various platforms, a positive and smooth self-checkout is a very rare experience. This is now one more thing that we just put up with...despite the general feeling.

Again, this is a result of a series of logical decisions. The problem is that the context window for those decisions got smaller at each and every step.

The end result is a lot of effort and an outcome that may—or may not—align with the actual business goals.

While there are formal methods of doing service design, at it's core, simply asking questions and listening to the feedback will improve your team's workflow significantly.

However, the principles proposed in "This is Service Design Doing" are a great way to establish a shared understanding of what you're setting out to do.

In the simplest terms, those principles are:

• Take the customer's perspective (human-centered)
• Involve a diverse set of stakeholders in development (collaborative)
• Small experiments, fast feedback loops (iterative)
• Visualize and orchestrate the whole process (sequential)
• Get out into the actual environment of the service (real)
• Address customer needs sustainably throughout (holistic)

"This is Service Design Doing" is an excellent starting point. It's not the only reference out there, but it's very approachable and the Methods book is a great playbook to help you implement changes in your team.

There are a lot of different frameworks for doing risk and threat assessments. There are advantages and disadvantages to each, though really any will do.

The fact that you're conducting assessment—and regularly updating them!?!—is the most important thing.

How many folks use one of these frameworks? Or something similar?

Do you conduct the assessment when the team is writing the code and building the solution?

...when they are testing the solution out?

...or maybe when it comes time to run the solution?

Trying to start and then finish an assessment just as things are going to production is far too common. We—the security team—end up in this position often because of some of the service challenges we're talking about here.

Of course the answer is that you should be doing risk assessments as a continuous process. There is assessment activity at all stages of solution development.

But, this only works if you're collaborating with the builder team. If you have the trust of other groups in the business. You have to work together and towards a common—and commonly understood—goal for this to actually work.

How do you end up in this utopia? This fictitious, "it's easy to put on PowerPoint" world?

The honest, open answer is, "Slowly, patiently, with a series of small steps that each get your closer to your shared goals."

Let's start by looking at the service design principles and the questions we can ask ourselves in order to start to find the path forward.

If we take the customer's perspective, we should have answers to the following questions:

1. What are we doing this?
2. What do I get from it?
3. How can I make this easier?

When it comes to risk assessments (and other security work), often the answers are:

1. Not sure.
2. No idea.
3. Just not do it?

Those are not great answers and they are strong indicators that we—the security team—need to be doing a much better job of communicating.

When addressing a good representation of your stakeholders, ask the following of your own team (security):

1. Will the same process work for everyone?
2. What are the key outcomes?
3. Are we removing waste from this process?

Making small changes and getting feedback as quickly as possible is one of the most important things you can do for your work.

1. When was the last time we asked if this worked?
2. Do we gather data on our process?
3. What adjustments have we made?

These are all questions that will help you build your feedback loops and hope you to create a truly iterative process.

In the examples we worked through today, we saw the value of taking the big picture view. Understanding the entire process is the only way to avoid the shrinking context path like we saw with the self-checkout example.

1. Does our work start and stop at our team "borders"?
2. How much do we know about our customers?
3. What happens after the assessment is done?

Visualizing and orchestrating the whole process is key to breaking out of your silo. It's how you counter the limitations of the functional team structure.

Too many teams lay out their workflows based on their understanding and expectations of the customer. While it's possible that this might be accurate, it's unlikely.

Getting out and experiencing your customer's reality will help you understand their perspective. That understanding will lead you to better solutions.

My pharmacy didn't understand the majority of its customers. They missed the fundamental frustration that self-checkouts bring up with their older customers. No one wants to feel like they don't understand or that they are the problem and "don't get" the technology.

1. Have you sat with your customers? With theirs?
2. How often do you connect with the business?
3. Do you know how other teams work?
4. Have you tried their work, their tools?

1. Are you working other make things simpler?
2. Can you help the customer do more on their own?
3. Is there something working really well? Can you do more of that?

Sustainability in processes is tied to complexity. Do not attempt to design a process that covers 100% of the edge cases. A workflow that solves 80–85% of the most common cases and has an allowance for the remaining 15% will be far more effective.

When making a decision, the simpler path is where you should be aiming.

If your customers are unhappy, you have work to do. Frustrated teams work around security workflows. Not because they don't want to be secure, but because they want to get their work done.

Security is in their way. You have to avoid that at all costs.

So, do we think that the structure of our teams is influencing our workflows? And that these workflows are not serving our needs or our customers?

I do. And I think we need to change. I confident we can change and that those changes don't need to be all compassing to start.

We start by choosing to address these gaps.

We build a network of support within the business. Build understanding of how other teams work, how they communicate, and how our shared goals align.

You cannot succeed as a security team without the support of other teams in the business. The numbers simply don't add up. You need to succeed together.

The good news is that you have the same goals, you just may be speaking different languages right now or failing to share each others perspective.

You can address these challenges and improve your security by working together. And that starts with you taking a small step towards that goal.