Maya Kaczorowski

What do you call that thing when your vendor gets hacked?

Fri, 12 Sep 2025 07:00:00 GMT

The thing that bugged me the most in the past week about the Salesloft Drift incident and the npm Chalk/Debug packages issue isn’t how disruptive they were — we really did get lucky with npm — but that we still don’t have a standardized way to talk about these situations. (Also, when you go to the Salesloft website, the banner is about a merger and not the security incident. Why did you think you were getting all this traffic?)

When software or hardware has a vulnerability, it gets a CVE. You can reference it, check if you’re patched, and customers immediately know what you’re talking about. Everyone is speaking the same language.

But vendor breaches don’t get CVEs. Salesloft won’t get assigned one — you don’t care what version of Salesloft you’re running (disclaimer: I mean I assume not? I have no idea how Salesloft is deployed), you care about whether your data (or your vendor’s data) was exposed as a result of the breach. Just like when Snowflake’s customer environments were breached in 2024, or Okta’s support tickets in 2023, or SolarWinds in 2020. If I’m a customer of any of these companies, I need a way to communicate with my customers about potential impact. Welcome to the reality of *aaS: your security posture now depends on a vendor three layers deep that you’ve never heard of.

Two simple things would help:

A unique reference identifier, like COMPANY-2025-007. It’s not unreasonable that we’ll get two of these incidents from the same company in a year, so “Company 202X” won’t cut it and isn’t easy to search for. I worked on GKE where we added our own reference numbers to security bulletins and copied that idea at Tailscale — even if just to give our customers a way to consistently reference these. (I would highly recommend doing this.) Ugly, but it works.
Machine-readable security bulletins, like an RSS feed with incident details that organizations can ingest automatically (some trust center updates are getting closer to this), or even better, something I can poll for this incident identifier. These should be separate from regular security bulletins because they’re informational updates, not necessarily actionable alerts. Your subprocessor list is hopefully public, and if a company on that list has a major incident — or if it’s just a big enough deal — you’re going to need to publish something anyway, even if it’s just “we’re not affected”. (We also suffer from not doing this for black swan events like log4j.) People ask, so just give them what they want.

We already have the concepts! We just haven’t applied them to vendor breaches yet.

MCP is the new interface for security tools

Fri, 28 Mar 2025 07:00:00 GMT

In the past few months, Model Control Protocol (MCP) servers have slowly been gaining in popularity. But this week, something incredible happened — we hit an inflection point. Three different people I know built MCP servers for interacting with security tools since Sunday. One did it in just 20 minutes. I expect I’ll hear about another one by the end of today.

MCP is an open protocol, originally developed by Anthropic, that allows models to interact with external tools and systems. It’s another interface, like an API — and it’s opened a whole new realm of possibilities. MCP implements tool calling, but provides a common format (with a spec) for the industry to connect agents to tools, with an understanding of what the tool does and how to use it. It’s a standard that lets you expose a tool’s capabilities — so you can build agentic systems that call different agents in order to access specific information or use specialized knowledge. Even though MCP is open, most people I know use it with Claude (and generally only really use Claude). OpenAI recently already added MCP support as well, in a desperate bid to continue to be relevant (or am I the only one who doesn’t see the enterprise value of Ghibli-style images?).

Security teams stand to benefit enormously from MCP for three key reasons:

Security is fragmented: We have dozens of tools generating alerts, logs, and findings. MCP can pull this disparate data together without custom development.
Not all security professionals code: Many security analysts and leaders aren’t engineers who can code. MCP bridges this gap, allowing non-technical users to get the insights they need through natural language.
Security drowns in data: Everything in security needs context. Only data engineering deals with more information volume — but those folks already know how to query it effectively.

This is a complete rethinking of how we interact with security tools. This reminds me of Tines, which rethought how we create workflows for security tools. Tines is a general-purpose connector, but its entrypoint to the market was in security because the pain of tool fragmentation hits us hardest.

Using a model with MCP servers doesn’t only give you a way to ingest, analyze, visualize — and so understand — information, but also to act on it. You can take actions, such as create a new group, or ack an alert. Users will absolutely not realize the power of this and accidentally perform sensitive actions, like delete their account. But the potential is there to simplify a lot of complex security work.

The “single pane of glass” vendors have been promising for years will finally arrive, but it won’t be another dashboard. An MCP-enabled client becomes the front end of choice, with an LLM building custom visualizations on demand for whatever specific question you have. This is more than just an evolution of a Slackbot, it’s fully custom to your specific user needs. You don’t need an enterprise- or security-specific agent. You just need an MCP-enabled agent — and to control what it has access to.

If you’re building security tools, your job isn’t UI anymore — it’s data and interfaces. Products selling just “visibility” will face a reckoning as LLMs become the interface. In the past few months, users have built MCP servers for security tools they use, such as Ghidra, VirusTotal, Keycloak, Snyk, Trivy, and many many more. But in the last few days, vendors have realized the value and already come out with MCP servers, like at RunReveal and Semgrep. We’re seeing the same evolution we saw with unofficial versus official Terraform providers.

Remote MCP servers are even more exciting because they don’t require local deployment — they allow your local client to connect with web-based servers (like those provided by your security SaaS tools). Squint and this looks like service-to-service communication.

This allows us to actually build agentic workflows. What if the model received an alert, automatically investigated it, and took a remediation action? I know this is what we’ve all been talking about, but now it’s possible. It’s something we could incrementally add to our existing stack.

The security of these MCP servers will be critical. MCP servers must use OAuth 2.1 for authentication and authorization — well, if they implement any auth at all 🥲. We’ll also need audit logs and approval flows.

The challenge isn’t just implementing MCP — it’s also figuring out how to properly implement user prompts for OAuth (which I’ve written about previously) and approvals for sensitive actions. And, managing these at scale, when every employee in your environment has a slightly different set of permissions and access.

MCP is poised to fundamentally transform security operations, turning LLMs into the interface of choice for your security stack.

FedRAMP by the numbers

Thu, 20 Mar 2025 07:00:00 GMT

Conversations about FedRAMP in security seem to have accelerated recently, irrespective of the political climate. It seems like everyone is getting FedRAMP, and tech execs are wondering, should we get FedRAMP? Will that help us increase sales?

I have never worked on a product while it underwent FedRAMP authorization. I am underinformed about the whole process — so naturally, I am curious about how it works. There seem to be more conversations about FedRAMP.

So, I set out to understand if the rate of FedRAMP authorizations increased recently, and if getting a FedRAMP authorization helps accelerate sales to the US government.

Luckily for us, since FedRAMP is a marketplace — and the data is public (well, at least for now) — I decided to dig in.

Don’t need more context? Skip to the analyses

Context and terminology

Let’s get some facts straight about the Federal Risk and Authorization Management Program (FedRAMP), so that we can focus on the data.

FedRAMP was introduced in 2011 as more and more technology companies were beginning to offer cloud-based services. The Federal Information Security Modernization Act (FISMA) introduced minimum security requirements for federal agencies in 2002 — although the US federal government had an existing procurement process for on-premises software, they wanted to adopt more cloud-based services, and so needed to adapt. FedRAMP essentially extends those requirements to provide a standardized way to authorize cloud-based services.

FedRAMP is an authorization: you obtain an Authorization to Operate (ATO) a product for the US government. A cloud-service provider’s (CSP) cloud-service offering (CSO) is FedRAMP authorized, not FedRAMP certified or FedRAMP compliant. By that, the US government is saying, we will accept the risk of the service you’re offering. Although a third party auditor assesses your controls, they’re not making a decision or recommendation, just an assessment. Whereas a certification might verify you meet a strict set of requirements, in theory, an authorization could still apply to a product with a concerning but still acceptable risk. I am not aware of whether this actually happens in practice, but the choice of vocabulary here is to be noted.

The only time you need FedRAMP is to sell a cloud-based service to a US federal agency in the executive branch. FedRAMP recognizes and includes CSOs with IaaS, PaaS and SaaS services. You do not need FedRAMP to sell an on-prem service, you do not need FedRAMP to sell to the legislative (e.g., Library of Congress) or judicial (e.g., Supreme Court) branches, and you do not need FedRAMP to sell to other US government agencies, such as state schools or city governments. Although having FedRAMP authorization may make it easier to sell to these organizations, it is not required — at least not by law, though it may still come up in your sales conversation. Having FedRAMP authorization is not necessarily sufficient for federal agencies either — kind of how having SOC2 certification doesn’t make you secure, if you’re selling to say, the military, they will have higher requirements (look into DoD Impact Levels).

You can still build a product that helps an organization meet NIST 800-53 requirements — that is, help your customer meet FedRAMP requirements — without yourself being FedRAMP authorized. You only need FedRAMP authorization if you are directly processing certain kinds of data.

FedRAMP has four impact levels: High, Medium, Low, and Low-Impact SaaS, which is related to the risk of the service and the data being put in the service. It’s what it sounds like: High is harder and has more requirements than Medium, which is harder than Low. High impact data is “the government’s most sensitive, unclassified data”, which is “usually in Law Enforcement and Emergency Services systems, Financial systems, Health systems”. These were the three original levels; then, Low-Impact SaaS (LI-SaaS) was introduced in August 2017, for “SaaS applications that do not store personal identifiable information (PII) beyond that generally required for login capability (i.e. username, password, and email address)”.

To obtain FedRAMP authorization, you need to show that you meet a specified list of controls — for FedRAMP Low, there are 125 controls. Of these, the most onerous requirements are those to use FIPS-validated cryptography and to patch vulnerabilities within specified timeframes. There are lots of other requirements: employee account lifecycle management, restrictions on remote access, centralizing logging, documenting and justifying enabled ports and protocols, etc. — but these are generally good security recommendations that many organizations already meet. Like other compliance frameworks, FedRAMP doesn’t necessarily have that many strict technical requirements, and rather the real challenges are organizational, to adopt and automate processes.

FedRAMP authorization process

There were historically two ways to obtain FedRAMP authorization for your CSO: from the Joint Authorization Board (JAB) for use with multiple agencies, or from an individual agency.

If you went the multi-agency route, you’d submit your CSO to the JAB for review. The JAB was made up of CIOs from the Department of Defense (DoD), Department of Homeland Security (DHS), and General Services Administration (GSA), and made sure you met the general requirements before granting provisional authorization (P-ATO). Then, you’d get full authorization when a specific agency used your service. FedRAMP says that “a JAB P-ATO would be better suited for cloud services that are Moderate and High Impact”; I’m assuming this is because of how difficult and detailed the higher impact reviews are.

Or, you could have asked a specific agency. They’d stick their neck out to use a particular CSO, and and drag you through the process as your sponsor. FedRAMP allows ATO re-use, so that if one agency cleared you and you were already on the FedRAMP Marketplace, there would be much less work to get approved at another agency. This encouragement was formally written into law in December 2022, with a requirement to “provide a secure mechanism for storing and sharing necessary data, including FedRAMP authorization packages, to enable better re-use of such packages across agencies” — so an agency buying the same CSO could just go pull the docs that were already submitted, review them themselves, and issue an authorization.

FedRAMP continues to evolve. As of June 2024, FedRAMP’s governance has changed again, with the FedRAMP Board replacing the JAB for FedRAMP governance. All authorizations are now conducted by individual agencies — and all of these authorizations are treated exactly the same. Continuous monitoring (ConMon) of the CSO’s controls is a multi-agency responsibility, with a lead agency for each primarily responsible.

Suppose you want to get FedRAMP. What do you go through? You as a CSP first prepare your systems by implementing required security controls, then you undergo rigorous assessment by an auditor, and finally you submit comprehensive documentation for review. Mostly, it’s a lot of paperwork.

There are three designations in the FedRAMP authorization process:

FedRAMP Ready, which is when an independent Third Party Assessment Organization (3PAO) has attested to your security controls, and completed a FedRAMP Readiness Assessment Report,
FedRAMP In Process, which is when your application is actively being reviewed, and
FedRAMP Authorized, which is the authorization has been provided, it’s on the Marketplace, and now available for re-use by other agencies.

Hypotheses

I have a few hypotheses about FedRAMP authorizations, which I wouldn’t say are out of the norm with others in the industry. These hypotheses (biases?) aren’t based in data, though.

Given I’m analyzing the data in more detail, I thought I’d write these down, and check how I did at the end. (If you want, take the chance to note down your own assumptions before reading ahead.)

Hypothesis: There has been an increase in FedRAMP authorizations in the past 3-5 years, but that’s been mostly re-uses, not authorizations of new CSOs. That is, it’s new agencies authorizing and buying products that are already FedRAMP authorized, not new products getting FedRAMP authorized. The introduction of a LI-SaaS impact level may also have helped contribute to the increase.
Hypothesis: Microsoft and Amazon sell the most to the federal government. Microsoft is so good at government sales they’re getting anti-trusted.
Hypothesis: The main agencies buying cloud software are… I don’t know? I don’t have a good understanding of how typical government agencies buy, and how open — or eager — they are to using newer technology. So maybe I’ll go with the Department of Defense at the top of the list.
Hypothesis: There are probably <10 assessors who do the vast majority of the authorizations. I’m guessing these are small boutique firms — there’s probably not enough volume or specialized expertise for someone like an Accenture.
Hypothesis: FedRAMP reflects existing federal spending moving from on-prem to cloud services within a vendor, not the adoption of new vendors. That is, I don’t think that obtaining FedRAMP authorization leads to net new business for most companies. I also don’t think I’ll have the data to prove this either way.

With that, let’s dig in! Data time 😎

Getting the data

So you know how I said that you could “just look at the data”. Well, kind of.

I hadn’t been on the FedRAMP Marketplace in a long time. I knew there was a table of information on authorizations, but was pleasantly surprised to find a nice “Export CSV Data” button. Yes!!!

Good news: this gives you a dump of FedRAMP data, at that moment in time.

Bad news: it doesn’t have absolutely all of the information that’s available on the site, including a bunch of nice things I was hoping to analyze:

Impact level: whether a CSO has been authorized at High, Medium, or Low isn’t included in the data set. What? But it’s right there on the marketplace landing page. That page is kind of annoying though because it isn’t searchable because the company names are logo images, not text — though there is a little search bar for the table itself.
Deployment model: whether a CSO is deployed on a public cloud, government cloud or hybrid. This is available in the marketplace table (and on the CSO detail page).
Authorization status / Authorization details: when the CSO completed the FedRAMP authorization process, including when it was FedRAMP Ready, In Process, and Authorized. This is available on the CSO detail page.
Independent assessor: which 3PAO completed the Readiness Assessment. This is also available on the CSO detail page.
Dependent products: which CSO’s FedRAMP-ness is a dependency for other CSOs. This is also available on the CSO detail page.

I wish I could say I used one of the newfangled web scrapers to get the data I was missing. I tried two of them and despite a permissive robots.txt for the FedRAMP site, couldn’t get anything usable. After calling in some reinforcements, it seems that after you go to a specific CSO page, you’ll need to refresh it to get the data, which trips up these scrapers. My original dataset, including impact levels, authorization status, deployment model, independent assessors, and dependencies, is from January 25 2025. I collected further data on authorization timeline on March 17 2025. In that timeframe, there were 31 newly added CSOs and a total of 158 new ATOs, which are missed in this analysis. There were 24 newly authorized CSOs, whose authorization data is used in measuring timelines but not used for reporting on status.

The analyses below disregard data from 2025 when looking at years, and also disregards data that was clearly incorrect — like authorizations that happened before FedRAMP was established, and in the future. I kept all incorrect data in as long as it wasn’t relevant to the question being asked.

Which is all to say, my dataset and analysis is not perfect. If you want to run an analysis yourself, grab the latest csv from FedRAMP. For what’s missing, here’s the code to scrape the FedRAMP site, or get the JSON from 18F (which I only found later 😅).

Analyses

How have FedRAMP authorizations changed over time?

The use of FedRAMP over time has that nice hockey stick appearance that is the envy of all startups.

But as we know, growth is what matters. Let’s answer the question that started all of this first: has the rate of FedRAMP authorizations increased?

So… not really? The rate of authorizations notably increased from 2018 to 2019, but since then has been relatively flat.

This is just initial authorizations, though. Let’s take into account authorization re-uses.

Aha! Our work here is done. There has been a significant increase in re-uses in the past several years — which contributes to my perception that FedRAMP is just so hot right now. Also of note: some agencies have new uses for the same CSO, sometimes multiple times in the same year.

Another part of the hypothesis had been that maybe there are more FedRAMP authorizations for lower impact levels — is that also a contributing factor?

Partly — the introduction of LI-SaaS in August 2017 led to a small increase, but the real difference here was that increase in re-uses. This is likely due to a simpler, less bottlenecked process for re-uses, as well as increased demand for CSOs from federally agencies generally: due to the pandemic and how widespread and popular SaaS tools have become.

Here’s a fun little animation of the uses of CSOs, with their aggregate authorizations and re-uses over time.

Will FedRAMP authorizations keep going up?

Although a lot of CSOs are already authorized, how many are waiting for authorization?

Wow, we’re waiting on a lot more CSOs coming down the pipe.

Although I stated earlier that there were three main phases to authorization: Ready, In Process, and Authorized — from FedRAMP’s own docs — the marketplace listings for newer products split “In Process” into “In Process: Review” and “In Process: Finalization”.

How long does a CSO typically take to go through the FedRAMP process, then?

This is some of the messiest data: a lot of dates are missing, and there are inconsistent states that you think shouldn’t be possible — like CSOs that are authorized before starting the process. However, looking at news coverage, these mistakes are seemingly accurate. There were so many CSOs that were In Process before being Ready that this must be a valid state.

To go from Ready to In Process (Review) takes a median of 213 days, about seven months.

To go from In Process (Review) to Authorized takes a median of 334 days — eleven months, or almost a year.

Longer times aren’t associated with higher Impact Levels, either — even though LI-SaaS applications go quickly, they take longer to get started.

The process has gotten slightly longer in recent years — likely due to high demand — but not significantly worse.

We can do some nice little predicting and extrapolate where we currently are for all the CSOs currently in review — some of these should have been authorized by now, but there are a huge number waiting to be authorized.

So maybe part of the conversation around FedRAMP is because of this: so many CSOs have submitted products for review, and are now waiting. We might be at the beginning of a big upswing. There’s no reason to think that trend will slow down.

Who’s selling?

Which CSPs are benefitting the most from selling to government agencies? It’s what we expected: Microsoft and Amazon have the most uses across their CSOs, with ServiceNow, Zscaler, and Salesforce rounding out the top 5.

Note that GitHub is listed separately. GitHub Enterprise Cloud was In Process when it was acquired by Microsoft, and now has 19 uses.

In fact, Microsoft and Amazon are so popular with FedRAMP that they each have their own special authorization codes AGENCYAMAZONNEW and MSO365MT. They’re not the only ones — these seem to have been primarily used for some of the first FedRAMP authorized products: AINS (AGENCYHUDSAAS) and Tyler (SOCRATA), as well as other agencies: USDA (AGENCYNITCIAAS) and Workplace.gov (AGENCYWC2) — yes, an agency can use another agency’s services, which also need to be authorized. 18F also offered its services to other agencies with Cloud.gov.

There are also several CSPs I didn’t recognize:

AINS dba OPEXUS sells you a way to deal with FOIA requests (makes sense)
Granicus is the government’s marketing automation platform (i.e. Hubspot), which is much less creepy than the “citizen experience platform” they brand themselves as
Tyler is the government’s case management and business process management solution, with a “near-monopoly” on “glitchy” software products that “aren’t perfect” and — among other things — have ended up “causing longer jail stays”

… and many more excellent software companies.

Seven CSPs make up the top 25% of all FedRAMP uses.

Adobe and Oracle each have 8(!) CSOs. That’s a lot of paperwork.

It’s important not to forget the long tail here — 85 CSPs, or 29% of all providers that have a FedRAMP authorization, only have a single use.

In 2017, FedRAMP said that “Moderate Impact systems accounts for nearly 80% of CSP applications that receive FedRAMP authorization” — indeed, it’s 70-80% now, and that’s also true for applications which are In Process or Ready.

Who’s buying?

The top agencies purchasing CSOs are the Department of Health, the Department of Energy, and the Department of Commerce.

These top departments are buying a lot of CSOs — with the top three agencies making up a quarter of all FedRAMP use, and the top eight altogether being over half of all FedRAMP use.

The vast majority of agencies, however, still barely use FedRAMP, with most using far fewer CSOs.

Although some uses are at the agency level — e.g., NASA is an agency which purchases 22 CSOs, including all three major cloud providers, Slack, Salesforce, and Zoom — for some agencies, uses can be both at the agency- and at the sub-agency level, so we can dig a bit deeper.

Looking at the fifteen federal executive departments, most usage is still at the department level, but there are some sub-agencies that are unexpectedly high in their FedRAMP usage, such as Argonne National Laboratory.

The top sub-agencies overall by number of FedRAMP uses are:

Defense Information Systems Agency, in the Department of Defense, with 62 uses
Internal Revenue Service, in the Department of the Treasury, with 50 uses
Federal Emergency Management Agency, in the Department of Homeland Security, with 47 uses

FedRAMP usage can be very varied. It’s weirdly used in some places of the government, but not necessarily where I expected. This is probably far more based on local culture, risk tolerance, and workflow needs; although data classification prevents cloud services from widespread use in some agencies, such as the Department of Defense.

Who is actually using CSOs with High impact? In 2017, about half of FedRAMP High use was the Department of Defense and the Department of Veterans Affairs.

It’s changed significantly since then: although overall, the top agencies by FedRAMP use are the Department of Health, Energy, and Commerce; the Departments of the Treasury, Homeland Security, and Defense are much more heavily represented in FedRAMP High uses. Veterans Affairs is mostly using FedRAMP Medium, and doesn’t break into the top agencies using High.

If you’re looking at getting FedRAMP authorization, you now know which agencies you’re the most likely to be selling to.

Are these the same agencies, however, that are also the most likely to help get go through the initial authorization? Unsurprisingly, generally, yes, with the Department of Health, Veterans Affairs, and Energy being the most common for initial authorizations (excluding the legacy JAB authorizations).

However, if you’re selling your product to one agency — and only one agency — the most common is the Department of Veterans Affairs. The Department of Veterans Affairs has 23 CSOs with no re-use by any other agency. This shouldn’t be surprising since the VA is largely a medical provider — these are mostly healthcare related, and include classic hits like Abbott’s LibreView for US Government, and CirrusMD Virtual Health Chat for Government.

Who (else) is profiting?

Microsoft and Amazon are clearly the winners when it comes to selling to federal government agencies. But like with any market, there is another set of organizations profiting here: the auditors.

There are only 45 of them. Well actually, there are 45 recognized 3PAOs, but only 30 of them have ever completed an assessment for a CSO that’s now authorized. The top auditors all have a significant list of products under their belt: Coalfire does Amazon, IBM, Google, and Oracle; Schellman covers the next tier of tech companies like Figma, MongoDB, Qualtrics, and Scale AI; Microsoft uses Kratos.

Schellman and Coalfire each do more than 20% of FedRAMP readiness assessments. Together, the top three of Schellman, Coalfire, and A-LIGN complete 57% of all FedRAMP readiness assessments.

If you’re evaluating the top auditors, you should also take into account how quickly those assessments have turned into authorizations in the past.

But, there is also someone else profiting, or maybe, double dipping: the other CSPs who are providing CSOs are also making money from new CSOs, since if you’re not hosting your own infrastructure for your CSO, then it’s built on another CSO. “When your software sits on a FedRAMP Authorized infrastructure, it will inherit controls from that authorized system”, in turn making it easier for you to obtain FedRAMP authorization.

Here, Amazon really wins again — but it’s also where we see how a common tech stack in tech companies developing CSOs leads to a common tech stack in the government. If you’re selling to a federal agency, they probably want to sign in with Okta.

Here’s a graph if you want to explore dependencies. (I couldn’t find a way to make it more digestible, sorry.) There also seems to be a strategy to architect and obtain authorization for a Medium impact CSO, and then build your High impact CSO to depend on the Medium impact one, presumably to only have to do the diff of the paperwork?

The median government tech stack

Say you’re a median government agency: you use 10 CSOs (ignoring those agencies who use zero). You’re probably using:

AWS US East/West, AWS GovCloud, Azure Commercial Cloud or Azure Government for compute
ServiceNow Government Community Cloud for IT helpdesk
Office 365 or Box for word processing, spreadsheets, and slides
Salesforce Government Cloud Plus for CRM
Zoom for Government or Webex for Government for video conferencing
AINS for FOIA requests
Okta IDaaS Regulated Cloud for SSO

… and everything else is hosted on-prem!

Although you can see total CSO use and total agency use in the tables on the FedRAMP marketplace, what I really wanted was a table I could easily search to see who uses what. Here’s that table for you 👍 (Are you finding the same agency using the same CSO multiple times? Yes.)

Conclusion

Let’s review our original hypotheses and recap.

Hypothesis: There has been an increase in FedRAMP authorizations in the past 3-5 years, but that’s been mostly re-uses, not authorizations of new CSOs.

Yes. But we’re also at peak authorizations in process. There are more re-uses, some more CSOs entering the market with LI-SaaS, and also a lot of new CSOs coming up soon.

Hypothesis: Microsoft and Amazon sell the most to the federal government.

Yes — and that’s even true when GitHub is counted separately.

Hypothesis: The main agencies buying cloud software are… I don’t know?

The answer is the Department of Health and Human Services. The agency with the most unique purchases is the Department of Veterans Affairs.

Hypothesis: There are probably <10 assessors who do the vast majority of the authorizations.

Way more concentrated than expected. The top three auditors complete 57% of all authorized CSOs.

Hypothesis: FedRAMP reflects existing federal spending moving from on-prem to cloud services within a vendor, not the adoption of new vendors.

I didn’t think I’d get the data to show this either way, and I don’t! The top CSOs in government are generally some of the top tech companies. Although Microsoft might be seeing its buyers move from on-prem to O365, given that Salesforce is the original SaaS, I’m guessing it’s supplanting whatever on-prem CRM the government had. So, there is opportunity for SaaS vendors in government, but it’s still predominantly the well established SaaS vendors — or specialized vendors targeting only government requirements and sales.

If you’re considering getting FedRAMP authorization for your product, I’d focus first on understanding and executing on government sales, generally. Once you understand what you’re getting into, I’d take the time to think through and understand:

Which agency will work with you to get FedRAMP authorization?
Which other agencies are realistically going to buy your product?
Which 3PAO is right for you, and how long do they expect to take?

I’d also do a sneaky look at the public dataset to see if my main competitor is pursuing FedRAMP authorization, and at what step they are.

I’m not a data scientist, and this whole project was a great excuse for me to play around with government data and use the tools made accessible to visualize that data, like Datawrapper and Flourish.

Back to why I spent countless hours in pivot tables, is FedRAMP more of a thing lately? Why are FedRAMP authorizations and uses on the rise? There are a few possible reasons:

There is a drive from the government to move towards more modern technology, and vendors are responding by getting FedRAMP.
SaaS providers see a big opportunity in government that’s worth the upfront price tag.
The process to get FedRAMP authorization — and re-use products already that are already authorized — has gotten easier.

It’s probably a little bit of all of the above.

Now that I’m much more familiar with FedRAMP, it’s also clear that the government and the FedRAMP team have made a concerted effort to modernize FedRAMP. FedRAMP has documentation, a roadmap, and a changelog — we’re treating it like you might a tech product. The FedRAMP team is doing a really good job.

The FedRAMP “strategy for 2024 centered on tackling some of the root causes that have held FedRAMP back from being able to make the bigger changes needed to reduce the time and cost of the process and center FedRAMP around risk management.” They’re planning on publishing metrics on the program, and further developing machine-readable “digital authorization packages”.

I’d still love to see some of this data made easier to analyze, with what I had to collate above available in a single csv. It’s not like this data isn’t available, it’s just a bit of a pain to process — it could be even easier to explore.

FedRAMP is a complex and time-consuming process, but becoming increasingly more common — so if you’re planning to pursue it, at least now you know what you’re signing up for. Tell your VP Sales you have the data.

AI agent identity: it's just OAuth

Mon, 20 Jan 2025 08:00:00 GMT

I keep getting asked (by non-security folks) about how organizations should handle AI agent identity and permissions — I’m clearly not the only one. Security folks, though, aren’t really talking about this. The perception is that we need new solutions for authenticating and authorizing AI agents. Do we?

No, really, do we? I was talking to a friend of mine about this, and he made a really good point: if you were a developer building a random SaaS app, why would you prioritize making it easier for agents to use your app, instead of just building your own AI agent or assistant in your app? I’m embarrassed to say that I hadn’t stopped to consider if I agreed with the inherent assumptions present in the question.

I think that yes, SaaS apps will want to make it possible to manage agent authentication and authorization, but it might be a little while until we get there. There is definitely a desire from consumers — an agent will only become more useful as it’s more personalized, and able to take more actions on your behalf. But it might be a while until we get there on the app side: first, we’ll probably see apps try to build some AI functionality in-app first, and then maybe they’ll be overwhelmed with requests when individuals give agents access to their accounts directly. Eventually, though, I suspect it’ll look kind of like the adoption of Plaid: initial reticence that this will lead to more competition, then increased interest when that connectivity makes it easier to provide higher-value services.

So I think we will need a solution here. So why aren’t security folks talking about this? As with many other market-wide changes, the difficulty doesn’t lie in innovating new technology — most of what we need already exists in OAuth — but in adoption. The real challenge is that many organizations and products haven’t fully embraced and implemented OAuth, regardless of any interest in agents.

Why OAuth makes sense

When building an app that agents can use, we need a way to grant them limited, controlled access to resources. These automated agents need to access data on behalf of users or organizations, but with appropriate restrictions and the ability to revoke access. We need to be able to restrict their access — they need different levels of access for different tasks — and we need to be able to audit what they’re doing.

OAuth provides exactly this: a standardized way to delegate limited access to resources without sharing full credentials. It allows applications to request specific permissions (scopes) on behalf of users, with those users explicitly approving what access they’re granting.

Granular permission scopes in OAuth are perfect for agent access patterns. Instead of giving an agent full access to an API, you can grant specific read or write permissions to particular resources. These scopes can be composed as needed, allowing for precise control over the actions an agent can take.

OAuth also lets you handle the distinction between user-level and organizational-level permissions. When an agent needs access to a user’s calendar, that should be tied to that user’s identity. When it needs broader access to organizational resources like a shared knowledge base, that should be tied to an organization.

OAuth tokens are meant to expire (or at the very least, be revoked), unlike API keys which tend to live forever in your environment. Yes, you can also have long-lived OAuth refresh tokens, but the access token itself is meant to be short-lived. (Yes, some also live forever.) This natural life cycle means you don’t have to worry about persistent access.

OAuth makes sense for AI agents. Again, I’m not the only one who has come to this conclusion.

Implementing OAuth scopes for AI agents

Have apps that should have really not implemented OAuth support? Yes. The minimum bar these days is SSO — you’re lucky if you get OAuth, SCIM, or audit logs.

So let’s assume you’ve gone through the hype cycle and are now on the slope of enlightenment and decided that yes, your app will be adding support for agent identities. What would you need to do?

Start with your existing permission model

Actually, pause — before thinking about building agent-specific authorization in your application, take a step back and make sure you have solid authorization fundamentals, with clearly defined roles that map to actual user needs and workflows, with proper documentation of what permissions each role includes. These roles should follow a logical hierarchy — say, if an admin role inherits reader permissions, that pattern should be consistent throughout your system; or, an auditor role should have read access everywhere. You can also get more fine-grained and allow user-defined roles, which are groupings of permissions. (I wouldn’t suggest you start here, though: users will come up with weird groupings of permissions that will then be hard to debug, the roles won’t get updated when you add new permissions, and most of your users will have similar role needs so you should really give them a decent starting point. Once you’re a cloud provider with dozens of different products and hundreds of permissions, sure, allow user-defined roles.)

Instead of creating ‘AI agent’ permissions, fix your existing ones. If a global API key is the only granularity of permissions you have today, that’s your actual problem. You need separate read and write permissions for each action, and you need to clearly document what each permission allows.

Implementing organizational scopes

When developing OAuth scopes, they should ideally match your existing permission models — if you have reader and editor roles, you should have corresponding OAuth scopes. Don’t just make it one allows-everything scope like an API key — you don’t want users to be constantly over-provisioning access. Follow your existing role inheritance patterns. There’s no need to create a separate hierarchy just for OAuth.

A significant part of the work to support OAuth is also giving admins controls to not allow certain OAuth apps or scopes. This is the real value of using OAuth — admins will have the ability to easily set policies for their entire organization — right now, they have no idea who is bringing a personal OpenAI API key to work. Consider only letting admin users generate OAuth tokens with the most sensitive or permissive scopes.

In your audit logs, every action taken with an OAuth token should be logged with both the token identifier and the associated user or organizational context.

Where AI agents need special handling

While most permissions can map to existing patterns, there are some agent-specific considerations:

Rate limits serve different purposes. For human accounts, they’re a safety net in case of compromise. For agent tokens, they’re an essential control — you need separate limits to ensure automated access doesn’t impact human users who actually need to get work done. If my agent gets rate limited then, I, as a human, should still be able to use the service. You’re most likely already doing this when implementing OAuth, but it becomes particularly important for dealing with agents.
Cost controls matter more for automated access. Consider restricting operations that could incur significant costs to an organization if an agent goes wild, like expensive API calls or queries.

Still use API keys for organization access

We’ve been talking about supporting authorization for user-level activities, but you may also want to authorize agents at the admin-level to perform organization-wide duties. You could have the agent use OAuth to get authorization as a specific admin, but for long-term persistent access, this organization level set of duties is usually supported via API keys. You’ll still want to take into account some of the above considerations, like rate- or cost- limiting API keys separately.

Why this solution kind of sucks

Why isn’t this already possible? Because OAuth isn’t as widely supported and as visible as it should be. (Also, it’s not necessarily the easiest thing to implement.)

But building proper OAuth support isn’t just work for service providers — the flip side is that identity providers need to step up too. An admin should be able to see, for everyone in their organization, which users have granted OAuth access to their accounts and how that’s being used — but this list of connections is not often reviewed today, making those blocking controls even more important. (Check out your Google account connections to find apps you forgot you signed into 5 years ago.)

Where this solution fails completely

I’ve also completely ignored where agents are clearly heading — being able to perform operations as if they were you, on your device, like Claude MCP, Claude computer use or ChatGPT desktop. This is a much more annoying problem to solve authorization for.

Apple has taken the lead in improving privacy and security settings overall by giving you controls over which apps can say, find devices on your local network, but these are still pretty coarse-grained. It’s like when a Chrome extension asks to see and change all browser data. We have more permissions and controls than we did a few years ago, but these aren’t fine-grained enough for the world we’re about to be living in.

I don’t know what the right solution here is. It’s not a separate user account on your device, as that’s completely segmented from your user account, but you are trying to share data. It’s also not something like a Linux process user owner, where you can attribute a process to a specific user, since it’s not like only Claude is running Spotify; you are also running Spotify. So, here, we probably do need something new, even if it’s just more fine-grained permissions. Either your OS will need to give you a reasonable way to limit what a local agent does, or an app will need to give you a reasonable way to separate ‘who’ is taking an action, or both.

What bothers me the most about the on-device agent identity story is that it’s the lowest common denominator. You can design the perfect solution for letting an agent use an OAuth token to access a website as you, but that falls apart the second the agent is a stowaway in your browser session, and likely only detectable with some sketchy adtech behavioural analytics.

Start by building decent OAuth scopes

Forget about agent-specific solutions for now, and instead focus on implementing proper OAuth scopes in your product. Build clear, well-documented permission models, and create granular scopes that map to real use cases. The effort you put into improving your permission model will benefit both humans and agents alike.

Dear Santa, all I want for Christmas is better security tools

Tue, 24 Dec 2024 08:00:00 GMT

A whimsical holiday letter I imagine a CISO might write. Happy holidays!

Dear Santa,

My security team has been working incredibly hard this year — responding to incidents at 2am, reviewing dozens of product launches, and somehow still finding time to implement more robust authentication. (Have you considered implementing passkeys at the North Pole?)

Solutions, not problems

Santa, I know you’re already tracking countless security alerts at the North Pole, so you’ll understand this one: please help our vendors realize that we don’t need more tools highlighting problems. I don’t need another system telling me we have 437 critical vulnerabilities or that someone shared a doc with the wrong permissions. At this point, I could build a museum of monitoring tools, each showing me problems none of them can fix. What we need are tools that actually help us fix these issues. Send me an automated PR with the dependency update. Show me one-click options to adjust sharing settings. Help me solve problems, not just find more of them.

The real cost here is that every new alert without a solution creates more work for an already overwhelmed team. Finding issues is the easy part my team needs help with the harder challenge of fixing them efficiently at scale.

I wish I could have tools that understand our environment well enough to safely automate fixes while handling exceptions (big ask, I know). Tools that could analyze impact, suggest practical solutions, and help us implement them efficiently. That would let my team focus on the complex security challenges that really need human expertise, instead of drowning in alerts and routine fixes.

Modern developer experience

Santa, could you please help everyone understand that security engineers are engineers — they need APIs, not ancient artifacts. We need tools that work like modern developer tools, not like security tools from 2005. Having to click through nested menus to find basic configuration settings is just frustrating.

Security tools should have CLI support that isn’t just a wrapper around API calls, APIs that are actually documented, and UIs that make sense. And please give us reasonable defaults. If most of your customers are going to enable encryption at rest or set up expiring access, maybe start with those turned on.

The impact goes beyond just lost time — every minute my team spends fighting with clunky interfaces or re-implementing basic security controls is a minute they’re not spending on more valuable security work. Modern engineering teams automate as much as they can — my security teram wants to do the same. Our tools should help with that, not fight against it. When engineering teams can self-serve security with tools integrate naturally into their workflow, everyone wins.

Bring us security tools that are a joy to use — where configurations are version controlled, changes are integrated into our CI/CD workflow, and the API documentation is actually up to date. My team needs to focus on securing our systems, not reverse engineering how to use our security tools.

Platforms over point solutions

While we’re talking about consolidation — could we maybe get fewer standalone tools in our stockings this year? I’ll need a spreadsheet just to track all our security solutions. We don’t want another vulnerability scanner — that requires its own vendor assessment, another set of credentials to manage, and another training session to schedule.

We need platforms that reduce our operational overhead. Something that brings related capabilities together, simplifies our vendor relationships — and yes, shows up as one line item instead of twelve. Tools that understand they’re part of a larger security ecosystem, not an island unto themselves.

Beyond the obvious cost implications, every new tool means another procurement cycle, another security review, another set of processes to document. My team needs to focus on securing our organization, not becoming experts in twenty different tool interfaces.

Santa, bring us unified platforms that solve whole classes of problems. Tools that work together through standardized integrations instead of creating new silos. My team deserves better than spending their days building glue code between security tools.

Complete environment coverage

Santa, I have a confession: we’re not just running one cloud provider with identical VMs anymore. (Shocking, I know.) We’ve got Windows laptops, Mac workstations, Linux servers, containers, multiple clouds, SaaS tools, and even some systems that might be older than the elves. Our security tools need to handle this reality.

We need solutions that can actually work across our entire infrastructure — not just claim they do in the sales pitch. That means supporting every endpoint type we manage, understanding each cloud provider’s quirks, and yes, even dealing with those special snowflake legacy systems.

Security gaps aren’t just technical debt — they’re real risks. Every environment we can’t properly secure is a potential blind spot. Every platform we can’t monitor means more uncertainty and risk. Every “we don’t support that yet” means more custom tooling my team has to maintain.

Santa, bring us tools that truly understand our complex environments. Tools that handle the messy reality of enterprise infrastructure, not just the clean architecture diagrams from the sales deck. My team needs complete coverage, not just coverage of the parts that are easy to support.

Scale security impact

Finally, Santa, what my team really needs is the ability to scale our impact. Our engineering team keeps growing (which is great!), but unfortunately, my security team can’t grow at the same rate (which is… challenging). We want to scale our security engineer to developer ratio non-linearly — and we want tools that let us do this.

We need solutions that help automate routine work and enable developers to handle security tasks safely on their own. Give me guardrails that scale, not gates that fail. Self-service capabilities that don’t create more work for my team. Workflows that adjust and scale with our organization.

Security teams will never grow as fast as engineering teams — nor should they need to. Every manual review, every access request ticket, every “ask security” checkpoint creates friction that slows down the business. And more importantly, it burns out my team.

Santa, bring us tools that actually help us scale. Tools that make both security and development teams more effective, not just more busy. My team needs to be able to secure our growing organization without working nights and weekends to keep up.

Well, Santa, that’s my security team’s wish list this year. I know it’s a lot to ask, but we’ve been really good — we patched quickly after the xz fiasco, we’ve completed all the critical items from this year’s incident postmortems, and we’ve even managed to get engineering to use our approved CI/CD pipeline most of the time.

I promise we’ll leave out cookies for you — and only you. And yes, we’ve finally updated our incident response plan to account for your annual December 24th authentication exception.

My team deserves tools that solve problems instead of just finding them, that work like modern software instead of ancient artifacts, that cover our whole environment instead of tiny pieces, and that help us scale our impact across the organization. And if you could wrap all of that up in solutions that actually work together… well, that would be a Christmas miracle.

Here’s hoping for a more secure and less exhausting 2025.

Yours truly,
A wishful CISO

The road to zero trust is paved with good intentions

Thu, 12 Dec 2024 08:00:00 GMT

This blog post is a written version of a talk that Eric and Maya gave at NorthSec in 2022. You can also watch the recording and get the slides. This blog post is also cross-posted to Eric’s blog.

We put together the material for this post in 2022 during the peak of the zero trust hype cycle when RSA show floor booths, marketing materials, analyst reports, and even a US executive branch memorandum couldn’t stop talking about zero trust. Though the cycle’s moved on — to “AI-powered security” — it doesn’t feel like zero trust has necessarily progressed. Have we successfully adopted zero trust, and baked these principles into our industry? Or have we thrown a few identity aware proxies on the problem and called it a day?

A few years on, we still struggle to agree on what “zero trust” means, much less how to implement it. If you’re on a journey to zero trust, how far have you gotten in the past few years? And, how much further do you have to go?

What is a zero trust architecture?

A zero trust architecture is a security model that requires every user and device proving trustworthiness at every access attempt — essentially, treating every access as potentially untrusted until proven otherwise.

Traditional network architecture relied on a network perimeter to delineate between trusted and untrusted users: trusted employees inside a firewall, vs. untrusted parties outside of it. By moving to a zero trust architecture, the location of an individual, specifically, which network they are on, is no longer what determines whether the individual is trusted, but other context is used to determine whether they can access a given application. There is no longer such a thing as a privileged, physical, corporate network.

BeyondCorp, first introduced in a 2014 paper, is Google’s original, specific implementation from which the broader generalized set of principles for zero trust architecture emerged.

A zero trust architecture is made up of users, devices, access

A zero trust architecture asks how users gain access to corporate resources — for example, how a new sales manager on their Macbook might gain access to an internal wiki. Authorization considers the user and the device they’re using in order to make a determination about access:

Users: Who is connecting to your resources? Your users are your employees who need access to corp services. You need to figure out how they authenticate, what kinds of credentials they can use, and how you’re going to manage group membership.
Devices: Where are they connecting from? You need to decide what kinds of devices you’re okay with accessing your resources (a corporate issued laptop, a personal phone, etc.), and the minimum security of a device (device management, patch level, etc.). (See also Maya’s rant on device trust in zero trust architectures.) A user’s credential is only as secure as the device that that credential resides on.
Access: Should this user and device pair have access? The mechanics are the hard part here. A simple case like accessing an internal website might just require an SSO proxy with an allowlist of users, while SSHing to a prod machine will be more involved.

Note that we’ve intentionally left out networking — it’s not a critical component of a zero trust architecture, and not part of the access decision. Which network you’re on (including if you’re on the corporate VPN or not) isn’t used to make a decision — a decision is made entirely based on user and device.

The road to zero trust

To understand how far along you are in getting to a zero trust architecture, we’ve put together a maturity model. This is less of prescriptive guidance of what you should do when, and more of a realistic way to help you evaluate where you are today, and what’s left.

As your adoption improves, from left to right, your organization has more capabilities in terms of how it secures its users, devices, and access to applications:

Level 1: Inventory. You’re treating a VPN as the access control point for the applications on your network. You can enumerate users and devices — you’re using an SSO to inventory users, and have a manual way to list devices. These are table stakes IT and security capabilities.
Level 2: Management. You have per-service authorization that lets you segment access to specific applications, for example by using a proxy. You can measure most and enforce some security controls. You use a VPN for your network, and MDM to track and measure your devices, and your users use SSO and MFA, ideally security keys. This is where most security-focused enterprises are today.
Level 3: “Zero Trust”. You’re all in on and aligned to what the market believes is a zero trust architecture — and maybe you’ve even bought a solution that bills itself as zero trust! You can tier users and devices based on measurements, and enforce tiered access to applications based on those characteristics. This is what we typically see called “zero trust” in the market today.
Level 4: The long tail. This is what many aspire to, and think they’re being sold, but no one has yet to fully achieve. You want to be able to dynamically enforce risk-based access to applications. There’s a long tail of user, device, or access issues, such as SaaS apps, that make this very hard to get right today.

Let’s dive into more detail about what your organization’s controls look like at each level.

Level 1: Inventory

At this level, you have an inventory of your users and devices.

You have a way to enumerate users, most likely through a centralized identity provider and single-sign on (SSO). This is typically tied to your HR information system, so when a new employee joins, changes teams, or leaves, their identity can easily be updated. Unfortunately, a lot of SaaS tools charge you extra for the ability to use SSO with their application (the SSO tax); or sometimes even more for the ability to enforce SSO.

You have a device inventory (maybe a spreadsheet maintained by IT), and some minimal management of devices, such as network credentials. This can be complicated if you allow employees to access resources on their personal devices, have a mix of operating systems, or need to support mobile devices.

You use a flat, traditional network, where if you’re on the network, you’re trusted — just like with a traditional VPN. You know and ideally can control which users and devices can access the network, even if provisioning is manual. But, you’re not segmenting users and devices within your network.

If you’re only dealing with corporate devices, and only self-hosted applications, a VPN plus SSO is a good initial set of tools to limit access to your internal applications.

Level 2: Management

At this level, you can measure most, and enforce some, security controls. You’re moving beyond basic enumeration of your users and devices, and into hardening.

Your users have more secure authentication methods — specifically, hardware security tokens and WebAuthn. (Please, just use security keys.) You have some differentiation between users based on properties (usually group membership) to allow for effective role-based access control. This ties into your HR system to get updated information on a user’s role. You are also able to create groups based on organizational hierarchy (e.g., everyone in sales).

Your devices run a mobile device management (MDM) solution or similar capability, and you can enforce basic actions like remote wipe. You also have a way to measure properties such as OS patch level. Open source tools like osquery make this easy. Devices are identifiable through per-device credentials to tie them back to inventory during access decisions.

You’ve moved from a binary access decision based on whether an individual is in your organization, to per-application authorization with role-based controls. It’s easier to maintain these controls if they’re centralized, rather than built into each internal application separately — which typically means an L7 proxy for browser-based traffic, and a different strategy, like a jumpbox, for SSH-based developer traffic.

Level 3: “Zero Trust”

At this level, you can enforce access based on device characteristics.

Beyond securing only user logins, you’re also taking into account derived credentials like SSH keys, browser cookies and access tokens, which are often less well protected than a user login. A strong login with SSO and a security key isn’t as useful if the user then generates a powerful, never-expiring API token that ends up in a git repo. You’re limiting potential unauthorized access with shorter session policies for Cloud CLIs, and restrictions on OAuth applications on corporate services.

In addition to measuring when devices are out of policy, you enforce access based on these characteristics. So, you are requiring patching of a device for access to your resources, or forcing periodic reboots of devices. The issuance of device credentials are authenticated using a hardware-based identity: a TPM or a platform-specific implementation like Android’s Strongbox, or platform APIs like Apple’s Managed Device Attestation.

And finally, you move from having access solely based on the user, to tiered, or differentiated, access. Tiered access is based on device state: a personal vs. a corporate device, patch level, and compliance with security configuration requirements. Individual devices can “lose trust” and access rights based on inventory properties, like failing to apply an update timely.

You’re really doing zero trust now.

Level 4: The long tail

At this level, you can dynamically enforce dynamic, risk-based access to all of your applications, for all of your users, on all of your devices. For example, you can change access requirements on the fly based on risk, like requiring users to re-authenticate.

— reality check —

That’s not real.

This is where it really goes to shit. There’s a long tail of things that are either hard to get right or not yet solved, which make it effectively impossible for a normal organization to completely adopt a zero trust architecture: SaaS applications, truly risk-based access, device state, and all the random devices on your network. There are always exceptions in security — so the problem becomes, how do we deal with these exceptions?

SaaS applications

Applications that you can’t put behind a proxy are hard to control access to. How do you put Salesforce behind a proxy? SaaS apps generally only provide SSO as a means for delegating access to your company. If you’re lucky, there might be an API for syncing groups for role based authorization. Third-party device authentication is effectively unsupported by the industry.

One workaround for dealing with SaaS apps is to peer them to your network, so that ingress traffic to the app is only allowlisted for your corporate network’s IP ranges. So for an employee to access an app like Workday, they need to pass their traffic through your corporate network, out the other end, and on to the application.

In addition to being slow, this solution is a step backwards from what we’re doing with zero trust — it assumes your network is flat, and if someone is on your network, then can now access this application.

Risk-based access

The dream is to have nuanced authorization decisions based on risk. The failure mode is to overfit the model, and fall to the temptation of having too many tiers. If every application in your environment has its own set of requirements, it’s hard to standardize and even harder to debug why someone may or may not have access to an application.

Ever more fine-grained requirements will demand more of your access infrastructure. A bug tracker will need per-bug permissions. An admin panel may need just-in-time authorizations to permit support staff to debug the specific account they’re assigned to.

And, the more you rely on specific measurements, the more the lack of formalized APIs becomes apparent — did half of your employees really fail to apply a patch, or did macOS tweak its naming convention again?

Device state

Device state is useful for making an access decision — however, it’s self-reported by the device. When your device has been compromised, it starts lying to you. Trusted boot and device state attestation are meant to help you detect when this has happened — these work much better for closed ecosystems (like macOS), and are much more complex for open ecosystems with multiple vendors (like Windows).

Device state goes completely out the window as soon as you have to provide access to devices managed by another organization. If you’re large enough, you may be able to require use of a managed device by the vendors.

Network devices

Your corporate network has a lot of legacy devices that don’t meet your Level 3 security controls — like printers. Printers don’t have secure boot. Your MDM probably can’t manage your printer. The easiest way to set up your printer is with a fixed IP address. It’s still a device on the network, and you want to be able to manage access to it in the same way as my other network devices, but you can’t. Existing networks with legacy devices will have a harder time here — maybe if you’re at a startup, and you’re trying to adopt a zero trust architecture, you can just not have a printer.

So, ignoring our long tail of complications, what would the ultimate level of attaining zero trust look like?

— back to regularly scheduled content —

You have a decent way to manage users and user access to SaaS applications, and correlate logs for when users accessed those applications. There are only really two partial solutions today: either host everything yourself, which you see becoming more common with larger infrastructure tools with on prem solutions; or just be Google or Microsoft and use your single sign-on to access everything.

In terms of legacy devices, your zero trust architecture includes all of the devices you have in your corporate network — because they’re all a point of entry to the network. Usually, the easiest way to address this long tail of devices is to move them off the corporate network, instead of making sure they meet the requirements they need to be on it.

And lastly, with all of that information about users and devices, you can make real-time access decisions that aren’t just rule-based, but that change based on what you know, quickly enough that the user doesn’t notice or get frustrated. This isn’t easy.

Zero trust is a continuous journey

The wrong takeaway of this blog post is to treat the levels like a checklist — but, if that’s what you’re looking for, and you’re at the very beginning of your zero trust journey, take this as directional guidance on where you’re headed.

The reality is that zero trust is never “done” — it’s a continuous journey (and the friends you make along the way). You should consider what’s right for the risk your organization is facing — it might not be worth it for you to reach the top level. Focus on the core components of a zero trust architecture — users, devices, and access — and what you can do to improve in each area, regardless of where you’re starting.

What sucks in security? Research findings from 50+ security leaders

Tue, 10 Dec 2024 08:00:00 GMT

This blog post is cross-posted on tl;dr sec and on Maya’s blog. This is a version of a talk that Maya gave at October’s SnooSec meetup — get the slides.

Throughout the fall of 2024, I interviewed 57 security leaders to understand their biggest pain points in our industry, which is to say, I asked them “What sucks in security?”. These weren’t just theoretical discussions — I wanted to know what is currently challenging for their teams and where they are choosing to invest their limited resources, as I investigate how I can best contribute to security — but they were sometimes bordering on CISO therapy sessions.

In each discussion, we covered their role, team structure, infrastructure choices, and priorities. But most importantly, I asked them where they are encountering problems in their security programs. Every conversation was different: I didn’t follow a perfect script and ask the same questions every time. So, the data I collected is often incomplete — it’s a great indication of broad trends in the industry, but not a perfect data set.

tl;dr: The top technology pain points are:

Ticket-based and inconsistent access management,
Disparate vulnerability prioritization and remediation workflows,
And obtaining and using SaaS logs.

A proliferation of vendors, underclear ownership of services and assets, and the difficulty of successfully communicating and explaining security risks exacerbate other security problems.

Demographics

Through warm intros, I connected with primarily CISOs at tech-forward companies with 500 to 5,000 employees in the United States:

I spoke to 29 CISOs, but also 25 other security leaders and practitioners responsible for some aspect of security in their organization, e.g., head of Cloud Security or head of SecOps. The remaining 3 interviewees were those who were the closest equivalent to a security leader in their organization, including a CTO, VP Eng, and CEO.
Their organizations ranged from 30-person startups to 240,000 employee multinationals, with a median of 1,525 employees.
Their security teams ranged from 1 to 300 people, with a median of 14. (The data is imperfect, as I didn’t always ask this question.) This means that the security team was from 0.7% to 7.5% of total employees in the organization.

There was definitely a bimodal distribution of companies I talked to, possibly reflecting the current market reality we’re in — at the first peak, a set of security-forward startups with a handful of security folks, and at at the second peak, a set of larger companies with a few thousand employees and a security organization with multiple teams.

The security team’s responsibilities sometimes but not always included IT. They usually included privacy engineering, where the organization had this function. And I observed a trend to move towards having the Data or BI team report to the CISO — an evolution of the role into more of a cross-functional platform team (separate from DevOps or infrastructure), rather than just security.

In addition to the data being incomplete, it is inherently biased — these are folks I was personally connected to, who are generally more security-minded than average, and who are generally at smaller, more experimental companies. This by no means is what the entire industry is experiencing, but can be thought of as a bellwether for where we might be headed.

Top security issues

I went through all of my conversation notes, and picked out the top complaints from each leader I spoke to, and then took the logical next step of making a word cloud. This is the first (and probably only) time I’ve made a word cloud for its intended purpose! Bask in its glory. (I feel bad for the CISO whose top issue was “team morale” — I hope your team is feeling better now.)

Looking at the themes from the top unprompted complaints from each CISO, three issues emerged consistently across conversations: access management, vulnerability management, and SaaS logs. Let’s see that chart again.

Access management

I can’t provision appropriately scoped access, so my team is overloaded with access request tickets.

Access management remains a critical challenge, with teams overwhelmed by access request tickets. Entitlement changes are highly manual, requiring approvals for even routine changes. The team reviewing these access requests, typically IT, lacks sufficient context to make informed decisions — the confused deputy problem.

One security leader described having 700+ access tickets a month, which are only successfully processed thanks to external IT support. Another CISO lamented that the trend to improving access ticket “approval is a pressure relief valve,” that has made the process of reviewing access requests faster, but hasn’t actually improved the underlying difficulties they have with access management.

Entitlements in most organizations have grown organically, which means that “identity is a whole mess.” Permissions are often granted ad hoc — one-off permission requests, teammates’ permissions cloned, and brash overprovisioning in order to quickly unblock the business — so they need constant tweaking. This issue continues to persist because, for the most part, we don’t know what someone should have access to.

This is made more complicated by the separation between corporate (corp) and production (prod) access controls, and the tension between the IT and security teams. Since IT typically manages corp access, and security typically manages prod access — but the security team doesn’t want the IT team to be able to grant access to prod — most organizations have multiple authorization systems that they have to manage. “You’re never going to be able to centralize all of your identity in one system.”

Vulnerability management

I have too many vulnerabilities to patch, so I need to prioritize these and ensure they are handled appropriately.

Security teams are overwhelmed with vulnerabilities. Teams are moving to minimal container base images (such as from Chainguard) and VM golden images, as well as forcing ongoing redeployments of containers or rebuilds of VMs in order to reduce a significant volume of vulnerabilities. In addition to this, in order to “eradicate types of vulnerability”, security leaders are partnering with their Eng leads to reduce the number of languages they use — reducing the ecosystems they need to support for scanning, as well as moving to memory-safe languages like Rust.

Despite undertaking these broad reductions in the volume of vulnerabilities, teams are still overwhelmed — some of these efforts might take years to realize. They’re looking to prioritize their vulnerabilities based on reachability, business context, and exploitability. For many security leaders, “reachability is a problem” — they want to know, “is the code used anywhere at all?” Tools like Zafran and Vulcan aim to help with exposure management, but no tool has wide deployment yet. It’s really hard to get business context without having sufficient reach, so point solutions fail where Wiz excels: being able to see where a vulnerability is in their environment, and more important, in what critical applications.

The challenge of dealing with the growing volume of vulnerabilities is made worse by disparate scanners and disjointed workflows. Many scanners are point solutions, so organizations need to deploy multiple tools to cover their entire environment, across more than just AppSec (say, an old version of Adobe Acrobat on a server). The lack of integrated tooling creates overhead, with teams juggling notifications across email, GitHub, and Slack, depending on the tool.

“The entire lifecycle of managing vulnerabilities feels off to me,” noted one participant: they don’t have a reasonable process to ingest, dedupe, prioritize and assign vulns at scale, across their environment, cross-functionally. Finding bugs is a technical problem, fixing them is a human problem. Back in the day, this workflow might have been JIRA and measured MTTR, but most participants I talked to didn’t have a ticketing system and were tracking remediation in good ol’ spreadsheets. (Who doesn’t love a spreadsheet?) The push to move to a more organized solution comes from the need to meet SLAs for handling vulns under FedRAMP control RA-5d: to remediate high vulnerabilities within 30 days. “Because we have FedRAMP, vulnerability management is a big thing.”

The industry is ripe for change. As one security leader observed, “We are at the point for vulnerability management that we were in 2010 with EDR.”

SaaS logs

When there’s an incident, I can’t easily pull the logs I need to find out what happened in my SaaS environments.

Investigating incidents in SaaS environments is hampered by a fundamental lack of visibility. Organizations have an exploding amount of SaaS apps. At the long tail, many SaaS providers don’t offer audit logs at all; and at the fat part of the tail, put them behind paywalls. Even when logs are available, they might be incomplete, like missing login events.

“A lot of SaaS solutions are not set up in a way where their logs are useful for monitoring” and vendors “don’t have logs like I want”. Furthermore, the lack of standardization across tools creates ingestion challenges. “Having a common SaaS format would be tremendous,” explained one participant. Every new SaaS tool the business buys requires custom work to fetch logs, normalize them, and write alerts.

The inability to stream logs, or even access them programmatically, rather than through web portals, makes real-time monitoring and analysis difficult. The delay to get logs is painful. I heard of someone needing to log in to Stripe to manually look at logs in their web portal, another person waiting 4 hours to fetch the relevant logs from Slack, another waiting 2 days after emailing Notion support to get logs related to a potential employee issue, and yet another emailing GitHub for logs — who was thankfully able to provide some support even though their pricing plan didn’t include those logs.

I suspect more leaders would have brought this issue up, but hadn’t yet felt the pain — this pain point is only really felt by security leaders who have had to deal with incidents involving their SaaS tools.

Cultural complications

Beyond the technical problems, there were several organizational challenges that were force multipliers: they compounded all other issues and made security even harder to deal with. The top complications were vendor sprawl, ownership, and explainability.

Vendor sprawl

I have too many vendors, and they don’t always work well together.

The security vendor market has become overwhelming, with too many similar tools and confusing marketing. When I asked what sucks in security, one CISO bluntly and immediately answered, “I fucking hate vendors.” (As someone who now emails you to ask for your time… I’m sorry.)

“There’s just so many vendors” in the security space. CISOs are “inundated” as vendors “just keep trying to sell me stuff.” It’s difficult to tell vendors apart from their marketing, and getting a working solution requires stringing together multiple point solutions, each being sold for a few dollars a seat — which quickly adds up.

As you adopt more tools, it’s not just costs that accumulate, but risks as well. Every new vendor is a new security risk that security teams are insufficiently equipped to assess — “vendor security is just a complete joke.” One leader saw startup tools as particularly risky: since “I have to take on their security debt… in order for me to even consider a vendor… [I need] their CEO’s phone.”

The proliferation of vendors in an organization also means it’s hard to get an overview of security.

As one CISO shared, “my biggest problem, bar none, is I have no single pane of glass.” Reporting is ad hoc, and there is no single place to go to understand overall security posture. “Everyone is doing one slice of the pie [and] everyone has a dashboard,” but they just want one.

And the proliferation of vendors in the industry means that everyone’s stack is just a little bit different, requiring integration work to get tools working in each environment. The need to invest in custom integrations makes teams increasingly hesitant to adopt new solutions, as noted by another participant: “I can’t waste time on integrating a new vendor.” This also explains why the security industry seems to be the source of connection tools like Tines, Merge and Leen, which act as the missing glue between tools.

Ownership

I can’t keep track of who owns our ever-expanding set of services, assets, and apps.

Tracking ownership of services, assets, and applications has become increasingly complex. “It’s quite social and messy… more gardening than construction,” as one participant described it. Missing service catalogs, incomplete asset inventories, and unclear SaaS application ownership create operational friction.

Most organizations (of the size I talked to) don’t have a service catalogue — but if they do, it’s probably Backstage. They don’t know which team owns a service, which makes it hard to know who to ask or assign the job of applying a patch, fixing a vulnerability, or knowing who to pull in during an incident. “One of our biggest challenges is attribution: who owns that thing?… This isn’t just a security problem.” It also makes maintenance hard — if someone leaves, no one wants to touch a system for fear of breaking it. “Some systems didn’t have an owner… nobody wanted to mess with them since they’d been around for so long.”

“A lot of people think asset management is solved,” but it’s not comprehensive enough to cover the new kinds of resources we deal with today. Most organizations have an asset catalogue, but they consider it incomplete: it covers end user devices, and maybe servers, but not cloud resources like AWS accounts, or GitHub repos. That’s where Cloud Custodian and CODEOWNERS are helping organizations keep track.

Furthermore, since all you need to purchase a SaaS tool is a credit card, there are SaaS applications in each organization that are not centrally managed. It’s not always clear who manages these applications, or if they’re effectively abandoned. “We have quite a few tools that don’t have owners, that are scary.” There’s a general agreement here that CASB as a concept has failed, and we’re seeing the next gen of SaaS security tools like Nudge Security and Reco help address this.

Explainability

I need to be able to communicate security risks and requirements effectively.

Security leaders face ongoing challenges in communicating risks and requirements to their organizations. To be clear: this is… err… (by definition) the CISO’s job.

This starts at the board level, where teams struggle to find a common language for discussing risks and security investments with folks who don’t have any industry experience. Multiple CISOs talked about frameworks they had created, and had to recreate or change at each organization — we see this in the industry with teams trying to make comparisons to more familiar risks, like financial risk. Quantifying and explaining risk is “an area of extreme importance” but very low maturity in the industry.

One level down, security teams want to measure that their investments provide value and actually help improve security for their organization. As one CISO explained: “It’s hard to prove you’re getting what you think you’re getting.” Explaining their security priorities is difficult, with some teams mapping their priorities to the MITRE ATT&CK framework, an internal list of top threats, or collaborating with the business on risk prioritization. It’s exceedingly hard to define and move to something like probabilistic risk measurements. And, teams want to demonstrate that they’re safe enough, or justify further investment where needed: this is “where I spend most of my time.. how do we demonstrate we have a good security program?”

Asking for contributions from the business and justifying security controls that (might) add friction is another hurdle. So much of the work needed to make a security control effective isn’t done by the security team. To roll out any control, they need to really justify that it will have significant security impact, such as wipe out a whole class of issues. A CISO is worried about the perception from the CEO, who is “worried you’re adding friction to my barely profitable business.” CISOs are looking for strong evidence and data they can use to make decisions, and ways to communicate and measure minimum security requirements and security tech debt they ask their partner teams to take on.

Security tech stack

What the 80% use

My research revealed a security tech stack that was common across the majority of organizations (again, keep in mind the biased data set). I wasn’t explicitly seeking feedback on these vendors, but many folks volunteered their opinions:

Cloud providers: 😐 AWS, then 😐 GCP, then 🙁 Azure are all widely in use. Those with Azure have some regrets: we thought “Microsoft would help get us into [government] markets… it has totally not happened.”
Identity: 🙁 Okta and 😊 1Password. One participant said, “I hate Okta with all my soul… they’re the Microsoft of identity.” They didn’t even use Microsoft.
MDM: 🙁 Jamf and 🙁 Intune.
EDR: 😐 Crowdstrike, then 😐 SentinelOne, are both widely used. Crowdstrike was polarizing, with both strong positive and negative feedback.
CSPM: 😊 Wiz, which people love. One CISO commented that they trusted Wiz, since they “bailed me out.”
IaC: 😐 Terraform.
Observability: 😐 Datadog.
Data lake: 😐 Snowflake.

There are two spaces that stood out for not having any consistency: vulnerability scanners and SIEMs.

Vulnerability scanners

We have way too many vendor categories for vulnerabilities across code, containers, configs, and more: SAST, DAST, SCA, ASPM, CSPM, DSPM, … the list never ends (and changes every year). This is our own fault — we have no one to blame here but ourselves. One CISO lamented that they have “a whole portfolio” of scanners.

The main drivers of what tools to buy were value, cover-your-ass compliance, and cost — which helps explain why there are so many point solutions, and why users are so willing to replace them. I heard of at least a dozen different tools, including many open source tools. In rough order of most use: Snyk, Semgrep, Veracode, Tenable, SonarQube, CodeQL, Rapid7, AWS Inspector, Mend, Trivy, Clair, Socket, and Dependabot. There was serious consideration of open source tools: if I have to configure it anyways, should I do it myself? Is paying for this the best way to spend a significant chunk of my security budget?

Looking at the spread of scanners in most organizations, it’s not a surprise that vulnerability management — and especially a workflow for prioritizing and addressing vulnerabilities from multiple tools — as well as vendor sprawl are some of the top pain points I heard.

SIEMs

The trend in SIEMs is different: it’s organizations choosing to migrate off of Splunk over time, to a more modern SIEM like Panther, and then looking to migrate off of Panther to a lower cost solution if possible. “Nobody has come up with a Splunk replacement… there are a bunch of competitors, but no one seems to be able to knock them off the top of the stack.”

In rough order of use, SIEMs included: Splunk, Panther, Sumo Logic, ELK, RunReveal, Datadog, and Google Security Operations.

The main motivator to consider a migration is the cost of storage, and this move may be partly mitigated by choosing to stream a subset of logs only, filtering these with a tool like Cribl or LogSlash. The main requirement when looking for a new SIEM is query portability, to avoid getting locked into another SIEM. Frustration also comes from the work needed to ingest and normalize logs from various sources — there’s “no turnkey solution available,” with every solution requiring too much work to get set up. Again, not a surprise why SaaS logs and vendor sprawl are so painful.

Tools being built internally

I also asked folks what they were building or have built internally. Security teams have very limited engineering resources, so if they’re choosing to build something, it must truly be a priority, and something they can’t buy.

~20% of teams built a security data lake. This is an operational data store for security and compliance, including osquery logs from corp and prod devices, and cloud configuration scans from CloudQuery — this is more than just historical SIEM logs, but also asset information. The data lake can then be queried to find exceptions, look for TTPs, and verify controls.

~10% of teams built an on-demand access solution, to provide time-bound access to sensitive resources, like on-call access to prod, or support access to customer data. This may include an approval workflow, and is typically implemented by changing group membership in the identity provider. Although solutions like Opal exist, the heterogeneity of environments and the high cost of integration means that some organizations are choosing to build this themselves.

~10% of teams built a secret manager, and another ~10% built an interface to Vault. The secret managers were usually legacy — tools they might not choose to build themselves with what’s available today, but also tools that they aren’t incentivized to migrate off of. For those who built a Vault interface, this is usually a front-end interface to provide an improved developer experience with restricted functionality, since “using Vault isn’t intuitive… the tooling around it sucks.” It might also have a pluggable back-end to multiple regional instances of Vault, used for data locality or different environments.

What about AI?

Unprompted, AI barely came up as a security issue. (No, really, VCs! Also, please stop emailing me about this.) There is a lot of interest and excitement about using AI to improve security tooling, but far less about how to secure AI.

While not yet a pain point, security leaders are aware of AI security challenges. GenAI governance is the main concern, to ensure that only the teams who have been authorized to are using genAI, with only allowed data: just to check, “are you using the enterprise version?” This visibility and governance (rather than security per se) is being provided by tools like Lanai, Harmonic, and Kindo.

The next concern is preventing unauthorized data from being trained upon. This is just data governance: “You need to know what you’re protecting, and where it is.” This is a bit of a chicken and egg problem: they might not know exactly what and where their data is, and current DLP tools don’t do a great job of classifying non-traditional data, and “next gen DLP doesn’t really exist yet”; but they can’t train a model on their customers’ data to find it, as that’s explicitly what they’re trying to avoid.

For companies hosting their customers’ models, or serving multi-tenant models, workload isolation was a concern. They need an efficient way to isolate different customers’ code, models, or interactions, and gVisor is the current best option. A solution like Edera looks promising.

If there’s one takeaway I have — other than the fact that our work in security is never done, and our jobs are quite secure, in fact — it’s that a lot of the problems we’re facing aren’t new. Our industry cycles through technologies, and though new infrastructure like GenAI might need new tools to secure, it doesn’t need new concepts.

There are surprisingly few innovations that have really been step functions that wholesale eliminate classes of issues in the industry, such as hardware tokens or memory-safe languages. How can we get more of those, so that this list looks very different in 5 or 10 years?

The mention of any vendor in this blog post is not an endorsement. Disclaimer: I am an investor or advisor of Chainguard, CloudQuery, and RunReveal.

Delegating security remediation to employees via Slack

Thu, 17 Oct 2024 07:00:00 GMT

I’m noticing a new trend with security workflows in Slack: instead of relying solely on the security team to pass on a message or take an action to remediate an issue, another team is asked to and able to do so directly. This is shifting left, like in DevSecOps, but rather than meeting developers in their dev tools, we’re meeting employees in Slack.

SlackOps (or ChatOps) was pioneered by DevOps teams to take actions directly in Slack, not just receive notifications — to run commands to scale up your infrastructure, monitor the progress of a rollout, or even start a rollback in case of an incident. What I’m seeing in security is not only these operational actions, but also using Slack to help speed up the human interactions you might need as part of triage or remediation.

We can broadly categorize security interactions in Slack into four categories.

First off, we have security reminders. These are notifications to employees that security wants them to do something, that is, check a box. These reminders are asking your employees to complete security-critical tasks like taking a security awareness training, acknowledging a policy, or reviewing existing permissions.

Next, we have plain old alerts. These are notifications to the security team that something might be amiss, usually from a scanning or monitoring tool, like an open firewall port or a new user being added. These notifications come from anything that sends events to a webhook. Most security tools which do some kind of scanning and tell you, “hey, I found this” fall into this category, including Snyk, HackerOne, Truffle, Semgrep, and many more. They turn Slack into an inbox for the security team to triage issues.

With the advent of SlackOps, there’s a shift from just notifications to actions, allowing us to complete workflows in Slack.

For security, that’s a workflow for responding to an alert or another team’s request — “hey, security, can you review this product before launch? can I get access to that tool?” These effectively turn Slack into a ticketing system (but a ticketing system with memes), and lets the security team take action via Slackbots. In response to an alert of a suspicious login attempt, I might take an action like lock an account. PagerDuty lets you acknowledge an incident, assign a responder, run an incident response workflow, and even escalate — all without needing to leave Slack. Opal lets you approve an access request. And Tines lets you extend some of your existing security alerts to automatically take action, so that even if your tools are disparate and don’t come with native Slackbots, you can string them together to take action like block a domain based on threat intel about the URL. Organizations are also building Slackbots for their own internal workflows, like OpenAI’s bots for incident response, reviews, and triage.

Lastly, using Slack to delegate security alerts — this is the new trend I’m seeing. Rather than a security tool alerting the security team (in Slack), who then needs to find the right person to ping (also in Slack) — what if the tool just short circuited that and went right to the source (in Slack, of course). This is a change in who is alerted first. Rather than alerting Alice in security that Bob shared a Google doc with an external party, instead alert Bob and ask if he meant to share the doc — if he didn’t, then alert Alice. This applies to any situation where security needs to check: “did you mean to do that?” (This isn’t quite DevSecOps, and it’s not quite SlackOps, so maybe it’s SlackSecOps? I’m sorry.)

Kolide (now part of 1Password) seems to have popularized this first, notifying users in Slack that their devices don’t have disk encryption, have unencrypted SSH keys or account recovery passwords sitting around, and other failing osquery checks. And now, Nudge reaches out to SaaS app users to ask them to enable MFA, or confirm if they still need the account.

I think this is a huge opportunity to distribute security responsibility, and Kolide and Nudge really hit on something. There are many places this kind of workflow could be useful in security, where some enrichment or remediation action needs to take place, and the resource has a clear owner. The resource could be a device (Kolide), SaaS app (Nudge), file, database, …anything really. You could also do this with dev resources like VMs, containers, services, and GitHub repo, where you don’t otherwise have a natural place to provide an alert. (I’m not proposing duplicating alerts that already exist elsewhere, but rather replacing alerts that were previously manual). Where you have a defined policy, you can also propose a desired action to take rather than escalating the issue — to some extent, this is automating the first round of triage for exception management.

You could ask employees: Did you mean to share that Google doc with an external person? Do you still need that test instance that’s not getting traffic and you haven’t touched in a month? Did you mean to invite a contractor to use that app?

I understand why these types of workflows are emerging: security teams are spread very thin, and could always have more context on the business’ needs. Delegating these security alerts is removing the step of a human reaching out in Slack (no dreaded “hello”). Instead, the security team gets to spend more time on the exceptions that remain.

BeyondCorp is dead, long live BeyondCorp

Wed, 09 Feb 2022 08:00:00 GMT

With the US government’s recent memo on Zero Trust Cybersecurity Principles, there’s renewed interest (and investment) from organizations in adopting zero trust architectures. BeyondCorp, Google’s initial implementation which spawned the pursuit of zero trust in general, is still the guiding star for many organizations. It would seem that the authors of the US government’s memo have, just like the rest of the security industry, read the BeyondCorp whitepapers — and heavily based their strategy on BeyondCorp.

In reality, however, no organization has successfully implemented a fully zero trust architecture, and many proponents of zero trust — including the US government — have missed a key component: devices. Let’s ignore the memo’s recommendations on DNSSEC and STARTTLS, and focus just on the zero trust architecture.

What is zero trust architecture?

Traditional network architecture relied on a network perimeter to delineate between trusted and untrusted users, such as trusted employees inside a firewall, vs. potentially untrusted parties outside of it. By moving to a zero trust architecture, the location of an individual, specifically, which network they are on, is no longer solely what determines whether the individual is trusted, but other context is used to determine whether they can access a given application. There is no longer such a thing as a privileged, physical, corporate network.

(Most discussion about zero trust these days typically refers to users accessing an enterprise’s internal services, applications, or machines; rather than one service connecting to another service. This is unfortunate, as service to service communication is extremely relevant — even if you validate and allow an individual to access a low risk application, but that application then can make calls to a high risk application, you haven’t effectively protected your high risk applications. If you’re only protecting the front door, but the killer is already inside the house… well, you’re not going to have a good day.)

We can think of access controls for each application in terms of network segmentation. Each application is in its own segment (micro segmentation), instead of broad concentric circles of increasingly trusted applications. It’s not that a VPN is bad per se, it’s that any entrypoint or gateway that gains access to a broad set of applications, instead of only a specific application, is bad.

BeyondCorp, first introduced in a paper in 2014, is Google’s original, specific implementation from which the broader generalized set of principles (and analyst categories) for zero trust architecture emerged.

In BeyondCorp, the idea is that applications are available directly from the public web, not inside a trusted network, and that every request to access an application is a policy decision, based on the user, device, and application. Each of these characteristics can be along a spectrum:

User trust: Determined using a user identity (SSO token) and user database (identity provider). A user can be challenged to prove that they are trusted, such as requiring them to sign in or requiring a hardware second factor.
Device trust: Determined using a device inventory, device identity (certificate), and device measurements. A device might be untrusted, like a hotel lobby computer in a foreign country, or might be more trusted, like a fully patched Chromebook, using an MDM, in the company’s device registry, with a device certificate protected by a TPM, also used for secure boot to verify the device’s OS, connecting from the same physical network as the company’s head office.
Application policy: Each application can define a policy with a different fine-grained set of requirements for access, including different minimum requirements for user and device trust. If there is insufficient trust when a user tries to access an application, a challenge can be issued to the user to further authenticate.

This is an incredibly complex, sophisticated model for how to manage access to internal applications. BeyondCorp is the gold standard of what we should all aim for when designing zero trust architectures---including what the US government is now mandating.

There’s one problem: a fully zero trust architecture is incredibly difficult and incredibly expensive to deploy, and arguably, no one has ever achieved it.

Even Google’s BeyondCorp isn’t a fully zero trust architecture

Google adopted its zero trust architecture of BeyondCorp gradually, targeting both greenfield and brownfield applications, over the span of several years. Today, still, BeyondCorp isn’t 100% rolled out at Google. It never will be. There will always be a gap of enterprise applications or new tools being introduced which require work to integrate.

Like any system, there are exceptions — and those exceptions become more and more expensive (and therefore unworthwhile) to address. Citing another of the BeyondCorp papers:

Despite all the best efforts to define, roll out, measure, and enforce controls, you may inevitably face the harsh reality that 100% uniform control deployment is a mythical state where unicorns frolic unconcerned about malware and state-sponsored attackers.

Google has already invested substantial resources, over many many years, in developing BeyondCorp:

Fully achieving the goals outlined in this paper (and the more general goals of BeyondCorp) requires significant resources.

If it’s “significant resources” at Google scale, it must be a massive investment. I would venture to say that if a VP knew the initial cost and time it would actually take, they might not have made that investment. (It’s likely more than the $9.9M the Office of Personnel Management has set aside.) For Google, there was a clear reason to invest; but for a lot of other organizations, there isn’t.

I don’t want to paint the impression that Google’s implementation of BeyondCorp isn’t successful. It is — it’s a project that has significantly changed how Google operates and improved its security. It has introduced a new model for how to think about network security in the industry — zero trust architecture — and birthed multiple analyst categories, as well as many, many startups. It’s a multi-year initiative in security teams at every major tech company. The US government’s memo cites it in everything but name.

So then, despite its popularity, it shouldn’t come as a surprise that if Google hasn’t fully succeeded, companies other than Google haven’t either.

Security teams will laugh if you say you’re implementing a zero trust architecture

As with other groundbreaking research coming out of top tech companies, when the BeyondCorp paper was published, startups were created with the goal of reproducing such a zero trust architecture to make it available to anyone. They tried to offer, and I quote, “BeyondCorp… for the Cloud Native organization”, “BeyondCorp outside of Google”, and “BeyondCorp for the rest of us”, amongst others. They bought beyondcorp.com. They created an Alliance. The first BeyondCorp paper (there are many) was released in 2014 — but it’s 2022 now, so shouldn’t you, too, have a zero trust architecture?

What happened? Startups are going to startup, by failing, or getting acquired and absorbed. Today, you can use an OpenSSH ProxyCommand config to authenticate your SSH sessions using Okta, and you can limit access to an application from a device based on IP address and whether screen lock is enabled. That’s better, but not saying much, unfortunately.

The reality is if you say you’re “doing zero trust” to a security professional today, they’ll assume you’re naïve, and haven’t realized what you’re actually signing yourself up for. They’ve tried to use the tools already available in the market themselves to get closer to a zero trust architecture, and faced too many challenges. They’ve seen the market get excited, companies be born and die, and yet nothing really changed. If multiple companies registered in Delaware already died trying to make this happen, why would a company headquartered in Virginia be successful? This might be a rare case of security professionals being realistic, not overly pessimistic (as they are often portrayed).

Not to depress you even further, but it’s even worse. The tools that are on the market today aren’t even doing the hard part of zero trust yet.

Everyone seems to have missed the bit about devices

Recall that Google’s BeyondCorp has three pieces: users, devices, and application policies. But — you may have noticed — the heftiest section of that, by a long shot, is devices. Device trust should be determined using a device inventory, device identity (like a certificate), and device measurements.

So, what does the US government memo say in its barely one and a half pages on devices? That you should inventory your assets, and have government-wide endpoint detection and response. There’s more written on MFA than the whole topic of devices! (Not that MFA isn’t important.)

Why are devices so important? If you recall, if a user has insufficient trust to access a specific application, Google’s BeyondCorp can require the user to perform a challenge, such as re-authenticating or using a hardware second factor. But what’s completely impractical? Asking the user to change the device they are on! (Even my favourite Googler has 5 laptops at home, but only some of them work, even fewer are for work, and most of them are in the other room.) So, when you’re making a decision to authorize a specific access request, you have some data about the user, but ideally, you have as much data about the device as possible, since you can’t get any more. Most of what you can infer about the level of trust for a given connection comes from the device, not the user.

What should you, and the US government, be doing to measure device trust? To poorly summarize: have an inventory of your fleet, use an MDM to measure OS version, patch level, and encryption status, yes — the memo got this part. (By the way, have you tried to buy an MDM that covers more than three OSes lately?)

But also, use a device certificate that is specific to each device, protected by the machine’s Trusted Platform Module (TPM). To do that, you only need enough of an understanding of TPMs to implement secure boot to verify the device’s OS, as well as protected device certificates (sorry, NSA, I don’t think that your implementation in Java, which was susceptible to log4j, will do), and the ability to run an enterprise CA for those certificates, that is available whenever your employees need to access any applications. You already run one of those for your SSH certificates, right?

Where does that leave us?

So, someone at the US government also read the BeyondCorp papers, and also wants a true, working, zero trust architecture. Don’t we all. And I suspect that the government mandating this (by 2024!) won’t make it true — there isn’t a tech company where a top down mandate like this would work today, and I have no reason to believe the US government can do better.

These are the right goals. As an industry, we can continue to build pieces of an ideal zero trust solution. A solution that includes:

User authentication, based on single sign-on, and hardware second factors;
Device authentication, based on a device registry, a hardware-bound device identity, and measured device characteristics like secure boot;
Application policies, so that each application is micro segmented and enforces its own policies, with no single point of failure;
No public point of entry to the network;
End-to-end encryption with world-class cryptography;
… and meets all the traditional enterprise requirements, like audit logging or SAML integration.

I don’t think anyone today is positioned to build every part of that solution. It’s far too much for an organization even like the US government to get right in only 3 years. We’ll still be pulling together components piece meal, where they exist.

The US government memo feels not like a mandate for zero trust, but a mandate to insert “zero trust” in all of our marketing. Here’s hoping we get a few of the missing pieces out of it too.

Burning out and quitting

Sun, 22 Aug 2021 07:00:00 GMT

I’m burnt out. Or, I was.

I was recently unemployed — fortunately, by choice — and I was struggling to get out of bed by 10am, struggling to work out, struggling to only have one glass of wine, struggling to fall asleep. It’s not much different than how most weekends have felt since the pandemic began, but I was doing it every day, going on… too many months. And it’s not (just) the pandemic — it’s an overwhelming feeling of being done, done with this, whatever this is. I needed a break.

It’s hard to describe exactly what this feeling is, but burnout seems like the right descriptor. There isn’t a clear definition of burnout (and I’m not any more adept at describing it, especially not while experiencing it), but I am indeed experiencing the “exhaustion, cynicism, and loss of efficacy” that is its trademark. Being exhausted or cynical isn’t new for me, that’s my personality; I’ll overcommit at work, travel to six cities in a month, and make it home in time to pull out all the stops for an elaborate dinner party. The loss of efficacy is the symptom I’ve been wrestling with the most, for months now — and what I want back the most.

My experience

The pandemic

I think I burnt out around November 2020.

Like everyone, I went through phases of the pandemic (and no, I have no desire to read another pandemic memoir, including my own. I’m sorry). I was thankfully employed — doing well at work, though adjusting to more loneliness and working from home, and keeping busy with exercise. I started really losing it around June 2020, drinking too much. I noticed, reigned it in, and continued. But by November, I was tired. Tired of another monotone day. Tired of my inability to control basic choices that I knew would improve my lifestyle, like my volume of meetings. Tired of 7 hours of Zoom a day, only to try to cram more work in in the evening, to no avail. Tired of unnecessary drama at work — drama for the sake of drama. I was floundering.

I was working longer and longer hours, and getting less and less done. And being asked to continue doing that. Or worse, being told I’m doing a great job with one crisis, and to move onto the next fire. I didn’t think I was doing a great job. I was doing a terrible job.

And I definitely wasn’t fulfilled. I never got to finish anything. I barely got to start things. I was always tired. Always in another meeting. Always pretending everything was fine, to myself and to others.

I don’t think I noticed I was burnt out until early February 2021, almost six months later. Honestly, realizing it was kind of a relief. I hadn’t noticed how bad it had gotten. A few weeks later, I quit my job. And then a new, different kind of struggle started. Not knowing what to do with myself, or how to recover.

This burnout felt different from before

This experience was different from when I’ve burnt out in the past. A few jobs ago, I remember being really truly exhausted — going on weeks of working 80+ hours, staying up until 2am every night working to get back up at 7:30am — but I was still, shockingly, productive. (I blame my younger self, there’s no way I could sustain this for even days now.) One day, I was on a conference call (remember those? it’s like Zoom but you can mute to have side conversations with whoever is in the room!), and I just, uh, stopped. I couldn’t process what people were saying, I didn’t understand what was going on, I felt nothing. Like my brain shut down. I just stared off into space, looking outside at the sky. This lasted for a few minutes, and mellowed out over several hours, but it felt like something fundamental had just shifted. I could no longer will myself to work those long hours, or to go to the office, or to answer calls. And this feeling lasted for days. Each of those moments was a struggle. I took a month off work, and only returned for the time I needed to job hunt.

In retrospect, this might not have been a burnout. Or, it was on a vastly different scale, a 3 on a scale of 1 to pandemic (if you want an actual scale, check out the Buzzfeed-inspired Maslach Burnout Inventory). I guess I didn’t realize I was burnt out this time, because it was nothing like my prior experience — there was no singular event that felt like a step change. It was just the monotony of another exhausting day with 7 hours on Zoom, then trying to do real work, at 1am, with a glass of wine on the couch. It felt like I was making the best of a situation. I hated it. It hollowed me out. I had nothing to look forward to.

So tired. Everyone is so tired.

Meetings keep getting cancelled because no one has a topic. Or they end early because there's so little to say. Team chats are dead, too.

I know we all hate meetings, but this is a symptom of mass burnout.
— Jay Conrod 🌴 (@jayconrod) August 18, 2021

So why did I burn out? I don’t know. It’s not a single thing — like a specific work stressor — that caused my burnout. It was the neverending treadmill of yet another day’s worth of useless meetings, with a TODO list that only grows, while you get less and less done on it every day. There isn’t a single moment that causes burnout, but there is a single moment when you realize it — that what you’re doing is impossible, insurmountable, unachievable — and that you don’t care. You can’t do it. And you don’t want to anyways.

So, why am I writing this?

Well, first, it’s taken 10 weeks to get to the point of thinking about writing this to actually doing it. (I’ve finally read every fucking article on pandemic burnout.) Damn. I thought I had almost recovered a few weeks earlier — but I hadn’t. I’m still not fully myself again, and it’s hard to say how long that will take.

I’m writing this for two reasons:

This is a hopeful directive that you can treat yourself, and others, better than I did. The reception I’ve had from colleagues, friends, acquaintances to my burnout has been varied, and also, not always what I expected. And I know I personally would have been on the unfavourable end of that spectrum.
If you’re reading this, there’s a strong chance you’re burnt out too. We’re about to have, uh, a moment, so brace yourself. A lot more white collar workers are burnt out from the pandemic than they realize. (Blue collar workers are obviously burnt out too — and underpaid and mistreated — but hopefully that is already apparent.) A lot of folks are about to take extended leave, or change jobs — the rate of job quitting and switching is the highest it’s been since 2000. I’m hoping you can give yourself the freedom to recoup, and learn what to expect from yourself in this transition.

Responding to burnout

To vastly oversimplify, there are two kinds of people I’ve talked to about burnout — those who get it, and those who don’t. Those who get it, get it because they’ve felt it themselves. Or, they’ve seen friends or family hit a wall, become a ghost of their former selves, and know. They wouldn’t wish it on anyone. They know it just takes time.

At the other end of the spectrum, is, uh, what I used to be like. “Oh, yeah, I hit a wall one time, I took a few weeks off and felt fine. You’ve been off for, what, two weeks now? You’ll feel fine next week. Let me introduce you to some people. Let’s have a call next week.” Other people did this to me; but the worst part is, I did this to myself. Just one more meeting to connect. Just read one more blog post. It’ll be fine. You’ll feel fine.

I didn’t. I needed to step back, and actually do nothing — or as close as I could possibly fathom. I needed to completely remove any feelings of pressure, or any external, and internal, obligations. “You decide what to watch on Netflix because I literally can’t.” I’ve eaten more takeout in the last few months, than the whole pandemic; I didn’t have the energy to shop for groceries, or cook. I desperately needed to enjoy things again — so I could remember what that was like — so I could get back to enjoying ‘productive’ things too. Remember that producing recovery, relaxation, or joy for yourself is still being productive.

If you’re in the same situation, or you encounter a friend in the same situation, all I can suggest is to be kinder to yourself than you realize. Don’t add anything to your plate until you’re recovered — and please don’t be the person that does that to others. If you haven’t experienced burnout, then maybe, imagine 10x worse than your current perception, and act accordingly.

Taking time off or changing jobs to address burnout

It’s been a year. The media thinks we’re languishing, or, if we’re not, we might spend the summer living out our wildest dreams. There are So. Many. Predictions.

They don’t know, because we don’t know. I don’t know what I want to do today, or eat for breakfast, or which local coffee place to support — like I don’t know what kind of job I want, or if I want to still be in the Bay Area in five years. This past year has caused fundamental changes in our lifestyles, and caused us to rethink, or reconsider, what we really want. I’m overwhelmed by choice, and simultaneously, tired of making choices, so the solution is just to… do nothing.

Everyone I know is going to change jobs this year — for various reasons. I don’t always believe Bloomberg forecasts, but well, sometimes I do: The Great Resignation is coming.

Paraphrased from a few friends:

“During the pandemic, my increased work performance wasn’t recognized and wasn’t rewarded. If I’m not promoted at my next review, I’m quitting. Even if I’m promoted, I’m quitting.”
“I’m tired of being on this team, everyone I like working with is leaving. I need to figure out what’s next.”
“I have three months of vacation saved up. I’m going to take them and then quit. I don’t know what I’m going to do next.”
“It feels like I’m stuck in a cycle. We’re having the same conversations at work we had 6 months ago, and nothing’s changed.”
“I hate my job.”
“My manager has no idea what I do.”
And of course, “I don’t want to have to go back to the office, I want to work remote from now on. I’ve already bought a house in X.”

The narrative I’m seeing in the media is wrong — I don’t see people leaving their jobs to take unreasonable risks, running a full-time Etsy jewelry store or dog clothing dropshipping business. I see people leaving their jobs for other relatively secure jobs, because they’re fed up. Many are considering career downsizing, a previously unavailable option — (trying) to trade in unpredictable things you don’t like, for only the things you do. They’re burnt out, and need a break. “If you think you’re burned out, you’re burned out, and if you don’t think you’re burned out you’re burned out.”

A lot of tech employees have already changed jobs this year — it’s easier to interview than ever, when it’s just opening another window on your laptop, not taking a shared ride across town. But a lot more tech employees stayed — because of uncertainty, and, honestly, because of record stock performance. And they’re pulling the trigger now.

It’s going to be extremely hard to hire talent in 2021 — banking is already facing serious turnover rates, and beefing up recruiting teams. There are 10 million job vacancies in the US, more than there are unemployed people. And as hard as it’s going to be to hire talent, it’s going to be even harder to retain talent. (I’ll share some of my thoughts soon on building a culture to attract and retain talent.) And if you’re already down a recruiter, well, good luck, you can basically resign yourself to having a lot of open roles at the end of the year… which will make it just that much harder to retain talent. (Startups — you know those crazy valuations you’re getting right now? Go get the fucking people you want. Now!)

I never thought I’d take five months off, without being able to explain to a future employer what I was doing. It felt like too much. But here we are. I hope they’ll be empathetic and understand — and honestly, it’ll be a red flag if they don’t. But I also think that this will become so common, that it’ll be normal. “Oh, you took a few months off in 2021? Me too.” (“Oh you gained 15lbs in 2020? Me too.”) If you have the financial ability to, don’t feel like you can’t quit your job. Your sanity is worth it.

What’s next

I’m finally in a mindset where I can actually think about how to invest my time, not just be afraid I’ll never recover from this. End to end, it’s taken 6 months to realize I was burnt out while trying (and failing) to work, 3 months to recover, and then 2 months of vacation to feel excited to work again — which is longer than I ever would have expected. But I’m so happy I gave myself the time I needed. I don’t know exactly what I’m looking for, but I know what I’m not looking for: spending all day on Zoom.

I just went on vacation, and I just started a new job (slowly), one I’m excited for. Going on vacation felt normal, like other vacations — something I enjoy, that re-energizes me, that gives me new hope and new ideas and an itching desire to just get in front of a whiteboard or a keyboard and do something.

Looking back, the best moment I had in 2020 was over Christmas break, sitting on the couch with my laptop. I spent all day, maybe 8 hours, reading about SolarWinds. My boyfriend told me to stop working. It wasn’t work, and it was great. I was learning something. Completing something. Doing something because I wanted to do it, not because it was the next urgent thing that needed to happen. It felt like work used to feel like. That’s what I’m looking forward to again.