Securing and managing a build pipeline is really complicated. And esoteric. In fact, I’m willing to bet that if you put 10 platform engineers from different organizations in a single room, you’d likely find that they work with at least 13 totally different flavors of CICD pipelines. It’s fair to say that many will not have full knowledge of the exact set of CICD pipeline security best practices and tools they employ.
BastionZero has long provided engineers with trustless access to infrastructure. We’ve built out human-to-machine access to servers, containers, clusters, and databases. Our unique security model allows us to offer our service as SaaS with zero entitlements to your infrastructure. In other words, our SaaS service does NOT have privileged access to your infrastructure, and a compromise of our SaaS does not lead to a compromise of your infrastructure. But our customers asked to extend this to machine-to-machine access so that they could use BastionZero as a tool to control their CICD pipeline’s access to their infrastructure.
What a CICD Access Security Tool Needs to Accomplish
We at BastionZero started working on the problem of machine-to-machine access months before the CircleCI breach happened; but, to be honest, we hadn’t quite considered the threat model of “your entire CICD pipeline gets breached” until early January, when CircleCI was breached. (We were focused on (1) eliminating credentials to infrastructure targets like servers, containers, clusters, and DBs) and (2) reducing the risk that our BastionZero service itself becomes a point of compromise.)
But, the CircleCI breach encouraged our R&D team to stretch itself further and provide a solution that works also in the aggressive threat model where the CICD pipeline itself is breached.
Why is this hard? The problem here is long-lived credentials. Conventionally, a CICD pipeline is an automated process that needs to be able to deploy code at any time. And that means that the credentials required to deploy to the pipeline must be available at all times. And, conventionally, that means long-lived credentials like API keys need to be stored in the pipeline. There’s not always a human in the loop to enter a 6-digit code for Multi-Factor Authenticiation (MFA) or to press “OK” on a big red button on Slack. Instead, these deploy jobs are automated and are typically initiated when a PR is merged into master. Thus, when securing machine access to infrastructure, it becomes tricky to use the techniques we typically rely on to secure human access to infrastructure — namely, short-lived credentials and MFA.
Nevertheless, our awesome R&D team at BastionZero has developed several clever solutions to this problem, and we’re planning to release the first stage of our solution next week to provide our customers with a more robust CICD security tool than ever before.
In the meantime, let’s walk through a few of the key security best practices we considered when designing our service accounts solution to the problem of securely connecting your CICD pipeline to your infrastructure.
Important CICD Security Best Practices
- Eliminate single points of compromise. Your CICD pipeline is a single point of compromise — if it’s breached, all is lost. Our solution needs to limit the blast radius of a breach of your CICD pipeline; if the pipeline is hacked, an adversary should not be able to steal credentials that provide unfettered access to your infrastructure.
By the same token, we control access to your infrastructure via BastionZero, so we need to make sure that the BastionZero service itself is not a single point of compromise. As with everything in BastionZero, we will eliminate single points of compromise by having two roots of trust. A compromise of the BastionZero service does not lead to a compromise of the infrastructure, because the BastionZero service is not authorized to access the infrastructure without the participation of another root of trust.
- Go passwordless! Our design philosophy is to limit the headaches associated with storing many different server access credentials in your CICD runner. (This should be familiar to those teams who spent days or weeks urgently rotating credentials following the CircleCI breach.) Instead, the runner just needs credentials that allow it to log into BastionZero, which makes credential management and rotation a lot easier.
- Set up centralized policies. Our approach allows you to set up centralized and time-gated (just-in-time) policies that can control what CICD runner has access to what set of targets and at what time. BastionZero allows you to do this in a single centralized policy manager across all your clouds, datacenters, and accounts.
- Maintain audit logs. Given the risk associated with CICD pipelines, it’s important to get a handle on when exactly your deploy pipelines are running, and what they are doing. BastionZero does that, producing audit logs of what your CICD pipeline is accessing and what commands it is running. Apart from supporting compliance efforts, this level of audit logging is also super helpful when responding to engineering mishaps or unexpected security incidents.
Next week, we’ll be releasing the first phase of BastionZero’s service accounts feature.
