Well, CircleCI was breached. And many teams spent a large part of January rotating secrets and looking for indicators of compromise (IoCs). I think this breach is a really big deal, because it’s exposing the massive attack surface that can be created by our CICD pipelines.
This is barely even about CircleCI --- this is mostly about a fundamental security issue that affects almost any CICD pipeline, whether it’s built on CircleCI or not.
The fundamental issue is that a CICD pipeline needs the power to deploy code into your infrastructure. (The D in CICD is for Deploy…) And deployment requires a high level of privilege, including the ability to SSH into servers, to talk to APIs, to push code into containers, to spin infrastructure up and down, and a bunch of other sensitive stuff. And if your CICD pipeline gets compromised, all those privileges fall into the hands of an adversary. Which is about the worst thing that can happen.
Why are you talking about CICD pipelines? At this point you might ask yourself. Wait a minute. BastionZero’s platform provides trustless access to infrastructure for developers and devops folks. So why are you talking to me about CICD pipelines? Well, developers and devops folks aren’t the only ones that need access to cloud and data center infrastructure -- Automation needs access too. And since we’ve started building BastionZero, our customers and prospects have been asking us to help them eliminate the credentials in their CICD pipeline. After all, what’s the use of eliminating the SSH keys on your engineers laptops, if those same SSH keys are still living in your CICD pipeline?
So that’s why we, at BastionZero, have spent a bunch of time thinking about the operation and security of CICD pipelines. We’re specifically interested in how the credentials that your CICD pipeline stores and manages the privileges and credentials it uses to access and deploy to your infrastructure.
How CICD pipelines manage credentials and privilege.
Let’s look at how this issue plays out in a few different CICD pipelines architectures - cloud-hosted, self-hosted, CSP-native and GitHub Actions.
All of these architectures use a CICD runner, which is an agent that runs CICD jobs, typically spinning up when the job starts and then spinning down when it’s done. The runner needs access to the credentials required to run a job; whether those are SSH keys to control access to servers, passwords to database or API keys to control access to kubernetes, AWS or another cloud service provider.
Cloud hosted? CircleCI is a cloud-hosted CICD tool. (There are others like this, eg Travis.) The cloud service typically holds the credentials that are used to provision the CICD runner when it spins up. The runner then uses these credentials to access the infrastructure and deploy the CICD job. Because the customer’s credentials live in the cloud service, a breach of the cloud service can result in a compromise of all the customer credentials. (This is what happened with circleCI in January.)
Cloud-hosted CICD tools store credentials in the SaaS. When a job is initiated, a runner is spun up and credentials are sent to the runner from the SaaS. Notice that a compromise of the SaaS leads to a compromise of the credentials.
After a breach like this, you might conclude that the solution would be to move to a self-hosted pipeline, or one that is natively provided by your code repositories, like Github actions. But that’s not the end of the story.
Self-hosted? But it should be clear that using a self-hosted CICD pipeline is not a panacea. A self-hosted CICD runner (like Jenkins) is still entrusted with a high level of privilege. And, you’re subject to the age-old problem of keeping your self-hosted system patched, updated and secure against attackers. (Which can be hard to do; since 2011, cvedetails.com identifies over a thousand named vulnerabilities in Jenkins.) And attackers know that hacking into your CICD pipeline is a great way to gain control of your infrastructure; here’s a report from NCCGroup about the myriad ways these hacks can be accomplished, several of which are specific to Jenkins. Given all of this risks, it’s critical to maintain awareness and control of the various credentials and secrets that are accessible to your Jenkins servers.
Even a self-hosted CICD runner needs to store the credentials needed to interact with the infrastructure. This is why self-hosted CICD are a valuable target for attackers.
Native CSP tools for CICD? The similar considerations apply even with a tool that is natively provided by your cloud service provider (CSP), like AWS CodeDeploy. AWS CodeBuild still has the ability to deploy code into your infrastructure. The fact that AWS CodeBuild is provided by AWS does not eliminate the need to properly configure a pipeline and protect it against breaches.
GitHub Actions? Github's introduction of native CICD tooling has encouraged many teams to transition to GitHub Actions, a CICD service provided by the same service that many teams use as their code repository. On one hand, transitioning to GitHub actions eliminates the headaches associated with managing and securing a third party CICD tool. But on the other hand, your cloud accounts still trust GitHub actions to push code into your infrastructure, and bad things can happen if GitHub actions is compromised. This is a real concern, since we’ve also recently seen GitHub become a target for attackers.
How we’re approaching this problem at BastionZero.
Securing and managing a build pipeline is really complicated. And esoteric. In fact, I’m willing to bet that if you put 10 platform engineers from different organizations in a single room, you’d likely find that they work with at least 13 totally different flavors of CICD pipelines. And all this complexity makes it hard to reason about security.
Here’s where BastionZero comes in. One of the key security issues with a CICD pipeline is that the pipeline needs access to your infrastructure, and therefore, it needs credentials to your infrastructure.
So this is what we are working on at BastionZero -- instead of storing credentials to your pipeline in your CICD pipeline, you instead mediate the CICD pipeline’s access through BastionZero. And because of BastionZero’s trustless architecture, you don’t need to store the credentials to your infrastructure with BastionZero either. Instead, our trustless architecture ensures that the BastionZero service does not have privileged access to your infrastructure, nor does it hold any credentials (or have access to a vault that stores those credentials).
In the next few weeks, we’ll be releasing the first few phase of BastionZero’s service accounts feature, which can be used to connect your CICD pipeline (and other automated process) to your infrastructure via BastionZero. Sign up for our newsletter below to get updates on our latest releases, or reach out to sales if you’d like to learn more about how BastionZero can empower your engineering teams with trustless access to infrastructure in any cloud or data-center environment.
