Enabling Engineering Teams Through Developer-First Secrets Management

Secrets management is a challenge that every organization must tackle, from large enterprises to small development teams. The larger the organization, the more secrets tend to sprawl and the more engineering teams feel the friction or toil in organization processes. While a number of tools and products exist to provide solutions to secrets management, each will incur its own security or logistical tradeoffs. While there is no universal solution for secrets management that every organization should adopt, there are common patterns that most organizations evolve through. For enterprises that want to integrate security controls into an engineering organization of hundreds of developers, a pattern modeling a developer-first security approach is essential to match the speed and scale of modern businesses.

The problem with shifting left

Currently, “shifting security left” is the hot trend to scale security activities into engineering practices in modern organizations. Shifting security left attempts to improve security feedback lifecycles by inserting security activities earlier in developer workflows. The goal is to enable developers to catch security problems early and fix them when it is “cheaper” - during development, before the problem reaches the production environment.

However, for many organizations the shift left model is not producing meaningful results. Security teams are attempting to institute a cultural shift within engineering organizations without making requisite cultural changes inside the security program. They are certainly adding a lot of new security work for engineering teams - but these activities are not meaningfully impacting the risk posture of the business.

Shift-left security organizations abdicate ownership of the logistical complexities of integrating security initiatives into development workflows. They throw security signals to developers and make engineering teams accountable for figuring out how to juggle these concerns alongside their product, testing, and infrastructure responsibilities. Nick Liffen recently outlined the challenges this mindset creates for developers in the GitHub Universe 2022 talk, “Shifting left vs developer-first security.” Kelly Shortridge describes this practice as “security obstructionism.” Security practices that simply shift toil work from a security team onto an engineering team are an unfortunate corruption of the shift left mentality. Focusing security initiatives and products around a developer-first approach are essential for security initiatives to match the speed and scale of modern businesses.

A flaw I’ve seen with many well-intended shift left approaches is that they seek culture shifts solely within engineering and do not apply the same mindset shift within security. The security program should focus less on, as Kelly Shortridge describes, “security outputs as a proxy for progress” and instead on the needles that materially move the risk posture of the organization. A business risk is any exposure an organization has to factors that will lower its profit or lead it to fail. A security risk is, similarly, a vulnerability that threatens the company’s ability to achieve its objectives, including losing customer trust. Therefore, security practices that lower the risk posture of the organization while helping the business achieve its goals are crucial. A powerful way to help ensure security activities meaningfully improve the business in this capacity is to prioritize the contextual impacts when planning security initiatives.

Shift-Left Secrets Management

To understand the distinction between shifting left and developer-first security, let’s look at what a “shift left” approach to secrets management might look like.

Under this approach, developers are responsible for managing the secrets their systems need. Security offers a secrets store; they’ve set up HashiCorp Vault for the org to consume. Changes to secrets policy occur in a GitHub repository in an infrastructure-as-code setup. When developers want a new Vault role or to make changes to the secret paths accessible to them, they push a pull request to this repo and security approves and merges the changes. To assist with tracking ownership of secrets inside vault, the security team establishes required patterns for how developers’ must format roles and policies in their PRs.

So far, the bones of this approach look good. Developers have the ability to make the changes they need, as opposed to filing a ticket on another team and waiting. Security provides general best practices and expectations to streamline consumption in the org. Changes are managed through version control and codified in source code. Security has a bit of a gatekeeping role, but takes action frequently enough that this is more a questionable use of security’s time than a true impediment to engineers. The problems arise when you look at the day-to-day activities of developers interacting with this setup. Here is a made-up conversation derived from many very real threads I’ve seen in the past:

Developer: “Hey I am setting up secrets for a new service for project ImportantToTheBusiness. What am I supposed to do?”

Security: “Here’s the link to our wiki page.”

20 minutes pass

Developer: “Hi, I think I followed those instructions correctly. Here is my PR.”

Security: “No, this is not going to work how you think it will.”

Developer: “Oh, ok. What do I need to change?”

Security: “Here’s our troubleshooting wiki page.”

2 weeks of back and forth ensue

Security: “Here, I have redone your PR into the correct format. You should be good now.”

Developer: “Thank you!”

Security: “Here are the credentials to your secrets role. Don’t lose them. Store them safely when you create your pipelines.”

Developers: “How do I do that?”

Security: “That is your responsibility.”

While pointed, this is a realistic reflection of the interactions I’ve seen in organizations pushing a “shift left” mentality without the crucial contextual components of a developer-first mindset. Security sets up some secrets management “infrastructure-as-a-service” for developers to consume, and expects certain implementation patterns, but does not provide the “platform-as-a-service” resources to help developers implement those patterns.

Again, there are good bones within that conversation! Security has documented instructions engineers can follow. They step in and provide resolutions when developers are struggling. But, the overall process creates more problems for developers than it solves. There is high friction to getting a working setup and engineers often fail due to unclear regulations, requiring security to step in and do the work for them. At the point where the process no longer requires security’s involvement, they are hands off and unhelpful to the developer.

This “shift left” approach puts the onus of secrets management on the developer, but does not provide the guidance to allow them to be self-service. It solves for the security team’s problems without considering the developers’ issues. It lacks the contextual component of a developer-first security approach.

Developer-first security

So what does a developer-first security program look like? The key to developer-first security is contextual injections of security designs into the environment. The security program must integrate into the existing development workflows present in the organization by solving meaningful problems. Instead of creating a list of tasks for developers to follow to comply with a security requirement, security should design a paved path embedded within the normal operating workflows of the organization. Security should make the secure path the easy path for developers to tread. Security should operate as the equivalent of a platform engineering team, providing subject matter expertise and offering foundational secure-by-default practices that engineering teams can consume.

They must solve real business problems. The answer to “why do we have to do this?” should describe a clear path to enabling product objectives. “Because security says so” is the worst response, a legacy artifact of the combative security programs of the past who view their coworkers as the threat from which the business must be protected. Instead, the security team should operate as a collaborative partner, helping the business achieve its initiatives. A developer-first security program requires as large a cultural shift within the security team - perhaps larger - than the one within the engineering department.

Developer-first secrets management

So how do we adapt the secrets management program to a developer-first approach? While there should always be documentation, the program should strive to remove the need to use it. The on-ramp should be a paved path about which developers may read further, if they are interested. Turn the common and error-prone instructions into executable scripts that produce a viable solution. Expected structure or patterns within secrets policy should be codified through linters so developers are truly able to self-service their configuration and security can step back from gatekeeping pull requests. At DigitalOcean, our version of a secrets policy repository allows developers to run the equivalent of a ./bin/gen_new_service script that produces the secrets configuration most commonly employed for systems. Alternative use cases can be configured with flags on the executable, and those capabilities are documented for developers. Validation tests on the secrets PR check whether the desired configuration will break anything or fall outside of acceptable use.

Security should not abdicate any piece of the development workflow in which secrets are involved. Their recommended patterns and guidance should fully extend over the software development lifecycle. Within DigitalOcean, an interactive wizard exists to walk developers through the process of establishing a role to attach to their service, configuring multiple fine-grained access paths given common deployment models, and generating the engineering deployment pipeline(s) for the developer with a working integration of their new, specific secrets role built-in. This not only provides a paved path for the secrets management that security cares about, but a solution that makes it easier for developers to deploy their engineering pipelines, inside of which secrets consumption is just a small part.

Notably, the wizard does not ask the developer if they want to do “secret task A” or “secret task B.” The developer is not expected to understand the intricacies involved in making that decision. Instead, the developer is asked which engineering task they want to perform, and the tooling guides them through the relevant security steps for that task. The security team’s time can therefore be spent on adding additional common paths in the organization into the wizard, strengthening the available paved path. This solves engineering concerns alongside security ones. This makes developers’ lives easier. This is contextual, developer-first security. This requires engineering and product delivery capabilities within the security team, which may be a shift for some organizations. It is a necessary shift to transform from “security obstructionists” to developer-first enablers that embed security into the speed and scale of modern development team lifecycles.

A Developer-first solution to the secret zero problem

One friction point left unanswered from the initial “shift left” example is the requirement for developers to securely store their authentication credentials for the secrets store in their pipelines. At DigitalOcean, we’ve provided documentation outlining patterns that developers can employ to safely handle these across the various CI/CD tools we have used. Currently, developers are migrating to GitHub Actions. The introduction of OIDC support in late 2021 from Action workflows provided us an opportunity to develop a new, contextual paved path to help developers transition pipelines to GitHub Actions while removing the friction of credential management. Read our follow-up article to learn how DigitalOcean provides engineering teams with a developer-first approach to fine-grained RBAC while removing “secret zero” with GitHub OIDC and HashiCorp Vault.

Ari Kalfus is the Manager of Product Security at DigitalOcean. The Product Security team are internal advisors focused on enabling the business to safely innovate and experiment with risks. The team guides secure architecture design and reduces risk in the organization by constructing guardrails and paved paths that empower engineers to make informed security decisions.