How we’ll solve software supply chain security

Who owns software supply chain security? Developers? Or the platform and security engineering teams supporting them?

In the past, the CIO, CISO, or CTO and their security team would decide which Linux distribution, operating system, and infrastructure platform the company would be getting its support contracts and security SLAs from. Today, developers do this all in Docker Files and GitHub Actions, and there isn’t the same kind of organizational oversight that existed before things shifted left to developers.

Today, compliance and security teams define the policies and higher level requirements, while developers get the flexibility of choosing whatever tooling they want, provided it meets those requirements. It’s a separation of concerns that greatly accelerates developer productivity.

But as I wrote previously, Log4j was the bucket of cold water that woke up organizations to a systemic security problem. Even in the midst of all this shift-left developer autonomy and productivity goodness, the open source components that make up their software supply chain have become the favorite new target for bad actors.

Open source is great for devs, and great for attackers

Network security has become a far more difficult attack vector for attackers than it once was. But open source? Just find an open source dependency or a library, get in that way, and then pivot to all of the other dependencies. Supply chains are really about the links between organizations and their software artifacts. And this is what attackers are having so much fun with today. 

What makes open source software great for developers also makes it great for hackers.

It’s open

Developers love: Anyone can see the code, and anyone can contribute to the code. Linus Torvalds famously said, “Many eyeballs make all bugs shallow,” and that’s one of the big benefits of open source. The more people look at things, the more likely bugs will be found. 

Attackers love: Anyone with a GitHub account can contribute code to critical libraries. Malicious code commits happen frequently. Libraries get taken over and transferred to different owners that don’t have everyone’s best interests in mind.

A famous example was the Chrome plugin called The Great Suspender. The person maintaining it handed it off to someone else who immediately started plugging in malware. There are numerous examples of this type of change from benevolent contributor to malicious contributor.

It’s transparent

Developers love: If there are issues, you can look at them, find them, and audit the code.

Attackers love: The vast volume of open source makes code auditing impractical. Plus, a lot of the code is distributed in a different source than how it is actually consumed.

For example, even if you look at at the source code for a Python or Node.js package, when you run pip install or npm install, you are actually grabbing a package from what’s been compiled, and there’s no guarantee that the package actually came from the source code that you audited.

Depending on how you consume source code, if you’re not actually grabbing source code and compiling from scratch every time, a lot of the transparency can be an illusion. A famous example is the Codecov breach, where the installer was a bash script that got compromised and had malware injected that would steal secrets. This breach was used as a pivot to other builds that could be tampered with.

It’s free

Developers love: Open source comes with a license that guarantees your ability to freely use code that others have written, and that’s awesome. It’s much easier than having to go through procurement to get a piece of software improved internally.

Attackers love: The Heartbleed attack from 2014 was the first wakeup call showing how much of the internet’s critical infrastructure runs on volunteer work. Another famous example was a Golang library called Jwt-go. It was a very popular library used across the entire Golang ecosystem (including Kubernetes), but when a vulnerability was found inside it, the maintainer was no longer around to provide fixes. This led to chaos where people were forking with different patches to fix the bug. At one point there were five or six competing patch versions for the same bug, all making their way around the dependency tree, before a single patch finally emerged and fixed the vulnerability forever.

Open source is great for software supply chain security too

The only way to make all these links stronger is to work together. And the community is our biggest strength. After all, the open source community—all of the project maintainers who put in their time and effort and shared their code—made open source pervasive across the industry and inside everyone’s supply chain. We can leverage that same community to start securing that supply chain.

If you are interested to follow the evolution of this software supply chain security domain—whether you are a developer, or a member of a platform or security engineering team—these are some of the open source projects you should be paying attention to:

SLSA

SLSA (Supply chain Levels for Software Artifacts, pronounced “salsa”) is a prescriptive, progressive set of requirements for build system security. There are four levels that the user interprets and implements. Level 1 is to use a build system (don’t do this by hand on a laptop). Level 2 is to export some logs and metadata (so you can later look things up and do incident response). Level 3 is to follow a series of best practices. Level 4 is to use a really secure build system.

Tekton 

Tekton is an open source build system designed with security in mind. A lot of build systems can run in ways to be secure. Tekton is a flagship example of good defaults with SLSA baked in. 

In-Toto

In-Toto and TUF (below) both came out of a research lab at NYU years before anyone was talking about software supply chain security. They log the exact set of steps that happen during a supply chain and hook together cryptographic chains that can be verified according to policies. In-Toto focuses on the build side, while TUF focuses on the distribution side (was it tampered with?). 

TUF 

TUF (The Update Framework) handles automatic update systems, package managers, distribution, and sets of maintainers signing off through quorum. TUF also specializes in cryptographic key recovery when bad things happen.

Sigstore

Sigstore is a free and easy code signing framework for open source software artifacts. Signing is a way to establish a cryptographically verifiable chain of custody, i.e., a tamper-proof record of the software’s origins. 

Better guardrails for the software supply chain

Over the last 10 years, the selection of tooling and security both shifted left to developers. I believe we’re going to see developers continue to maintain their autonomy in selecting the best tools to use, but that the responsibility for a governing security posture and related policies needs to shift back to the right.

A common misconception is that security teams spend their days reviewing code line by line to find security bugs and make sure there are no vulnerabilities. That’s not how it works at all. Security teams are much smaller than developer teams. They are there to set up processes to help developers do the right things and to eliminate classes of vulnerabilities, rather than one security bug at a time. That’s the only way security can keep up with teams of hundreds of engineers.

Security teams need a standard set of processes for locking down roots of trust for software artifacts, and developers need a clear path to balance open source selection against clearly defined security policies. Open source posed the problem, and open source will help find the answers. One day, developers will only deploy images that have been vetted to prevent known vulnerabilities.

Dan Lorenc is CEO and co-founder of Chainguard. Previously he was staff software engineer and lead for Google’s Open Source Security Team (GOSST). He founded projects like Minikube, Skaffold, TektonCD, and Sigstore.

—

New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to newtechforum@infoworld.com.