Reproducible and immutable builds can improve trust in the software supply chain after the SolarWinds hack

The ripple effects of the SolarWinds attack have continued into the first weeks of 2021. That’s not likely to stop — a new strain has recently been identified, highlighting just how deep and sophisticated this particular attack was.

While we’ve looked at a number of ways we can minimize the risks of supply chain attacks, there’s also a need for more comprehensive self-examination and reflection throughout the industry. True, there are a number of things that can be done at a surface level to minimize the risks of attacks like this happening again, but there are a number of ideas that would demand significant change that might be the only truly effective way forward.

This centres on reshaping the build process in a way that ensures safety and security at every level, and throughout the stack. To a certain extent DevSecOps — as a movement or discipline — has urged the industry to shift its focus. It’s always been obvious that security will never be effective if it’s just something that gets tagged on at the final stages of the development lifecycle, but at least now we have a term that can embed its importance in the industry’s collective imagination.

Equally however, the dominant ethos of agility and the urge to remove any friction between build and deploy is precisely what leads us into the sticky sorts of situations that appear to have reached their apotheosis in the SolarWinds attack.

Understanding the SolarWinds attack

If you want to understand why this should be the focus of our security efforts moving forward, you only need to look closely at how the SolarWinds took place — primarily, the attackers managed to gain access to Orion (SolarWinds’ monitoring platform) precisely because they injected code that allowed them to read changes in the build in a way that gave them the necessary intelligence to delve deeper without alerting SolarWinds engineers.

Take, for example, the third strain of malware identified by security researchers last month. This strain, although only just identified, is thought to be the first piece of malware used by attackers. Its purpose, ZDNET explains, was fairly simple:

“CrowdStrike [the security researchers] said Sunspot [the malware] had one singular purpose — namely, to watch the build server for build commands that assembled Orion, one of SolarWinds’ top products, an IT resources monitoring platform used by more than 33,000 customers across the globe.

“Once a build command was detected, the malware would silently replace source code files inside the Orion app with files that loaded the Sunburst malware, resulting in Orion app versions that also installed the Sunburst malware.”

The key word here is ‘silently’. Engineers at SolarWinds were unable to identify this alien code that had been embedded in their product.

Although we’ll never be party to SolarWinds’ engineering practices and process, there are some key measures that will form an effective defense against build pipeline compromises.

Broadly there are a number of key things we need to introduce to our builds to ensure greater security: segregation, reproducibility and immutability. If we can do that we will go a long way to ensuring much greater trust and confidence throughout the software supply chain.

Immutable builds: consistency in your build environments

The first element that we can do as software engineers to ensure absolute trust in software and minimize the risks of attacks that enter via build systems is to ensure consistency in our build environments. This is done through something called immutable builds.

What are immutable builds?

Immutable builds should be thought about in one key way: they are isolated environments. Building in isolated environments ensures that free of the residue of previous configurations, which can not only undermine the quality of our software (in the sense that it runs as expected), but also introduce inconsistencies that can be exploited by malicious actors.

Using immutable builds goes further than standard configuration management. Using configuration management tools can be useful but they don’t demand isolation — configuration drift is possible, and often very hard to track, particularly if you’re dealing with complex and scaled builds.

Fortunately, immutable builds can also be scaled up quickly — you simply need to replicate the immutable build.

Ensuring a sterile environment

In a sense, the need for immutable builds are a bit like the need for sterile work environments in a scientific setting.

Think of it this way: if the space and tools being used to carry out an experiment aren’t sterile, you could introduce unwanted variables that would undermine the validity of your experiment. Similarly, of your build environment isn’t immutable — ie. not sterile — you might well undermine the integrity of your software.

How can we implement immutable builds?

An immutable build environment can be based on containers or full virtual machines depending on the requirements of your tool chain.

Whatever your approach, what is important is that the entire build pipeline from toolchain configuration to signing the resulting binaries should be isolated from the rest of your network. Only after the binary has been signed will any tampering will be easy to detect.

Trust and immutable builds

When we talk about trust in terms of immutable builds we’re not so much talking about how a customer trusts a vendor. It’s more about the ability of the engineers to trust that what they’re trying to build is what they’re actually building.

Of course, this will ultimately have an impact on trust between users and vendors, and beyond that across the industry as a whole. Immutable builds, then, are the first step in ensuring that we have transparency and control over the highly sophisticated systems on which millions of people depend.

Reproducible builds: integrity in the build process

Immutable builds guarantee consistency in the environment in which we build software: on top of that we need to confirm integrity and guarantee trust in how software is actually built.

This can be done by something called reproducible builds.

What are reproducible builds?

Unlike the notion of immutable builds, reproducible builds is an idea that has cohered into a specific movement. At its heart, it is dedicated to making it easier to verify the code on which software runs.

This is particularly important when it comes to proprietary software. Although anyone can access the source code for open source tools and projects, proprietary software is typically pre-compiled when distributed. This makes it impossible to verify whether the binary code corresponds to the source code (pre-compiled doesn’t mean it’s still source code, it means that the source code has been bundled up in a way that makes compiling faster).

Reproducible builds allow engineers to check that the code in proprietary software matches up with the original source code — if any code has been changed, say as part of an attack, it’s easily identified. In other words, ensuring reproducibility allows a shipped binary to be checked against its source by a third party

This isn’t easily tackled, however. Due to a range of issues, such as untracked pre-built library binaries, profile based optimisations, and time stamps it’s difficult to guarantee reproducibility.

This doesn’t mean that it isn’t worth taking the time and care to actually do it.

The benefits of reproducible builds

The benefits of this approach in relation to the SolarWinds attack are obvious. Indeed, the risks acknowledged by the reproducible builds project are eerily familiar. Without reproducible builds in place “attacks on build infrastructure… would provide access to a large number of downstream computer systems. By modifying the generated binaries here instead of modifying the upstream source code, illicit changes are essentially invisible to its original authors and users alike.”

Putting reproducible builds into practice

The Reproducible Builds project highlights three key steps that are needed to implement its principles. First, it demands that the build system “needs to be made entirely deterministic”. In other words, the output of executed code is always exactly the same. Second, it demands consistency in the build environment — this comes back to immutable builds. And finally, it requires end users to be able to recreate that build environment to accurately validate that code.

What all these steps should ultimately do is normalise those issues — variables — to ensure consistent outputs. In this sense then, you can continue the science metaphor. Reproducible builds are about ensuring that not only is an experiment conducted in a sterile environment, but also that it is carried out in a way that is transparent and easy to replicate. If it’s not, then it’s almost impossible to properly determine what’s going on and what your results actually mean.

Conclusion: trust is built on an attention to detail

There’s no quick and easy fix for improving trust in the software supply chain. Moreover, immutable builds and reproducible builds are complex; they require a significant amount of additional effort and thinking. This is particularly true as we begin to adopt them. This could be tough in a culture that has valorized speed and delivery much more than it has reliability and security.

This isn’t to say that we need to move more slowly if we’re to build trust. We can still be agile, and we can still be adaptable. But it will require a change in attitude and culture — it will take time and energy to embed the processes and principles that ensure trust.

Given the scale of the SolarWinds attack, however, this has to be a price worth paying. Over time, as culture changes and demand grows, tooling designed to make such practices easier will likely emerge. This will democratize trust which is essential; it shouldn’t, after all, be exclusive to particular players in the field with money and resources.

It’s worth remembering Ken Thompson’s assertion — made back in 1984 — that “you can’t trust code that you did not totally create yourself.” If we follow his thinking, perhaps we need to acknowledge software and trust can only ever exist at a distance from one another.

“No amount of source-level verification or scrutiny will protect you from using untrusted code” he said. “As the level of program [sic] gets lower, these bugs will be harder and harder to detect. A well installed microcode bug will be almost impossible to detect.”

Perhaps we should have paid more attention back then, not so much to lapse into defeatism, but at least to do more and try harder to build software with greater integrity and care.