Securing The Trusted Base
By Bob Aiello
Over the last several years, there have been many reported incidents where hackers have attacked banks, government agencies and financial services firms. Corporate security experts are hard-pressed to react in a timely manner to each and every attack. DevOps provides many techniques, including application baselining, build, release and deployment engineering, which are essential for detecting and dealing with these problems. This article discusses how to use CM Best Practices to establish secure application baselines which help verify that the correct code is running in production. Just as importantly, this fundamental practice enables IT personnel to discover if there are any unauthorized changes, whether they be caused by unintended human error or malicious intent.

Ken Thompson won the coveted ACM Turing award in 1983[1] for his contributions to the field of computing. His acceptance speech was entitled “Reflections on Trusting Trust” [2] and asked the question “to what extent  should one trust a statement that a program is free of Trojan horses?” After discussing the ease with which a programmer can create a program that replicates itself (and could potentially also contain malicious code), Thompson’s comments highlight the need to ensure that we can create secure trusted application baselines. This article will help you get started delivering systems that can be verified and supported, while continuously being updated as needed.

The secure trusted application base needs to start with an operating system that is properly configured and verified. Deploying applications to an untrusted platform is obviously an unacceptable risk. But applications themselves also need to be built, packaged and deployed in a way that is fully verifiable. The place to start is baselining source code in a reliable version control system (VCS) that has the capability to reliably track the history of all changes. A common source of errors is missing or ill-defined requirements, so traceability of requirements to changesets is fundamental. Baselines provide for the management of multiple variants (e.g. bugfixes) in the code and, more importantly, the ability to reliably support milestone releases without having to resort to heroic efforts to find and fix your code base.

The application build itself must be fully automated with each configuration item (CI) built with an embedded immutable version ID; these secure IDs facilitate the physical configuration audit, essential for ensuring that the correct code can be verified as having been successfully being deployed. The CIs should themselves be packaged into a container which is then cryptographically signed to verify the identity of the source and also verify that the container has not been compromised. The only way to get this right is to build the package in a secure way and deploy it in an equally secure manner. Deming was right when he pointed out that quality must be built in from the beginning and this is a fundamental aspect of the application build used to deploy a secure trusted base. The deployed application must be baselined in the runtime environment and constantly verified to be free of unauthorized changes. This approach fundamentally provides a runtime environment that is verifiable and actually facilitates the automated application deployment pipeline. The secure trusted base must be frequently updated to provide new functionality and also be capable of retiring obsolete or unwanted code. Creating the automation to reliably build, package and deploy code on a constant basis is essential to implementing a secure trusted base. While you cannot test quality into a system, building systems that are fully verifiable is very important. The deployment pipeline itself must be designed to be fully testable itself, so that any automation issues are immediately discovered and resolved.

Complex systems today also have many interfaces that must be understood and verified. This may involve checking the feed of market data or simply ensuring that correct ports have been provisioned and remain available for interprocess communication. In fact, monitoring the application environment is another crucial aspect of ensuring the secure trusted base. You do not want unnecessary ports opened, possibly resulting in a security risk, and you also do not want application issues due to a port, required by the application system, being closed. There are similar procedures to provision and verify the operating system
itself.

In the secure trusted base, we automate the build of all code components and embed identifiers into the code to make it possible to audit and verify the configuration. We also provide a reliable packaged container that cannot be compromised and we deploy in a way that is fully verifiable. Most importantly, we ensure that unauthorized changes can be detected and then remediated if they do occur. This may sound like a lot of effort and it actually does take a lot of work, but the costs of failing to take these preventable measures are frequently much higher as those firms who have been caught with “their code down” have painfully learned. The DevOps approach enables the development of these automated procedures from the very beginning of the application lifecycle which is really the only viable approach. More importantly, the DevOps approach to creating an automated deployment pipeline enables you to rapidly build, package and deploy applications using the same procedures in every environment.

My career has often been focused on joining a team that is having problems releasing their code. The first thing that I do is push for more frequent releases and, whenever possible, smaller in scope. We automate each step and build in code to test and verify the deployment pipeline itself. The codebase that can be quickly and reliably deployed is inherently more secure. Ideally, you want to start from the very beginning of the software and system development effort. However, more often than not, I have had to jump onto a moving train. Taking an iterative approach to scripting and automated the application build, package and deployment will help you create a secure and reliable trusted application base!

1. http://amturing.acm.org/award_winners/thompson_4588371.cfm
2. https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf