Sceptre: a tool for driving AWS CloudFormation

Henry Bell 1st February 2017

Sceptre is a tool for driving AWS CloudFormation, which has recently been open-sourced by Cloudreach. The project and source code can be found on GitHub.

 

Background

If you want to get straight to the technical nitty-gritty — feel free to skip this section…

At Cloudreach we have many years of experience building elegant, resilient and cost-effective platforms in AWS. As well as these three key qualities, we’ve always placed great importance on repeatability, i.e. being able to create cookie-cutter clones of environments and environment components, rebuilding as and when required.

CloudFormation
is the provider-intrinsic facility for AWS infrastructure orchestration, and is key to achieving this goal of repeatability. Cloudformation relies on two industry standard document formats:   YAML and JSON to capture the configuration information and this presents some challenges if you are a cloud developer. For example JSON doesn’t allow comments, also neither of JSON/YAML allows the addition of complex logic available natively in programming languages such as Python.

This drives the requirement to generate templates programmatically, i.e. treating them not as source code but instead as build artifacts that are not intended for direct human consumption. As well as this, breaking an environment down into multiple component stacks is critical to limit the blast radius of any potential issues, and this can be tricky to do when managing multi-stack environment dependencies by hand.

Features

 

To help improve customer’s experience when using Cloudformation, Cloudreach developers embarked on developing Sceptre. Sceptre allows customers to provision, modify, and destroy Cloudformation templates in a predictable & repeatable manner, allowing developers to concentrate on building better environments. As a common platform used to drive internal code reuse, Sceptre allows for tried and tested code to be used across projects, customers and environments. Additional and extensible features build on the facilities offered by raw CloudFormation, which adds to the richness of the AWS automation and orchestration ecosystem.

As a tool for creating and managing the lifecycles of CloudFormation stacks, Sceptre offers the user all CloudFormation stack primitives (such as create, update, delete, and so on), but builds on these with the concept of environments, each comprised of per-environment configuration data and a set of common templates. Similar operations can be performed on entire multi-stack environments as on the component stacks.

Runtime dependency resolution is used to manage dependencies between stacks; this means that environment components will be built in the correct order, with multithreading taking care of building multiple stacks in parallel when no such dependencies exist, which means that environment build-time can be reduced.

To support good practice in environment separation, Sceptre can operate cross-AWS-account, assuming roles in each target account and thus removing the need for individuals to manage this. Sceptre fully supports CloudFormation change sets — which show in advance of an update any changes to resources that will be performed — helping to avoid any nasty surprises when performing stack updates.

Extensions

Sceptre has two facilities for extension: resolvers and hooks. Sceptre custom resolvers allow for dynamic values to be pulled from AWS (or anywhere, really) and injected into stack configuration data. A Sceptre hook is a generic piece of code that can be used to perform actions before or after any operation, thus supporting actions that may not be possible with CloudFormation (for example, working with AWS resources not yet supported, or operating completely outside of the platform, such as updating third-party DNS). Once registered, Sceptre handles the calling of these resolvers and hooks transparently.

Written in Python, Sceptre allows you to write templates in JSON or YAML as well as use dynamic templates generated using Troposphere. Sceptre has a principal aim of being as unopinionated as possible, and doesn’t force you into writing your code in any particular style. It can be used from the command line and as a Python module on any platform, and is designed to be used from build/CI servers as well as from development machines.

 

Alternatives

Although in some senses similar in functionality to other frameworks that cater for dynamic template generation and management of CloudFormation stacks (such as Stacker, StackMaster and cfn-flow), as mentioned above, we wanted Sceptre to be as unopinionated as possible and allow developers to use whatever software engineering or environment composition patterns they want. To have the tool written in Python was a requirement driven by both existing AWS tooling in use and Cloudreach developer preferences, and given that Sceptre is used by many of our enterprise customers, to have automated unit and integration tests in place is a must. Finally, inbuilt cross-account operation and extensibility via hooks were strongly desired features as well.

 

Getting Started

To get started using Sceptre, the documentation is here. The documentation contains a walkthrough, which is a good place to get started. If you’d like to contribute, of course pull requests are welcome, but we suggest getting in touch first before spending time implementing new features as per the contributors guide. You can contact the team via sceptre@cloudreach.com or the GitHub issues page.

Later posts will cover some useful patterns and code that we’ve developed — for now we hope you get some value from Sceptre and look forward to hearing from you.

 

Henry Bell

James Routley

Sean Rankine

 

With thanks to:

Adrian Fernandez

Amy Bruce-Gardner

Luke Plausin

Marius Galbinasu

Matthew Taylor

Max Eskenazi

Oliver Van Goethem

Yvonne Beumer