Chef And Retrieving Secrets From Vault Using The GCP Authentication Backend

Senior Cloud Systems Developer, James O'Hara, describes how he and his colleague Stuart Stone developed a Chef helper library to ease the pains of retrieving Vault secrets in the Google Cloud Platform.

While developing Configuration as Code for Google Cloud instances for clients, it became increasingly necessary to pull secrets securely from Vault in order to complete application bootstraps. It was decided to use the GCP Auth Backend for authentication so that no specific Vault tokens were stored in Chef Databags or the CI/CD engine. This would also be of use when using Chef-Solo. Although, at the time of writing, there is a Vault Chef cookbook available in Chef Supermarket to retrieve secrets, it doesn’t include the necessary methods to authenticate using the GCP Auth Backend.  

How GCP Authentication Backend Works

The below flowchart outlines the process required to successfully authenticate with Vault using GCP Auth.

How GCP Authentication Backend Works

So, as shown in the diagram, a Google Compute instance, that needs a secret from Vault, must first query the Google Compute metadata server to retrieve a signed JWT (JSON Web Token). The token is then presented to Vault by the GCE instance. Vault will then verify this JWT against the Google OAuth2 API. If successful, then Vault will issue an authentication token back to the GCE Instance. The GCE instance can then use this token to authenticate with Vault and retrieve whatever secrets are available to it (which is governed by a Vault Policy)

These steps are broken down as follows:

  • The client obtains an instance identity metadata token on a GCE instance.
  • The client instance sends this JWT to Vault along with a role name.
  • Vault extracts the kid header value, which contains the ID of the key-pair used to generate the JWT, to find the OAuth2 public cert to verify this JWT.
  • Vault authorizes the confirmed instance against the given role, ensuring the instance matches the bound zones, regions, or instance groups. If that is successful, a Vault token with the proper policies is returned.

If this process can be achieved from inside Chef, then the secrets would be stored directly in Ruby objects and subsequently consumed by any cookbooks/recipes.

Cookbook Development

In order to expedite the above process from within Chef, a cookbook has been developed which will use Ruby’s net/http libraries and the Vault library to complete the process. This is contained in a Chef resource called vault_secret. The resource requires that the chef url, the secret path, and the and the Vault role be passed to it. The resource will then call a helper library function called authenticate! which will also pull the Instance IAM Service account from the GCE metadata using Chef Ohai in order to form the JWT request. Once this has completed successfully, Vault issues an access token which is kept in memory by the official Vault Chef Library. The vault_secret resource will then retrieve the secret from vault at the path that was passed to it, and can be read using a :read function.  

Requirements
  • Vault must be running on a Google Compute Instance or Managed Instance Group
  • The Vault Google Compute Instance(s) must be running as an IAM service account which has permissions to query the Oauth API :
service_account_scopes = [

"https://www.googleapis.com/auth/compute",

"https://www.googleapis.com/auth/logging.write",

"https://www.googleapis.com/auth/cloud-platform",

"https://www.googleapis.com/auth/monitoring.write",

"https://www.googleapis.com/auth/devstorage.full_control",

]
  • A Vault Role should be created and bound to the IAM service account of the client GCE Instance.
  • A Vault Policy should be created to allow the above Role read-only access to the required Vault secret backend.
Testing
  • Testing of this cookbook was undertaken in a GCP Project with a Vault service deployed.
  • A test secret was written into Vault ( under secret/cheftest ) {'gregg'='turkington'}
  • A Vault role was created called chef which used GCP auth backend and was bound to a specific GCP IAM service account
  • A Vault policy was created allowing the chef role Read-Only access to secret/*
  • A test recipe was created that consumed the new authentication library, passing in the Vault URL, the secret path and the Vault Role to the vault_secret resource
  • The test recipe then writes this secret (key='gregg') to a file on the instance
  • Inspec then verifies the contents of the file is correct ('turkington')

Testing proved successful, when run. Also when the Vault policies were changed to prevent access, the recipe did error and wrote the Vault error responses directly to the console (eg '403 - access denied error', during the chef-solo run.  

Further Work

Additional resources, helper functions could be added to use GCP IAM, AWS EC2 and AWS IAM authentication either wrapped in the vault_secret resource, which would call separate helper functions, depending on a parameter that specifies the GCP Auth method.

To see the repo check it out on GitHub or Chef Supermarket.

Cookbook development by Stuart Stone and James O’Hara

  • gcp
  • chef
  • vault
  • google
  • recipe