Cloudreach Secrets: Implementing App Modernisation
In the previous post, we gave some background on Cloudreach Secrets - an application designed a few years ago by a Cloudreach employee for limited uses, but with a lot of potential.
In this post, we take on the challenge of modernising Cloudreach Secrets and outline our plan and process.
The moment I mentioned an opportunity to work on the modernisation of one of our internal tools, I was pleasantly surprised by the response within Cloudreach’s engineering teams. Immediately came the wacky and wonderful tinkerers fascinated by revisiting something long forgotten with the intention of dissecting it, understanding it, and recreating it in a fresh way. Let’s face it, talented engineers love a challenge.
Even more interesting was what our wizards were able to conjure. The same problem sent to two engineers yielded very different approaches and solutions. Testament to the creativity of the engineering mindset.
The original solution looked something like this:
As previously noted, a very traditional lamp stack, a bunch of web servers running on EC2 in autoscale groups with a mysql RDS instance backing it.
So what did some of the engineers come back with?
1) Jakub Zygmunt, our Amsterdam-based Cloud Architect, came back with the following:
2) Giulio Calzolari, our Munich-based Cloud Architect, came back with this:
Both very worthy solutions, and each had their advantages with elements worthy of implementation in a final solution.
Jakub, focused on modernising the backend logic, the application core, alongside the approach to the problem of sharing secrets in general. Giulio on the other hand looked at modernising the application by providing a suitable backend for an existing and familiar interface.
Very valuable learnings for any modernisation piece in general, what’s the strategy and what do you want to achieve? It is likely that different goals will demand varied approaches.
The decision came to pool both of the efforts and ideas, only to merge into something that looks like this:
Taking the best of both worlds, we would end up with a modern and extendable core, which supports the familiar interface out existing users are already comfortable with.
We would work to extend the core of this solution, to then allow for a CLI interaction for bulk load of secrets via more technical users into the system, whilst gradually adding features by updating the state machine powering the step functions.
We know you all love a good story, but also technical details are a must, so here are some of the modernisation approaches:
Core and Infra Modernisation
- Core logic rewritten in python - a microservices API as opposed to using PHPasswordPusher. Running as lambda functions behind API gateway allows us the flexibility of extending the core as well as the functions responding on various calls to the system.
- Orchestrated and deployed using "sceptre" / serverless instead of at the time pure cloudformation. Such an approach allows us to take advantage of some of the extended logic via templated language supported in Sceptre.
- CLI component developed using python and Google OAuth library - When configuring the CLI a browser redirect is used to authenticate against Google & a token is cached locally. This can be used for bulk load of secrets into the system via our admins.
- We added step functions to handle secrets expiration - Previously the code that checks if a secret expired was called only when someone requested the secret. By adding step functions we separate that functionality allowing us to expire the secret immediately after the time runs out.
- Added passphrase on secret mechanism - before storing the secret, we can optionally provide a passphrase that is used to encrypt the secret using Fernet - an implementation of symmetric (also known as "secret key") authenticated cryptography.
- Original webapp extracted as static site / leaving original jquery for UI interaction - we took the lift and shift approach to deliver a working solution in the shortest time, thus as the first step we reused as much code as possible.
- Jquery is old, Node seems to be the new king, so we bundled node libraries (request, aws4) using webpack and dropped them in S3.
- We added Cognito Federated Identities and STS credentials to enable storing secrets only for authorized members.
- Separated API code from UI code - the serverless microservices contain code only for the api and is UI agnostic, so we can use it from the web based view as well as the cli tool. The web code has been moved to static files hosted on Cloudfront.
- API Gateway protected via Cloudfront distribution combined with Lambda @ Edge which is used to redirect the user to a static page when a secret is expired and removed from the system.
- Storing static content in Cloudfront to utilise WAF on the solution should there be such a need - with AWS WAF in place we can implement a number of security protections, including the application of preconfigured templates that include a set of WAF rules, which are designed to block common web-based attacks such as bad bots, SQL injection, cross-site scripting (XSS), HTTP floods, and known-attacker attacks.
- AWS Certificate Manager is leveraged for cost efficient SSL certs which will automatically renew and rotate in line with the service offering.
With all of the changes above, we delivered a working poc in 1 week with a couple of days of preparation. We now have a serverless tool in place which is considerably cheaper than the original solution, whilst being simpler to support - removing any operational overheads in requirements for patching our machines over time. Sure there is still much to do to further productionize and extend the solution, but with the right approach results can be achieved quickly.
It’s important to remember App Modernisation is an ongoing process. Here are some of the other changes we would like to implement in time:
- Clean and sanitise the POC code.
- Write a formal installation manual & readme.
- Extend solution to allow storage of binary secrets.
- Introduce a feature to allow customers to share secrets with developers using a public/private key to add even more security to the system.
- Use SES to send links to shared secret consumers and creators.
- Remove jquery and rewrite UI in react or angular.js
We hope you found this insight interesting. For the next post, we plan on open sourcing our implementation… stay tuned!