Spot Donation Service
In Dan’s recent blog post he asked “Attending a Hackathon when you cannot hack, how bad could it be?”. Not bad at all, it turns out. The aspirationally named Team Victory rose to the challenge, and won the .create(v=2.0) Cloudreach hackathon with an idea aptly titled Spot Donation Service. In this blog post, we will give you a high level technical overview of how we addressed the theme of the hackathon: Cloud with a Conscience.
The hackathon theme and rules
On the Friday, we received the theme of the hackathon as well as some examples to get the ideas flowing. The task was to create something to illustrate the Cloud with a Conscience theme through tackling an environmental issue or creating a solution to benefit vulnerable members of society. You get the idea. Giving something back.
The rules were simple. Build something, using at least one piece of new cloud technology (released in the past 6 months), which would be judged on closeness to the theme, completeness, innovation, and technical difficulty.
Spot Donation Service
We started by whiteboarding ideas on Friday evening, and eventually settled on what would become the Spot Donation Service (SDS).
Our goal was to take the excess spend in AWS (inefficient, unrequired, or underutilised capacity), and find a way to reduce this, whilst also giving back to charity. With a limited 48 hours to deliver a functional Minimum Viable Product (MVP), we decided to focus on EC2 Spot Instances. Below is a quick primer from AWS about what exactly Spot Instances are:
“Spot instances provide you with access to unused Amazon EC2 capacity at steep discounts relative to On-Demand prices. The Spot price fluctuates based on the supply and demand of available unused EC2 capacity.
When you request Spot instances, you specify the maximum Spot price you are willing to pay. Your Spot instance is launched when the Spot price is lower than the price you specified, and will continue to run until you choose to terminate it or the Spot price exceeds the maximum price you specified.”
Our idea was to create a new price, the charity price, which redefines the maximum spot price a user is willing to pay for an instance. The user will always pay the charity price, which will be set between the spot price and the On-Demand price. This means the user still saves money vis-a-vis an on-demand instance, but can donate the difference between the charity price and the spot price, allowing the user to both save money, but also give back to charity. This is shown graphically below:
From a business perspective, we proposed this could be a service Cloudreach offer as part of the reseller relationship with clients, allowing us to enable more socially responsible use of the cloud.
The Spot Donation Service MVP
With very limited time, we knew that we wouldn’t be able to implement and automate the entire process, i.e. from data collection through to billing. Instead, we chose to focus on generating, collecting, and visualising data. As our MVP deliverable, we aimed to create a dashboard which displayed what a Cloudreach client using the Spot Donation Service would see.
The tech behind the Spot Donation Service
Our design principles were to use as many new services as possible and to priortise services and products which we had never used before. We wanted to learn as much as possible, as well as create a great MVP. We also used serverless products instead of self-hosting and used open source products where possible. These principles and our MVP requirements resulted in us using the following technologies (bold indicates newly available services, italics represents technology new to our team):
- AWS Lambda – in a VPC, and running Python 3.6
- AWS S3 – for data storage before processing
- AWS EC2 – to host the below two servers
- Spot Instance Data Feed – provides hourly data about spot instance price data (including instance type, and prices)
- AWS Regional Pricing API– data from this was collected, but ultimately not used as part of the MVP dashboard
- Nginx / Let’s Encrypt / Route53 – to provide an easy and secure connection to Grafana
- AWS CloudFormer (still in beta) – to see how close it was to being able to fully replicate our manually generated (via console gui) architecture into CloudFormation.
This image illustrates how the architecture and services came together
- Users run spot instances, which generate spot instance data feed data. This data was reported as a CSV file into an S3 bucket (which must be in us-east-1 only).
- This data was then replicated to eu-west-1 (this was done to prevent accidental deletion of our data, as we manually deleted and re-copied data to the eu-west-1 bucket to test the Lambda).
- When a data feed data file is added into the eu-west-1 bucket, a Lambda (within VPC) is triggered. The Lambda parses the data file and pushes the resulting data to Influx.
- Influx and Grafana run on an instance within the same VPC as the Lambda. Grafana is set up to dashboard the data within Influx.
Overall, and in retrospect, this is a relatively simple architecture, and it met our design principles and goals well. As we purposefully chose unfamiliar technologies our progress was expectedly often slow and frustrating during the course of development, but as we set a realistic MVP goal we were able to achieve it, and we learnt a lot.
We have put together an assortment of retrospective comments on the technology:
- Working with Lambda can be very tedious – a framework definitely makes the process simpler. In this case, we used Zappa to package our Lambdas, rather than packaging the dependencies manually.
- Hard lessons were learned dealing with InfluxDB and Grafana requiring the data to be very specifically formatted. A key example is that if the time keys were not correctly inserted in a specific format, then the data wouldn’t be visualized properly.
- CloudFormer was able to accurately recreate the majority of our manually created resources. We found it easy to use, and would use it in the future as a starting point for moving manually created proof-of-concept infrastructure into a production ready template.
- James loves additional sugar
In summary, we were really happy to have built a working product and are even more proud that our MVP was judged by independent judges as the winner and to have embodied Cloud with a Conscious. We’d love to work more in this space so if you have any comments or questions, please reply back to this post.
Thanks, Team Victory
- Neel – https://github.com/rdkr – https://www.linkedin.com/in/neelradhakrishnan/ – https://twitter.com/neelr
- James – https://github.com/jmuldoon – https://www.linkedin.com/in/james-p-muldoon/
- Daniel – https://www.linkedin.com/in/danieljsmythe/
For more hackathon related posts, click here
What will you create?