DevSecOps - Lessons from Las Vegas
My last post came from Seattle airport, but this was written over a craft ale in a Las Vegas hotel following a spectacular re:Invent conference where AWS, once again, demonstrated their truly insane capability to innovate at a rapid pace. While in Vegas, I attended a few really interesting sessions relating to security in AWS, so I thought a quick followup post to the recent DevSecOps post was in order.
First off, serious kudos to the team from Intuit for sharing their experiences openly - they provided a really interesting session. You should definitely watch the video yourself, but I’ll summarise a few key points below.
Earlier on in the day, an interesting point was made by a presenter quoting Mark Burgess’s founding views that control of environment will not be gained by forcing humans to do something by providing detailed instructions. Instead, one should use tooling to make it happen via declarative code. As obvious as this perhaps sounds now, how many of us have seen (or implemented!) giant security checklists in our organisations? Care to admit if they’re actually being followed? Or just left on the printer? Provided once during induction to new starters? Used at the end of the project when it’s too late?
Intuit re-stressed this point: you can’t just half-embed your security team in the development workflow and cross your fingers. You need automation and some serious thought about workflow. For example, you want automated alerts flowing to specific development team members, notifying them of security issues the moment they commit their code. Remember that good old nugget about catching software bugs early? Why not do something about it for security issues as well and functional bugs?
As with the example above of running checks on code as it’s developed, you need to proactively hunt for security issues in an automated fashion. The alerts should be delivered back to the people that can do something about them - the developers, a.k.a the guys earning the money - in a format they like, can read, and can understand. Consider a light spot of gamification to make this interesting for everyone. Intuit later presented a model using a centralised security accounts, with suitable IAM permissions of course, to access across their numerous accounts and monitor that approved baseline standards are in use. If the approved baseline is deviated from, this can be tracked. Who did it? Why? It’s possible to log everything. For example, how would you verify across thousands of instances that the next ShellShock-type vulnerability isn’t a risk in your estate? For network traffic monitoring, they noted the need for an expert third party tool. Their recommendation was FireEye. Here in Cloudy Towers, we’re proponents of AlertLogic, but the point remains the same: if you’re serious about security, you need a product like this in your processes. The concept of ‘Security Science’ was introduced, with an aim of removing fear from security and replacing it with mathematical proof: “If you can’t prove it, it doesn’t exist”. It’s a nice idea, focused around being able to provide your teams with the information they need to make decisions around topics such as password complexity (relatively obvious perhaps), but more interesting topics like "how often should I restart my instance?" or "should I switch from PHP to Ruby?". This was covered with a light touch at the end, but if you have any Stats gurus in your organisation, it could be worth a conversation. Finally, Intuit were honest enough to say, this was a tough journey, but one which was ultimately worth achieving. They’ve even been kind enough to open source some of their code - a great contribution to the community. Following announcements in Vegas, AWS are also making this journey easier for us all with new releases like Config and CodeDeploy.
Let’s make it happen. Let’s bring DevSecOps to the masses.