Smart Monitoring with Cloudwatch Logs and Dashboards

Alberto Alvarez 18th January 2016

Why monitor? Well, there are plenty of reasons for you to monitor your applications and services. The most obvious one is to ensure that your application is available and your clients are able to use it. In addition, you want to make sure that the systems are performing well and that you can anticipate any problems before they occur.

It’s not all about waiting until the systems crash, but about using smart approaches and identifying certain patterns in order to prevent the same issue from happening again. For example, you could scale your infrastructure in response to certain metrics, or gain insight thanks to a post-mortem analysis.

In Cloudreach we leverage Splunk as well as other tools to provide our customers the best level of visibility in their environments and assist with some of the concepts described above.

In addition to your preferred monitoring and log analysis tools, you can (and should!) use Cloudwatch. Cloudwatch gives you extra insight into any of your applications running on AWS, giving you the advantage of the native integration with many AWS services.

Remember: Identify key metrics for each service behind your application, and then monitor them all!

You probably use Cloudwatch in one way or another to, for example, keep an eye on the capital spend of your infrastructure, trigger off Autoscaling alarms or to simply have a quick glance over the number of incoming requests on your ELB.

Let’s have a closer look at functionality you may not have used yet:  Cloudwatch Logs and  the newly introduced Cloudwatch Dashboards, and see how you could exploit both services in a web application scenario.

 

Cloudwatch Logs

Cloudwatch Logs, a feature released last year, allows customers to feed logs into Cloudwatch and then monitor those in near real-time. This is a very powerful component as customers can still take advantage of the usual Cloudwatch features, which are extended to the log monitoring aspect.

For example, I could have my application logs forwarded to Cloudwatch Logs, and set up alarms based on text patterns. These alarms then could trigger off other events across my AWS environment like SNS notifications or Autoscaling actions. The possibilities are endless and the effort is minimal as Cloudwatch is already featured to work seamlessly with other AWS services. Additionally, Cloudwatch Logs can be extracted and transformed into Cloudwatch metrics by using Metric Filters.

At the moment there are two main methods customers can use to feed logs into Cloudwatch:

  • Using the Cloudwatch Logs agent: This agent will run on your server and will allow you to send any logs files to Cloudwatch.  
  • From another AWS Service: Currently Cloudtrail is the only service able to feed logs into Cloudwatch Logs outside the box as described in this article. Hopefully other logs generated by AWS services, such as the ELB Access logs, will be soon natively available in Cloudwatch Logs.

 

Cloudwatch Dashboards

Very recently, AWS introduced Cloudwatch Dashboards, which enables customers to visualize in real-time any Cloudwatch metric by using dashboards. This feature is quite interesting and definitely adds an extra level of maturity to the Cloudwatch product. After spending some time playing with Dashboards here are my key take away points:

  • Single view, real-time for a collection of Cloudwatch metrics, across different regions.
  • Auto-refresh support.
  • A unique URL is provided for each dashboard so anyone from your team with access to the AWS Console can access the dashboards conveniently.
  • Ability to select particular intervals across multiple graphs in order to investigate spikes and correlate events.
  • In addition to Graph Widgets, customers can use Text widgets to, for example, hyperlink a relevant Log Stream from Cloudwatch Logs or link a runbook from your Wiki.

 

Right, let’s get hands on.

I created a two-tiered WordPress web application composed of an Apache web frontend and a MySQL backend. I provisioned 3 Linux EC2 Instances behind an ELB and a MySQL RDS instance. Route53 handles the DNS.

I started by deploying the Cloudwatch Logs agent on all the Web Servers and configuring the access logs for each instance to be sent to Cloudwatch Logs.

After that, I’ve created my dashboard called “Website-metrics-dashboard”, and added the first metric Graph Widget into it: the number of incoming requests that hit my ELB.

 

Next, I’ve added a couple of RDS metrics (DB Connections Count, CPU utilization) plus a combined metric for the Inbound Network for all my backend EC2 instances. The dashboard is certainly starting to take shape:

 

We can also integrate Route53 metrics such as HealthCheckStatus which are actually stored on a different region than my Web application. Just by swapping regions through the console, I’m now able to select metrics from a different region and integrate them into my dashboard:

 

 

The last addition is a Text widget. Text widgets use markdown language and, in this case, I will add a couple of links to assist the team managing this Web application:

 

The last step is to rearrange and organize the layout (by simple drag & drop) and adjust the Auto-refresh interval to 1 minute.

The result is a great looking dashboard that provides valuable real-time insight into my web application. I also performed a few load tests against my web application using loader.io, just to have some traffic patterns to look at. A feature I really like from dashboards is that you can look into specific data points across all your graph widgets by just selecting the specific time in one of the graphs.

For example selecting a particular spike with the cursor:

 

Will instantly zoom in all the graphs:

 

There are many more possibilities, but this is just an example on how Dashboards can provide your team with a single view on what’s going on in your web application.

 

Definitely a convenient and powerful new feature that complements the existing Cloudwatch abilities very well .

I hope you’ve taken away something useful from this post and, remember, if you need an expert hand to look after your AWS workloads, please reach out to Cloudreach, world’s finest AWS MSP Partner!

 

Why not check out our Slide Deck on this topic?