Roomy - An IoT Project

We’re a creative bunch here at Cloudreach – when we’re not building cloud IT solutions for customers we’re usually busy making something cool for ourselves or experimenting with new technology. So when Amazon made their IoT service generally available in December last year, naturally we were excited.

 

What is IoT, and why is it important?

IoT stands for the "Internet of Things" – a new technology movement based on connecting existing devices to the internet to make them smarter. In the near future, smart devices built for applications in the home, industry, transportation networks, cities and elsewhere, will come with internet connectivity built in. This will allow devices to share operational information with the cloud, and with each other, to improve operational efficiency. Gartner predict that IoT security spending will reach $540bn annually by the year 2018, so this is definitely a trend to watch.

A classic example is the connected home. When you arrive home from work, your smart door will know that you’re at home and tell your smart bulbs to turn on the lights. Your radio could query your cloud music service and play your new favourite tune, while your smart kettle starts brewing your evening tea. All you’ll need to do is sit back and relax.  

 

Amazon IoT features

Amazon’s IoT service is designed to offer robust and secure connections from your devices to the cloud. Devices communicate with the lightweight MQTT protocol, messages are sent over secure TLS connections, which are encrypted with certificates. This means that the data sent to and from your device cannot be intercepted by a third party, and should arrive fast enough for realtime applications.  

AWS IoT tutorial demonstrates how the rules engine and device shadows work together

Amazon also provide some nice features which add value to their IoT platform. ‘Things’ and ‘device shadows’ represent your device in the cloud. When you interact with the IoT cloud your device talks to the ‘Thing’ endpoint. At times when the real device is offline, you code can still query or set variables stored on the shadow. Any changes to either the device or the shadow are synchronised when the real device comes back online.

The rules engine allows you to transform data streams coming from your device, push them to multiple sources and trigger events on other AWS services, like Lambda or Cloudwatch.

In order to connect a physical device to Amazon’s IoT service you first need a few elements in place:

  • A 'Thing'

This is the logical representation of your physical device in the cloud, it provide HTTP endpoints for the message brokering service, and stores shadow states for your device.

  • A certificate

This will authenticate your device and encrypt your messages, so that Amazon knows whether it really is your device that’s publishing data. The certificate comes as a public and private PEM key pair, and an X.509 SSL certificate signed either by Amazon or by your own root certificate.

  • An IoT security policy

If implemented properly this will ensure that your device can’t do anything it shouldn’t be doing, and will help to make your system secure.

Roomy - A cloudy IoT project

Maybe you’ve heard it all before, but when is it really going to happen? The answer is sooner than you think. We decided to build our own project to show how a connected IoT device can easily be built using current technology. We decided to create something that solves a practical problem while demonstrating how the value of small-scale embedded devices can be enhanced with cloud-based logic.

Finding a meeting room in the office can sometimes be a hassle. We use Google for Work to manage rooms and calendars, it allows you to pre-book rooms and track them on your calendar. But people don’t always stick to the plan - sometimes room bookings are made and forgotten about, other times people start using the rooms without checking if they were booked already, only to get evicted shortly afterwards.

Architecting Roomy

Our idea was to build a simple device which will provide Cloudsters with an easy way of managing room bookings. It should be easy to use, with a human friendly interface. It should enable you to see at a glance whether a room is free, and if so allow you to create a reservation at the press of a button. Live information about whether the room is occupied and booked could be broadcast to other devices (such as the lobby dashboard). Information about how the meeting rooms are being used and whether reservations are being wasted would be made available to provide useful insights. And, as we’re making it so easy for the user, "I didn’t have time to make a reservation" shouldn’t be an excuse for taking somebody’s meeting room… The process of creating a booking with roomy would follow a simple process.

  1. The user presses a button
  2. An MQTT message is sent to the cloud
  3. The rules engine triggers a lambda function...
  4. ...which queries the Google Calendar API…
  5. ...creates an event
  6. ...and updates the thing shadow to contain a return message
  7. Amazon’s IoT service ensures the message is delivered back to the device
  8. And the LCD display is updated by the device

 

As we wanted to know whether rooms were being used effectively we added a standard PIR motion sensor to the hardware spec. A message is sent up by MQTT every 5 minutes, this includes the number of times the PIR sensor was triggered in that time. The rules engine triggers the lambda to update room status information on screen when the message is received. This way the information displayed on screen is always up to date, and we can analyse data about the room usage.

Cloud Implementation

Create the thing endpoint

When you set up a device you can add up to three custom attributes. These are handy for keeping track of a large fleet of devices. New devices are added to the ‘thing registry’ -- a database which keeps track of all your IoT devices. You can search for entries in the registry by thing name, or by the custom attributes you define.

 

Amazon recommend keeping the thing name and client id the same for a tidy account.

Create a certificate

The certificate authenticates your device as it connects to the cloud. It’s a standard X.509 certificate system, the same as is used for standards like SSL. We used the one-click certificate feature, which lets you create and download the certificates instantly. Amazon also lets you register your own root certificate, so that you may sign and upload your own certificates. To send messages from your physical device to the thing endpoint you will need to activate the certificate and associate it with both the thing endpoint and a security policy. Then you can authenticate messages with that certificate. If you ever need to update or revoke a certificate you can do this at a later time.

Create a security policy

The last step is to create and attach a security policy to your thing. IoT security policies are similar to IAM policies, and use the same JSON syntax. All permissions are denied by default so you will have to write a document to allow your device access to the services it needs.

Architecting for Cloud IoT securely

Although it’s tempting to connect two devices with a single policy and allow access to “iot:*”, there are several reasons why you shouldn’t do it.

  1. Device shadows are built to ensure that state-altering messages are delivered to the device despite intermittent connections. If you connect more than one device with the same certificate this can cause inconsistencies.
  2. Best practice is to allow access only to data and services that your device actually needs. If you connect multiple devices with the same certificate they will all have access to each other’s data, which can cause issues with cross-publishing.
  3. Your devices are made to go out in the field - but there are hackers out there! If one of your devices is compromised and your certificates are stolen, an attacker will have access to any other devices and data sent using that certificate. So limit your risk by using separate certificates, topics and shadows.
  4. Shadow devices, certificates and policies are all free. You only pay for the number of messages you send and receive. Using one shadow per device will help ensure your shadow devices don’t send messages unnecessarily.

In our setup we used two policy documents to control what the IoT devices can do. One is a general policy which would be applied to a whole fleet of similar IoT devices, the second is individual to each thing. Each device should only be allowed to publish to its own topics and access its own shadow. Devices should not be allowed to connect with other Client IDs. Setting common permissions in a shared policy makes version control of the device permissions far easier than if we had replicated those permissions in a separate document for each device.

 

This is how we would structure our IoT resources for a fleet of roomies! The root policy restricts what topics and resources a roomy device can interact with. The policy is written in a generalised way using the ${iot:ClientId} intrinsic, which is replaced by the real client ID when a device connects to the service.  

 

Resource: roomy-root-pol

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": [

        "iot:Publish",

        "iot:Receive"

      ],

      "Resource": "arn:aws:iot:<account>:topic/roomy/request/${iot:ClientId}"

    },

    {

      "Effect": "Allow",

      "Action": "iot:Subscribe",

      "Resource": "arn:aws:iot:<account>:topicfilter/roomy/request/${iot:ClientId}/#"

    }

  ]

}

The first statement allows the device to publish and receive messages under the topic of ‘roomy/request/${iot:ClientId}’ (e.g. ‘roomy/request/roomy-00001’). This policy ensures that devices can only send and receive data on their own topics, the second allows the device to subscribe to messages addressed to itself. The device will also need permission to access the shadow via MQTT as we are using the shadow to deliver messages to the user, and to hold information which we want to persist between reboots (such as the location). The document below grants the device permissions to all the built-in AWS topics it needs to query and update the shadow thing. Like the roomy root policy, this document would be shared with all the devices in the fleet.

 

Resource: roomy-shadow-pol

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": [

        "iot:Publish"

      ],

      "Resource": [

        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/shadow/update",
        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/shadow/get"
      ]

    },

    {

      "Effect": "Allow",

      "Action": [

        "iot:Subscribe"

      ],

      "Resource": [

        "arn:aws:iot:<region>:<account>:topicfilter/$aws/things/${iot:ClientId}/shadow/update/accepted",
        "arn:aws:iot:<region>:<account>:topicfilter/$aws/things/${iot:ClientId}/shadow/update/documents",
        "arn:aws:iot:<region>:<account>:topicfilter/$aws/things/${iot:ClientId}/shadow/update/rejected",
        "arn:aws:iot:<region>:<account>:topicfilter/$aws/things/${iot:ClientId}/shadow/update/delta",
        "arn:aws:iot:<region>:<account>:topicfilter/$aws/things/${iot:ClientId}/shadow/get/accepted",
        "arn:aws:iot:<region>:<account>:topicfilter/$aws/things/${iot:ClientId}/shadow/get/rejected"
      ]

    },

    {

      "Effect": "Allow",

      "Action": [

        "iot:Receive"

      ],

      "Resource": [

        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/shadow/update/accepted",
        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/shadow/update/documents",
        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/shadow/update/rejected",
        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/shadow/update/delta",
        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/shadow/get/accepted",
        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/shadow/get/rejected"
      ]

    },

    {

      "Effect": "Allow",

      "Action": [

        "iot:GetThingShadow",

        "iot:UpdateThingShadow"

      ],

      "Resource": "arn:aws:iot:<region>:<account>:thing/$(iot:ClientId)"

    }

  ]

}

This last document is associated with only one certificate, it allows the holder of that certificate to connect only with a Client ID ‘roomy-00001’. By managing connect permissions in this way, certificate holders should not be able to impersonate other devices.

 

Resource: roomy-00001-pol

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": "iot:Connect",

      "Resource": "arn:aws:iot:<region>:<account>:client/roomy-00001"

    }

  ]

}

Permissions which are common between similar devices can be stored in only one document which is shared between all of the endpoints which use it. This approach makes version control of your policy documents simpler than if you had assigned one policy document per device.

Programming the Thing

Getting the physical device connected to the IoT service was fairly easy. Amazon provide support for devices running C or Javascript. There are also SDKs made especially for Raspberry Pi and Arduino Yun (which we chose). The SDK allows you to publish and receive messages, subscribe to topics and manipulate the shadow, all from within the Arduino sketch. You have the option of querying JSON messages on the Linux controller (which is good for memory management), or parsing messages on the sketch directly.The SDK comes with a couple of examples to get you started. The code for a really basic device could look something like the sketch below.

 

basic-iot.ino

 void setup() {

    myClient.setup(AWS_IOT_CLIENT_ID);

    myClient.config(AWS_IOT_MQTT_HOST, AWS_IOT_MQTT_PORT, AWS_IOT_ROOT_CA_PATH, AWS_IOT_PRIVATE_KEY_PATH, AWS_IOT_CERTIFICATE_PATH);

    if (myClient.connect() == ERROR_NONE) {

        success_connect = true;

        myClient.shadow_init(AWS_IOT_MY_THING_NAME);

        myClient.shadow_register_delta_func(AWS_IOT_MY_THING_NAME, message_callback);

        myClient.shadow_get(AWS_IOT_MY_THING_NAME, shadow_callback, 15);

    }

}

void loop() {

    snprintf(msg, MSG_LEN, "{"sensor":"%i"}"), sensor_count);

    myClient.publish("roomy/sensor", msg, strlen(msg), 1, false))

} 

On startup the sketch runs the setup() function and then executes the loop() function indefinitely. The sketch interacts with the Amazon IoT service using the IoT  object

1. Initialise

The myClient object is initialised from the aws_iot_mqtt class provided in the SDK with the setup() and connect() functions. Arguments (AWS_IOT_CLIENT_ID, …) are defined as macros in your aws_iot_mqtt.h file, which are copied from the IoT dashboard.

2. Connect to the message broker

The connect() method connects the device to the client. If the result is good then we proceed. Otherwise retry.

3. Register some callback functions to react to incoming messages  

 

<span style="font-weight: 400;">The shadow_init() method will get the SDK ready to interact with the device shadow. Then the shadow_register_delta_func method is used to register a message callback function. This function is invoked whenever a shadow update message is received from the broker. You must provide your own function to update the state of your device and report back to AWS. A simple message callback could look a bit like this</span>
 void message_callback(char* src, unsigned int len, Message_status_t flag) {

    if (flag == STATUS_SHADOW_ACCEPTED || flag == STATUS_NORMAL)

    {

      // Copy the state from the message
      if (flag == STATUS_SHADOW_ACCEPTED) {

         if (myClient.getDesiredValueByKey(src, “message”, buffer, 32) == ERROR_NONE) {

            strncpy(current.message, buffer, MSG_LEN);

         }

      } else if (flag == STATUS_NORMAL) {

         if (myClient.getDeltaValueByKey(src, “message”, buffer, 32) == ERROR_NONE) {

            strncpy(current.message, buffer, MSG_LEN);

         }

      }

      // Report the state back to AWS

        snprintf(buffer, BUFFER_LEN, "{"state":{"reported":{"message":"%s"}}}", current.message);

      myClient.shadow_update(AWS_IOT_MY_THING_NAME, msg, strlen(msg), NULL, 5));

    }

}

First the function checks whether the message status flag was successful. The value in the flag determines how the message should be read. Shadow query responses will have responses in the form of state/desired/key1... , whereas shadow updates have messages in the form of delta/key1…, so it’s important to use the right method depending on your message. Also, when you’re using the getDeltaValueByKey function it’s important to check the return value as it will clear the target buffer if there was a read error.

It’s important to report the state back to AWS, or it will assume that the update wasn’t delivered and will keep sending requests down to the device. We do that by constructing a simple JSON message and sending it back up to the broker with the shadow_update function.

4. Query the state of the shadow to initialise the device state

Finally a request is sent to the broker to query the state of the device shadow. If accepted the broker will send back a message containing information about the shadow device state.

5. Send data to cloud

Once the device is set up you can start sending data up to the cloud! Simply create a formatted JSON message and send it with the client.publish method.

We found it useful to define a typedef struct for the device state. This would also enable you to do things like copy and store previous states using memcpy, if that was a requirement in your application.

 

#define MSG_LEN 32

typedef struct {

        char location[MSG_LEN];

        char message[MSG_LEN];

        char owner[FLAG_LEN];

    } state;

state current;

state previous[3];

We added some buttons and a character based display so that users could interact with the device. The sketch is programmed to publish a message to the topic roomy/request/roomy-00001 when the button is pressed, with a request for time in the room. This triggers a lambda function through the IoT rules engine, which reserves the room on the calendar.

The Backend

We wanted our roomy device to make use of the Google calendar API to check whether the specified room was available. The idea is to allow the users to see at a glance whether the meeting room has been booked, and easily let other people know that they intend to use it.

The AWS IoT platform allows you to manipulate and distribute your device’s messages using their rules engine. We used it to trigger a Lambda function, but you can also send messages to SQS, SNS or Kinesis, store them in DynamoDB and more. You can even filter which messages trigger an event with an SQL-like syntax.

Google APIs validate with OAuth. In a normal web based application you would redirect the user to Google’s authentication page to enter their login details - however the lack of a web browser in this application complicates things. To work around this we stored the token in a JSON file stored in an S3 bucket, and created a two-stage authentication procedure.

This python code snippet demonstrates authentication using the oauth2client pip package, and initialising the calendar API service object.

import boto3

from oauth2client import client

from oauth2client.file import Storage

 

S3_client = boto3.Service().client('s3')

s3_client.download_file('roomy-bucket', 'auth.dat', '/tmp/auth.dat')

scopes = ['https://www.googleapis.com/auth/calendar']

storage = oauth2client.file.Storage('/tmp/auth.dat')

credentials = storage.get()

if not credentials or credentials.invalid:

flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE,

scope=scopes,

message='missing client secrets file')

# Don't refer to redirect URI

flow.redirect_uri = client.OOB_CALLBACK_URN

flags.noauth_local_webserver = True

print("Go to: {0}".format(flow.step1_get_authorize_url()))

if token is None:

raise NameError('You must go to the URL, authorise and restart with the token provided as an argument.')

else:

print('Authenticating')

credentials = flow.step2_exchange(token, http=None)

print('Storing credentials to auth.dat')

storage.put(credentials)

credentials.set_store(storage)

http_auth = credentials.authorize(Http())

service = discovery.build('calendar', 'v3', http=http_auth)

s3_client.upload_file('/tmp/auth.dat', 'roomy-bucket', 'auth.dat')

return service  

 

When we first run the Lambda function, the output asks me to go to the authentication URL. Because the redirect URI and noauth_local_webserver flag are set, Google displays the token on their page instead of redirecting me to another site. This means we can run the function again with the token as input to store it in the S3 file.

The token itself will expire in an hour, but Google don’t expire their refresh tokens. So we won’t need to repeat the process unless we revoke access from the Google account or change the scope of the Lambda function. Now that the API service object is created, we have access to the Google Calendar API! Communication is handled with JSON objects. This code snippet shows how you can query the freebusy interface to check if a room is available:  

 

def checkAhead(service, minutes, room, tStart):

    tEnd = tStart + datetime.timedelta(minutes)

    request = {

        "maxResults": 1,

        "timeMin": tStart.isoformat(),

        "timeMax": tEnd.isoformat(),

        "items": [

            {

                "id": room,

                "resource": True

            }

        ]

    }

    availability = service.freebusy().query(body=request).execute()

    busy = availability['calendars'][room]['busy']

    return busy

If the room is free, we can make a new event and add the room resource to indicate that it is being used.

def addEvent(service, user, room, minutes):

    tStart = datetime.datetime.now(tz=pytz.utc)

    busy = checkAhead(service, minutes, room, tStart)

    end = start + datetime.timedelta(minutes=minutes)

    if (len(busy) > 0):

        # There is a clash

        clash = availability['calendars'][room]['busy'][0]

        print('Meeting room occupied. Clash: {0}'.format(clash))

    else:

        # Create the event

        event = {

          'summary': 'Roomy - Impromptu Booking',
          'location': 'Cloudreach HQ',
          'description': 'Created in Google API for Python',
          'start': {

              'dateTime': tStart.isoformat(),

              'timeZone': pytz.utc.zone

            },

          'end': {

              'dateTime': tEnd.isoformat(),

              'timeZone': pytz.utc.zone

            },

          'attendees': [

              {

                'email': room,

                'resource': True

              },

              {

                'email': user

              }

            ],

          'reminders': {

              'useDefault': False,

              'overrides': [],

            },

        }

        event = service.events().insert(

            calendarId='primary', body=event).execute()

        print('Event created: %s' % (event.get('htmlLink')))

        print('you have {0} mins'.format(minutes))

The lambda function then updates the device shadow state to include a response message. We did this in three lines using the boto3 library.  

botoClient = boto3.Session().client('iot-data')

state = {'state': {'desired': {'message': 'You have 15 minutes'}}}

botoClient.update_thing_shadow(thingName='roomy-thing', payload=json.dumps(state)) 

Delivering the message back to the device using the shadow state feature means that AWS IoT service will endeavour to get the message back to the device. Even if the connection drops out or the device loses power it will still be able to get its desired state from the device shadow hosted on the cloud. Remember to give your Lambda function access to the Amazon IoT service! The role you make will need to have access to update shadow states. The lambda role policy contains three statements. The first allows it to update the shadow state, the second allows it to write execution logs in CloudWatch, and the third allows it to store and retrieve the token file in S3.

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": "iot:UpdateThingShadow",

            "Resource": [

                "arn:aws:iot:<region>:<account>:thing/roomy-00001"

            ]

        },

        {

            "Effect": "Allow",

            "Action": [

                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],

            "Resource": [

                "arn:aws:logs:*:*:*"

            ]

        },

        {

            "Effect": "Allow",

            "Action": [

<span style="font-weight: 400; color: #339966;">                "s3:GetObject",</span>
<span style="color: #339966;"><span style="font-weight: 400;">                </span><span style="font-weight: 400;">"s3:PutObject"</span></span>

                ],

            "Resource": [

                "arn:aws:s3:::roomy/auth.dat"

            ]

        }

    ]

}

There is also a policy on the bucket which allows the lambda function to read and write the token file.

{

	"Version": "2012-10-17",
	"Id": "RoomyS3BucketPolicy",
	"Statement": [

		{

			"Effect": "Allow",
			"Principal": "*",
			"Action": [

				"s3:PutObject",
				"s3:GetObject",
				"s3:DeleteObject"

			],

			"Resource": "arn:aws:s3:::roomy/auth.dat",
			"Condition": {

				"ArnEquals": {

					"aws:SourceArn":

"arn:aws:lambda:<region>:<account>:function:<lambda-name>"

				}

			}

		}

	]

}

 

Testing the device

When we press the button this message gets sent up to the broker. That triggers the custom rules engine event, which in turn triggers the lambda function.


{ "room": "LON-MR-TEST", "user": "Anonymous", "thingName": "roomy-00001", "query": "booking", }

The CloudTrail logs indicate that the lambda executed properly, and updated the state. We can tell the shadow state has been updated as there are return messages on the browser also.  

And within a second or so the message is displayed on the LCD panel.    

And now, we have completed the communication loop. The device can send a message up to the broker, which invokes the lambda function, which queries the Google Calendar API and sends the response message back down to the device to display to the user.  

 

Summary

Cloudsters here in London are really excited to see some homebrew technology around the office. People have tried it and really like how it integrates well with our existing system. Some have asked me about adding features to connect it with our company dashboard, and doing analytics on the sensor feed - maybe this could be sprint two.

We like the fact that Amazon’s IoT service uses the non-proprietary MQTT protocol, one of the main criticisms of previous work in IoT is a lack of cross-compatibility so this is a step in the right direction. The AWS IoT platform has been built with security and scalability in mind, it integrates perfectly with other AWS services, so you can enhance simple embedded devices with the power of cloud-scale computing. Now absolutely anyone can build their own personal IoT cloud!

I’m really happy that we had the opportunity to experiment with technology on the cutting edge. Here at Cloudreach, we like to encourage personal development and creativity. It’s all part of being one step ahead and promoting personal growth - two of our core values. People who work for us or with us understand that we do things a bit differently, which makes for a refreshing and enjoyable company culture. I’ve only been here a few months but I already feel part of a team of well respected international cloud experts. The future’s looking Cloudy!  

 

Not if. When.  

 

Useful Links
http://docs.aws.amazon.com/iot/latest/developerguide/authorization.html For writing IoT access documents

http://docs.aws.amazon.com/iot/latest/developerguide/iot-dg.pdf The AWS bible on all things IoT

https://github.com/aws/aws-iot-device-sdk-arduino-yun If you’re using an Arduino Yun you will want to check this out

https://www.arduino.cc/en/Main/ArduinoBoardYun About the Arduino Yun

https://developers.google.com/google-apps/calendar/v3/reference/ Google Calendar API - you can also find information on other Google APIs on this site

http://www.libelium.com/top_50_iot_sensor_applications_ranking/ Some interesting IoT applications

  • iot