Two Blocks Away: Tangible Interface Empowered By Serverless Technology

Two Blocks Away (TBA) was the main attraction at Cloudreach’s booth during AWS re:Invent 2018 in Las Vegas. To play the game, players need to solve puzzles by selecting and arranging physical blocks. Over 400 participants took on the challenge during re:Invent 2018, and many were intrigued by its tangible interaction experience.

Let’s take a look at the game in action:

Our interactive demo Two Blocks Away is up and running 🎉🎉🎉 come to play our very fun game, test your AWS knowledge and be an architect for one day - you will also have a chance to win $150 Amazon voucher. See you at re:Invent booth 424! #awsreinvent #cloudreach pic.twitter.com/M7G6XckgjP

— Cloudreach (@Cloudreach) November 26, 2018

TBA players are presented with a series of on-screen, cloudy architecture puzzles. The aim of the game is to fill in the blanks with the correct AWS service printed on the physical blocks. With the blocks in place (with the correct sides facing up), the players push a button to submit the answer.

The combination of graphic interfaces and tangible interactions provides a playful and intuitive experience, much like playing with Lego blocks in our childhood. But tangible interfaces are traditionally considered challenging to implement for its reliance on hardware. TBA provides an interface that allows players to interact with by flipping and moving physical blocks, using only minimum electronic hardware (just a Raspberry Pi camera) and cloud-native serverless technology.  

The Two Blocks Away architecture

As introduced in a previous post, the key technology in TBA is Amazon Rekognition, an image recognition API offered by AWS. In order to provide a complete end-to-end game experience, some other AWS services were later introduced, such as AWS Greengrass and AWS IoT Device Shadow. This article will talk more in-depth about how these cloud services are used together to achieve tangible interaction in TBA.

The following diagram offers an overview of the final architecture.

TBA Architecture  

The web application is written in react.js. It consists of the main game screen, the sign-up page and the leaderboard. The web app is hosted in S3 as a static website and distributed by CloudFront.

For the physical devices, the diagram below shows how they are set up: A Raspberry Pi with a camera is mounted over the game screen and the blocks, and a physical button is connected to another Raspberry Pi.

TBA Set Up

How do components talk to each other?

In TBA, all the communication between different components are made through AWS IoT MQTT message broker. For example, the game screen subscribes to a topic called "TBA/Game/Start", whenever a player signs up, a new message is published to this topic, so the subscriber gets this message and a new game starts.  

  • AWS IoT Message Broker

AWS IoT is a collection of services, among which the Message Broker is one of the most fundamental services. The AWS IoT message broker is a publish/subscribe broker service that enables the sending and receiving of messages to and from AWS IoT. The message broker uses topics to route messages from publishing clients to subscribing clients, over MQTT protocol.

If you are not familiar with message brokers yet, it is important to know while most modern web applications adopt REST API, message broker is another major paradigm of communication, especially in IoT solutions. Supposing a scenario where you need to send a command to one of your Smart Home appliances, e.g. starting your washing machine on a mobile app, you probably don’t want your washing machine to expose a REST API to the Internet, nor would you want it to poll a service to check whether there is a new message coming in every second. This is where message brokers come into play. Instead of exposing your washing machine or constantly polling for new messages, your washing machine can subscribe to a trusted message topic, so when you send a command from the mobile app, the message is pushed to the washing machine.

There are other advantages of using message brokers as well, for example, it is intuitively event-driven, better supports one-to-many architecture and further decouples devices and services. Most importantly, they have a smaller network footprint. Unlike REST API or long-polling, which uses HTTP protocol, message brokers commonly use WebSockets or MQTT that has much smaller communication overhead. And compared with WebSockets, MQTT consumes less power and bandwidth, therefore is preferred by IoT devices.

What happens after the button is pressed?

When the button is pressed, the Raspberry Pi that is connected to the camera will take a picture and send to the cloud to process. In fact, at a lower level, this is a multi-step process:

  1. On Raspberry Pi #2: The script that listens to the “button press” event sends out a message to an IoT topic.
  2. On the cloud: The IoT Rule that listens to this IoT topic triggers a Greengrass Lambda function on the Raspberry Pi that the camera is connected to (Pi #1),.
  3. On Raspberry Pi #1: The Greengrass Lambda function takes a picture using the Raspberry Pi camera, uploads it to S3, and triggers a Lambda function on the cloud, which does all the subsequent calculations.

This process involves two other services in the AWS IoT suite:

  • AWS IoT Greengrass

Greengrass is a service that allows you to deploy and run Lambda functions at edge, in our example, the Raspberry Pi that the camera is connected to.

What is Greengrass? First and foremost, Greengrass is a device management tool that allows you to remotely manage the certificates and code (Lambda functions) on a collection of devices. Imagine you have thousands of IoT devices in a fleet, without remote device management, every time you need to update something, you may need to plug in, update and unplug every single device. It may sound too painful to be true, this pattern actually exists in many cases - think how you used to update your old car GPS? The remote deploy feature of Greengrass allows you to apply and replicate updates in a single action.

Greengrass is also a library (Greengrass Core) that you can install on devices, so Lambda functions can run locally on the edge, rather than on the cloud. The best application of this kind is machine learning at edge, e.g. rather than sending all the sensor data to the cloud to perform calculations, you can deploy the machine learning inference model to the edge devices to perform calculations locally.

Greengrass Lambda functions can access local device paths, which allows you to access things like camera, sensor and keyboard that are physically connected to the Greengrass Core device - this is how we use the Greengrass Lambda function to control the Raspberry Pi camera in TBA. However, while we didn’t have issue accessing common devices like Raspberry Pi camera and USB keyboards, we did have compatibility issue with some other non-standard devices. We hope the documentation on accessing local devices to be improved.

Another less known feature of Greengrass Core is that each Greengrass Core actually serves as a local message broker. So messages that can be processed locally can stay local without going through the Internet. This is extremely useful when sometimes not all local devices would have the ability to access the Internet, e.g. many Smart Home appliances can only connect to a local gateway. Also, messages sent to the Greengrass Core message broker can also be relayed to the cloud using the Subscriptions feature.  

  • AWS IoT Rules

IoT Rules is a service closely related to Message Broker mentioned before. It allows you to define rules that react upon the messages going through the broker. E.g. you can define an IoT Rule that once certain messages are published to a certain topic, invoke a Lambda function or forward it to Amazon Machine Learning to make predictions based on an Amazon ML model.

In the case of TBA, when a picture is successfully taken and uploaded to S3, a message is sent to the "TBA/Game/Capture" topic, and the IoT Rule that listens to this topic triggers a Lambda function on the cloud to analyse the image.  

How does TBA know whether my answers are correct?

Now we have the picture taken by the camera in the cloud, how do we know whether the answer provided by the player is correct or not? This was the most frequently asked question at the re:Invent booth.

Supposing the image below is captured by the camera, where the player placed the EC2 block on the top left and the S3 block on the bottom right, after calling the Rekognition DetectText API, we could tell:

  • The word "EC2" is detected and the coordinate is (x1,y1).
  • The word "S3" is also detected, and the coordinate is (x2,y2).

TBA Correct:IncorrectAn example of answers captured by the camera

After we get the centre coordinates, all we need to do is to compare these coordinates with the correct solution pre-coded in our programme. E.g. the following JSON script means we are looking for the text "EC2" at the (0.24, 0.47) location and the text "S3" at (0.65, 0.49).  

 

[{

name:'EC2',

position:{X:0.24,Y:0.47}

},

{

name:'S3',

position:{X:0.65,Y:0.49}

}]


An example JSON script that defines the correct solution

However, in the real world, we may still need to do a few other hacks to improve the reliability of the recognition. We are not going into the code level in this article.

How does TBA track the game state?

Throughout the game, certain information needs to be shared between components, for example, whether the game is idle or engaged - if there is already a game going on, we need to disable the sign-up page; current level and elapsed time, which are used to validate the answers and calculate the scores. This kind of information is called the game state.

In the beginning, we considered the use of DynamoDB to store the game state, however, we soon discovered DynamoDB is not a good fit - the game state needs to be updated frequently but can be short-lived, and the DynamoDB API is cumbersome but cannot easily “push” the updates to the clients.  

In fact, AWS IoT Device Shadow is almost exactly designed to store this kind of information.

  • AWS IoT Device Shadow

AWS IoT Device Shadow is essentially a storage space allocated for each IoT device. You can store any information in the Device Shadow in JSON format.

One of the greatest benefits of using Device Shadow is that it can be accessed through IoT Topics (and it also supports REST API), which means, a client can subscribe to and publish the updates to the Device Shadow. E.g. in TBA, the leaderboard subscribes to the update topic of the Device Shadow, so whenever the top 10 high scores are updated, the leaderboard page can fresh accordingly. In contrast, if not using the pub/sub pattern, the leaderboard page has to poll for any updates at a certain interval.

Additionally, IoT Device Shadow can be accessed when the device is offline (not connected to the Internet), and the metadata of each update, such as timestamp, is automatically managed. Although this feature is not used by TBA, it is quite useful in many IoT solutions. E.g. in freight management, it is highly likely that the vehicles will experience Internet disruption. Device shadow can track the vehicle status, e.g. the location last seen, even when the vehicle is temporarily offline, and re-synchronise when the vehicle comes online again. In fact, this explains why it is called "device shadow".  

Use of DynamoDB for the leaderboard

The user information and scores are stored in DynamoDB in TBA. DynamoDB is a fully managed NoSQL database on AWS. Comparing to the IoT services mentioned earlier, DynamoDB may be much more familiar to most people. Here we are only going to talk about two features of DynamoDB used in TBA, Streams and Secondary Index.  

  • DynamoDB Streams

Streams are useful when you want to implement features like "do something whenever a certain table is updated". In TBA, whenever the score table is updated, DynamoDB Stream triggers a Lambda function, which calculates the new top 10 high scores and updates the Device Shadow, before the leaderboard which subscribes to the Device Shadow is notified.  

  • Global Secondary Index

As a document database, DynamoDB is not typically used for analytical workloads, for aggregating data often means scanning the whole table. In fact, with carefully considered secondary indexes, DynamoDB can be proficient for some simple analytical workloads - e.g. getting the top 10 high scores in TBA is a classic use of Global Secondary Index - the use case is almost identical with the example provided in the DynamoDB documentation.  

Infrastructure as Code and automation

Infrastructure as Code and automation can greatly shorten development to deployment cycle in building serverless solutions. In TBA, Circle CI is used in conjunction with Github to automatically build, test and deploy the react.js application. Almost all the serverless infrastructure (apart from Greengrass, which is not yet supported by CloudFormation, however, can be automated through CLI), such as Lambda functions, S3 buckets, DynamoDB, IoT things and IAM roles are defined using Serverless framework and automatically deployed as part of the Circle CI jobs.  

Lessons Learnt

TBA is an interesting attempt to achieve tangible interaction using serverless technology. Some of the noticeable learnings are:

  • The high-level API of Amazon Rekognition enables reliable text recognition without the need to build custom machine learning models.
  • AWS IoT Core streamlines the device provisioning, management and authentication process. It is also supported by AWS CLI, SDK and CloudFormation, meaning that developers who are experienced with AWS can start work on IoT projects with little learning curve.
  • AWS IoT works well with the identity management tool Cognito, which allows developers to give IoT devices permissions to access other AWS services in the same way as what they would normally do for other types of applications.
  • AWS IoT Message Broker(Topics), IoT Rules and Device Shadow provide a practical pattern that allows developers to build industry-standard IoT solutions, and without the need to worry about underlying infrastructure.
  • AWS Greengrass is an innovative technology that empowers edge computing and eases the code deploy on large scale IoT fleet, although we still hope to see some improvements on the documentation, a more formalised SDK and compatibility of CloudFormation.
  • When working with serverless technology, Infrastructure as Code and automation can greatly reduce the development time, should be considered as a must-have.
  • If you are considering bringing a similar solution to a major event as we did, be prepared for a lagged, fragile and restricted network.

In conclusion, to build a tangible interaction solution like TBA traditionally would involve much lower level work, such as building custom machine learning capability and provisioning infrastructure. In contrast, serverless technology enables developers to stay focused on building the experience. Although TBA is just a fun experiment of cloud-native IoT technology, we could definitely see the potential of applying such technology to industry-scale solutions.

  • aws
  • serverless
  • dynamodb
  • serverless-technology
  • aws-reinvent
  • s3
  • aws-greengrass
  • aws-iot-device-shadow