Driving IoT forward: from Data Cars to the real-world

This year at AWS re:Invent we debuted Data Cars, a project led by Raid Sulaiman and myself, and run by Ricki Lawal. Data cars adds real-time cloud backed telemetry to a Scalextric set. Data Cars streams acceleration data from IoT devices onboard the slot cars, through a local device which performs data aggregation and preprocessing, and from there into AWS where the data is analysed and visualised.

In this blog I highlight the key AWS technologies used in building Data Cars and how the demo relates to the real-world. This blog is part of a two part series on Data Cars, the second of which will examine the architecture and hardware in detail - so stay tuned!

Which AWS services made Data Cars possible?

  1. AWS IoT Core is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices. IoT Core can support billions of devices and trillions of messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely. - AWS IoT Documentation [1]
  2. AWS Greengrass is software that lets you run local compute, messaging & data caching for connected devices in a secure way. With AWS Greengrass, connected devices can run AWS Lambda functions, keep device data in sync, and communicate with other devices securely - even when not connected to the Internet. - AWS IoT Documentation [1]
  3. Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. - AWS Kinesis Documentation [2]
How do these services work together?

An IoT device, or Thing in the AWS lexicon, is managed by the AWS IoT Core, and can either talk directly to AWS over the internet or talk to another device which runs Greengrass and relays to AWS.

As familiar examples, these concepts are similar to the Amazon Echo, which talks directly to Amazon over WiFi / the internet, as in the former; or Philips Hue lights, which communicate with a hub using the ZigBee protocol and the hub communicates with the wider world over the internet, similar to the latter.

We used Kinesis Data Streams to read and write directly to the Kinesis service and to visualise our data, as explained in detail below. Other options available from Kinesis include Kinesis Firehose and Kinesis Analytics. Firehose manages the ingestion of the data stream into other AWS services, such as S3 or Redshift, for later processing. Analytics builds on top of Data Streams or Firehose to enable the processing and analysis of streaming data using SQL, to create near real-time metrics for dashboards or alerting.

Pictured: Neel racing the Data Cars at re:Invent 2017

What is the real-world relevance?

The technologies described above enable highly scalable data ingestion and real-time processing. The Greengrass model enables low power and inexpensive sensors to relay their data to the Greengrass Core, which handles the heavy lifting of pre-processing and shipping the data to AWS. Additionally, AWS continues to make the ability to process stored data more accessible with new services such as SageMaker [3], which reduces the barrier of entry for machine learning.

As we built this system out and talked with people visiting the demo, the range of possible real-world applications discussed continued to grow: inventory tracking in manufacturing and logistics, healthcare monitoring, traffic and transport management systems, energy monitoring, home or smart building automation, remote sensing, and more. It was great to see people at our booth realising how this technology could be leveraged in their field.

In the same way that the cloud enabled the efficient use of compute resources, the ability to gather and process real-time data about physical systems using the cloud enables improvements in their efficiency. Many current applications of IoT are limited in scale and scope vis-a-vis what these technologies enable, but industries such as manufacturing and transportation have emerged as early adopters [4]. Realising value from IoT becomes increasingly important as demand on physical infrastructure increases. For example, in the context of a smart city, being able to optimise traffic junctions to increase traffic flow is valuable as there may be no physical room for upgrades of lanes in the future.

Another observation from this demo is how several fields, including security, embedded systems, and data engineering / science are becoming increasingly intertwined for an end-to-end IoT solution. Whilst services such as those provided by AWS lower the barriers of entry, expertise in these areas is required in order to fully leverage these new technologies responsibly and with efficacy.

Embedded systems and hardware connectivity expertise is required to effectively scale IoT solutions [5], challenges around security have recently been highlighted by IoT botnets such as Reaper [6], and data expertise is required to derive value, particularly given that less than half of data currently generated is put to any use [7]. As with all emerging technologies, organisational change must drive IoT’s adoption internally, as society adapts to the new possibilities [4, 7].

For further reading on IoT technology and Data Cars, a full technical walk-through is coming soon to this blog. For further reading on industry analysis, I recommend checking out McKinsey’s analysis [7].


[1] https://aws.amazon.com/iot/

[2] https://aws.amazon.com/kinesis/

[3] https://aws.amazon.com/sagemaker/

[4] https://www.gartner.com/technology/research/internet-of-things/report/

[5] https://www.mckinsey.com/global-themes/internet-of-things/our-insights/the-future-of-connectivity-enabling-the-internet-of-things

[6] https://krebsonsecurity.com/2017/10/fear-the-reaper-or-reaper-madness/

[7] https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/how-can-we-recognize-the-real-power-of-the-internet-of-things

  • iot
  • aws-reinvent
  • data-cars