Device Shadows – Part 1 (MQTT Topics and REST APIs)

Pradeep Singh | 24th Mar 2017

Device Shadow (or Thing Shadow) is a unique feature of AWS IoT Platform. It can help you design robust IoT Solutions, even for the devices with constrained and intermittent Network connectivity. Let’s explore more about shadows in this article.

Basics of AWS IoT Device Shadows:

A Device Shadow is a persistent virtual ‘shadow’ of a Thing defined in AWS IoT Registry. Basically, it’s a JSON State Document that is used to store and retrieve current state information for a thing. You can interact with a Device Shadow using MQTT Topics or REST API calls. The main advantage of Shadows is that you can interact with it, regardless of whether the thing is connected to the Internet or not.

1. Interacting with Device Shadows:

AWS IoT Platform allows you to manage Device Shadows using following two methods –

1. MQTT Topics
2. REST APIs

The following diagram shows a very simple interaction between the Things / Apps and the AWS IoT Thing Shadow (For accessing Shadows on AWS IoT, your devices/apps must be authenticated and should have correct Security Policy in place. To keep the diagram simple Authentication and Authorization blocks are not shown here) –

---

1.1 MQTT Topics for Device Shadow:

AWS IoT has reserved some MQTT Topics for different Shadow operations. Let’s explore these topics –

1.1.1 Update (or Create ) Device Shadow Topics:

Update topic creates a thing shadow if it doesn’t exist, or updates the content of a thing shadow with the “desired” or “reported” state data provided in the request. Messages are sent to all subscribers with the difference between “desired” or “reported” state (using delta topic).

$aws/things/THING_NAME/shadow/update	Publisher: Your IoT Application / Device Subscriber: AWS IoT Shadow Service Payload: Desired or Reported State Doc (JSON) Description: Publish to this topic to update/create the thing shadow.
$aws/things/THING_NAME/shadow/update/accepted	Publisher: AWS IoT Shadow Service Subscriber: Your IoT Application / Device Payload: Desired or Reported State Doc (JSON) Description: AWS IoT publishes “Reported” or “Desired” portion of the State Document to this topic, on accepting the update request.
$aws/things/THING_NAME/shadow/update/rejected	Publisher: AWS IoT Shadow Service Subscriber: Your IoT Application / Device Payload: Error Message (JSON) Description: AWS IoT publishes an Error Message (JSON Format) to this topic when it rejects update request.
$aws/things/THING_NAME/shadow/update/documents	Publisher: AWS IoT Shadow Service Subscriber: Your IoT Application / Device Payload: State Document with Previous and Current States (JSON). Description: Regardless of Delta, AWS IoT publishes a State Document with “Previous” and “Current” State information to this topic whenever an update to the shadow is successfully performed.
$aws/things/THING_NAME/shadow/update/delta	Publisher: AWS IoT Shadow Service Subscriber: Your IoT Application / Device Payload: Delta State Document (JSON) Description: AWS IoT publishes a response Delta State Document to this topic when it accepts a change for the thing shadow and the request state document contains different values for desired and reported states. (When the Device Shadow is created for the first time with a “Desired” state, Delta is published on this Topic. However, if the Device Shadow is created with “Reported” state, Delta will not be published.)

Following flowchart explains how AWS IoT Shadow service will leverage all the MQTT Topics related to “Update” operation –

1.1.2 Get (Query) Device Shadow Topics:

Get topic retrieves the latest state stored in the thing shadow (for example, during start-up of a device to retrieve configuration and the last state of operation). This method returns the full JSON document, including metadata.

$aws/things/THING_NAME/shadow/get	Publisher: Your IoT Application / Device Subscriber: AWS IoT Shadow Service Payload: Empty (Null) Description: Your Application can publish an empty message to this topic to get the thing shadow. Shadow will be returned to “$aws/things/THING_NAME/shadow/get/accepted” Topic.
$aws/things/THING_NAME/shadow/get/accepted	Publisher: AWS IoT Shadow Service Subscriber: Your IoT Application / Device Payload: Complete State Doc (JSON) Description: AWS IoT publishes a response state document to this topic when returning the thing shadow.
$aws/things/THING_NAME/shadow/get/rejected	Publisher: AWS IoT Shadow Service Subscriber: Your IoT Application / Device Payload: Error Message (JSON) Description: AWS IoT publishes an error response document to this topic when it can’t return the thing shadow.

Following flowchart explains how AWS IoT Shadow service will leverage all the MQTT Topics related to “Get” operation –

1.1.3 Delete Device Shadow Topics:

Delete topic delete a thing shadow, including all of its content. This removes the JSON document from the data store. You can’t restore a thing shadow you deleted, but you can create a new thing shadow with the same name.

$aws/things/THING_NAME/shadow/delete	Publisher: Your IoT Application / Device Subscriber: AWS IoT Shadow Service Payload: Empty (Null) Description: Your Application can publish an empty message to this topic, to delete a thing shadow.
$aws/things/THING_NAME/shadow/delete/accepted	Publisher: AWS IoT Shadow Service Subscriber: Your IoT Application / Device Payload: Deleted Version and Timestamp (JSON) Description: AWS IoT publishes a message to this topic when a thing shadow is deleted.
$aws/things/THING_NAME/shadow/delete/rejected	Publisher: AWS IoT Shadow Service Subscriber: Your IoT Application / Device Payload: Error Message (JSON) Description: AWS IoT publishes an error response document to this topic when it can’t delete the thing shadow.

Following flowchart explains how AWS IoT Shadow service will leverage all the MQTT Topics related to “Delete” operation –

For more details refer to the following link –

Device Shadows Data Flow

Using Device Shadows

Device Shadow MQTT Topics

---

1.2 REST APIs for Device Shadows:

AWS IoT platform allows you to access Device Shadows using REST APIs too. The REST APIs can pass through AWS API Gateway and can perform operations like Shadow Update (or Create), Get, and Delete. Let’s explore it in detail here –

1.2.1 URI/URL for Shadow REST APIs:

All the REST API’s for Device Shadows will have a common URI (Replace “Endpoint” and “ThingName” according to your AWS IoT Account) –

https://ENDPOINT/things/ThingName/shadow

1.2.2 REST API Authentication:

First of all, you need to create a new User from AWS IAM Module to get hold of “Access Key” and “Secret Key”. This IoT API user must have one of following permissions –

• AWSIoTDataAccess – Read / Write Permission for Device Shadows
• AWSIoTFullAccess – Read / Write / Delete Permission for Device Shadows

You need these Keys to sign the Authentication information. AWS uses Signature Version 4 process for sending the Authentication information (keys) with REST API calls. All the REST calls fRefer to the following link for more information about this process – http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html

1.2.3 HTTP Headers:

1. Content-Type: (application/x-amz-json-1.0)Use it for Update / Create Request with JSON Payload.
2. Host: (You AWS IoT Endpoint) Use it for all Shadow Requests (Get, Update/Create, Delete).
3. X-Amz-Date: (UTC and in ISO 8601 format: YYYYMMDD’T’HHMMSS’Z’. For Ex: 20170403T104958Z) Use it for all Shadow Requests (Get, Update/Create, Delete).
4. Authorization: Use it for all Shadow Requests (Get, Update/Create, Delete). For details on this refer to this Link.

1.2.4 Get Shadow:

Use HTTP GET method without any payload to fetch Device Shadow. The following diagram shows a simple get shadow HTTP Request. (More details at Link).

1.2.5 Update/Create Shadow:

Use HTTP POST method with JSON Payload to Update/Create Device Shadow. The following diagram shows a simple Update shadow HTTP Request. (More details Link).

1.2.6 Delete Shadow:

Use HTTP DELETE method without any payload to delete a Device Shadow. The following diagram shows a simple delete shadow HTTP Request. (More details at Link).

For more details on Device Shadow REST APIs Refer to following links –

HTTP Error Codes

Device Shadow RESTful API

Signing AWS API Requests

---

1.3 Shadow Operation JSON Examples:

You can publish following JSON message to MQTT Topic “$aws/things/THING_NAME/shadow/update” to update the Device Shadow with new desired state of “Light1” i.e. “OFF” –

{
 "state": {
 "desired" : {
 "Light1" : "OFF"
}}}

On successful Shadow update, If the previous “reported” state is “ON”; AWS IoT will publish Desired Sate to Delta Topic “$aws/things/THING_NAME/shadow/update/delta”.

The device that is subscribed to Delta Topic, will perform the requested operation (Turn On the light in this case), and report back the new State to AWS IoT Device Shadow, using Update Topic “$aws/things/THING_NAME/shadow/update” with following JSON message –

{
 "state": {
 "reported" : {
 "Light1" : "ON"
}}}

Your Shadow State Document may look like this –

{
 "state":
 {
 "desired":
 {"Light1":"OFF"},
 "reported":
 {"Light1":"ON"},
 "delta":
 {"Light1":"OFF"}
 },
 "metadata":
 {"desired":
 {"Light1":
 {"timestamp":1490540278}
 },
 "reported":
 {"Light1":
 {"timestamp":1490540231}
 }},
 "version":3,
 "timestamp":1490540318
}

For more details of JSON Syntax and samples refer to following Links –

Device Shadow JSON Document Syntax

Using Device Shadow

Conclusion:

Now you understand the basics of AWS IoT Thing or Device Shadows. In the next article, we shall see Shadows in action with Raspberry Pi.

Device Shadows – Part 2 (MQTT Based Prototype using Python and Raspberry Pi)

Device Shadows – Part 3 (RESTful API Based Prototype using Python)