2. Attribution
• The material contained inside is intended for teaching.
• This document is licensed under the CC BY-NC-SA license.
• Relevant sources are listed on the following References slide.
• All figures and text borrowed from these sources retain the rights of
their respective owners.
2
4. Acknowledgments
• Special thanks to Zach Shelby and Raj Jain for their high-quality
teaching material that I have extensively reproduced here.
4
5. Table of Contents
1. IoT and Web of Things networking
2. Message Queuing Telemetry Transport
3. Constrained Application Protocol
4. Implementations on Raspberry Pi
5. Programming with MQTT/CoAP APIs on Raspbian
5
7. Networking on the Edge
• Low power, low speed, lossy wireless networks
• Huge number of low capability end devices
• Automatically generated (big) data
• Terse, purposeful, uncritical
7
8. Current Human-oriented Protocol Stack
• Communications are mostly one-to-one/unicast and sender-oriented
• M2M comms do not need protocols for human comms
• High overhead features: smooth streaming, character echoing, presentation
format, etc.
• Low-end devices can not store/use a full network protocol stack
• Functionnality needs hardware which costs money
8
9. IoT Paradigms
• Wireless networks to connect them all
• Cheap and adaptive infrastructures compared to wired networks
• Many-to-few/few-to-many/many-to-many communications
• Group-oriented/multicast communications
• Can cope with huge number of devices
• Reliability through redundancy
• Losses and errors are frequent and unavoidable
• Large number of unreliable, uncritical messages
• Simple and autonomous devices
9
10. Zones of Interest
• In nature, zones of interest are formed by affinities
• Common medium used by all
• Individuals act upon specific signals among countless other signals
• Interested observing individuals can select a zone of interest
• Analyze/send data only to/from this zone
• Receiver-oriented
• Wireless channels can broadcast all messages to neighbors
• Neighbors process messages if they wish
10
11. Publish/Subscribe Paradigm
• Devices publish information tagged with category (i.e. zone)
• They broadcast it to everyone
• Devices subscribe to some categories
• They act only on what they are interest in
• No tight coupling between publishers and subscribers
• Unmanaged loose coupling
• Signals/messages are simple and small
• This communication paradigm scales
11
12. Publish/Subscribe Topologies
• With intermediary
• Publishers post messages to an intermediary message broker
• Subscribers register subscriptions with that broker
• The broker performs message filtering and “store and forward” routing from
publishers to subscribers
• The broker may prioritize messages in a queue before routing
• Without intermediary
• Each publisher and subscriber shares meta-data about each other via
multicast
• Publishers and subscribers cache this information locally and route messages
based on the discovery of each other interests
12
13. Functional Levels (IoT Hierarchy Revisited)
• End devices: mostly publishers
• Meshed to one another and connected to propagators
• Generate data (sensors) or act on command (actuators)
• Propagator nodes: brokers
• Connected to the Internet
• Process and aggregate data from end devices to integrators
• Integrator nodes: mostly subscribers
• Connected to the Internet
• Analyse data, control the system, and interface with humans
13
22. What is MQTT
• ISO standard publish-subscribe-based lightweight messaging protocol
• Telemetry = tele-metering = remote measurements
• However, no message queuing in MQTT
• Lightweight = low network bandwidth and small code footprint
• For use on top of the TCP/IP protocol stack
• The publish-subscribe messaging pattern requires a message broker
• The broker is responsible for distributing messages to interested
clients based on the topic of a message
• Data agnostic, adapt to any content formats
22
23. Publish / Subscribe
23
• Clients can subscribe to a topic
or a set of related topics
• Clients can also publish to one or
more topics
• Publish/Subscribe via message
exchanges
• Topics organized as trees using /
character
• /# matches all sublevels
• /+ matches only one sublevel
24. Quality of Service Levels
• Three levels
• 0 = At most once (best effort, no Ack)
• 1 = At least once (Acked, retransmitted if Ack not received)
• 2 = Exactly once
• Control Messages Sequence: Publish → Pubrec → Pubrel → Pubcomp
• Server keeps/retains messages even after sending it to all subscribers
• New subscribers get the retained messages
24
25. Control Messages
• Indicate the desired action to be
performed on the identified
resource/server
• Publish messages can be to/from
Client/Server
• Often, the resource corresponds
to a file or the output of an
executable residing on the
server
• CONNECT/CONNACK
• PUBLISH/PUBACK (QoS 1)
• PUBREC/PUBREL/PUBCOMP (QoS 2)
• SUBSCRIBE/SUBACK
• UNSUBSCRIBE/UNSUBACK
• PINGREQ/PINGRESP
• DISCONNECT
25
26. Sessions and Connections
• Clean or continuous sessions with durable connections
• At connection set up, if Clean Session flag → all subscriptions are removed on
disconnect
• Otherwise subscriptions remain in effect after disconnection → subsequent
messages with high QoS are stored for delivery after reconnection
• Wills
• at connection set up, a client can set a Will flag to inform that it has will
message that should be published if unexpected disconnection → alarm if the
client looses connection
• Periodic keep-alive messages → if a client is still alive
26
29. MQTT for Sensor Networks (MQTT-SN)
• Variation of the main protocol aimed at embedded devices on non-
TCP/IP networks such as ZigBee
• Zigbee is a low-power, low data rate, close proximity wireless ad hoc PAN
• Optimized for implementation on low-cost, battery-operated devices
with limited processing and storage resources
• Enforce short messages (<<128B)
• Support for sleeping clients, CONNECT message split in 3, topic names
replaced by 2B topic ids, predefined topic ids with no registration,
server/gateway network @ discovery procedure
29
37. What is CoAP
• CoAP is
• A very efficient RESTful protocol
• Ideal for constrained devices and networks
• Specialized for M2M applications
• Easy to proxy to/from HTTP
• CoAP is not
• A general replacement for HTTP
• HTTP compression
• Restricted to isolated “automation” networks
37
38. Features
• Embedded web transfer protocol (coap://)
• Asynchronous transaction model
• UDP binding with reliability and multicast support
• GET, POST, PUT, DELETE methods
• URI support
• Small, simple 4 byte header
• DTLS based PSK, RPK and Certificate security
• Subset of MIME types and HTTP response codes
• Built-in discovery
• Optional observation and block transfer
38
39. Transaction Model
• Transport
• CoAP currently defines UDP binding with DTLS security
• CoAP over SMS or TCP possible
• Base Messaging
• Simple message exchange between endpoints
• Confirmable or Non-Confirmable Message
• Acknowledgement or Reset Message
• REST Semantics
• REST Request/Response piggybacked on CoAP Messages
• Method, Response Code and Options (URI, content-type, etc.)
39
47. Caching
• CoAP includes a simple caching model
• Cacheability determined by response code
• An option number mask determines if it is a cache key
• Freshness model
• Max-Age option indicates cache lifetime
• Validation model
• Validity checked using the Etag Option
• A proxy often supports caching
• Usually on behalf of a constrained node,
• a sleeping node,
• or to reduce network load
47
51. Web Linking
• Web Linking formalizes links with defined relations, typed links
• HTML and Atom have allow links
• RFC5988 defines a framework for Web Linking
• Combines and expands the Atom and HTML relation types
• Defines a unified typed link concept
• A link can be serialized in any number of formats
• RFC5988 revives the HTTP Link Header and defines its format
• Atom and HTML are equivalent serializations
51
52. Typed Link
• A type link consists of
• Context URI – What the link is from
• Relation Type – Indicates the semantics of the link
• Target URI – What the link is to
• Attributes – Key value pairs describing the link or its target
• Relations include e.g. copyright, author, chapter, service, etc.
• Attributes include e.g. language, media type, title, etc.
• Example in HTTP Link Header format
Link: <http://example.com/TheBook/chapter2>; rel="previous"; title="previous chapter"
52
53. Resource Discovery
• Service Discovery
• Find the IP address, port and protocol of the service
• Usually performed by DNS-SD when DNS is available
• Resource Discovery
• Finding URIs
• Performed using Web Linking or some REST interface
• CoRE Link Format is designed to enable resource discovery
53
54. CoRE Link Format
• RFC6690 is aimed at Resource Discovery for M2M
• Defines a link serialization suitable for M2M
• Defines a well-known resource where links are stored
• Enables query string parameters for filtered GETs
• Can be used with unicast or multicast (CoAP)
• Resource Discovery with RFC6690
• Discovering the links hosted by CoAP (or HTTP) servers
• GET /.well-known/core?optional_query_string
• Returns a link-header style format
• URL, relation, type, interface, content-type etc.
54
56. Resource Directory
• CoRE Link Format only defines
• The link format
• Peer-to-peer discovery
• A directory approach is also useful
• Supports sleeping nodes
• No multicast traffic, longer battery life
• Remote lookup, hierarchical and federated distribution
• The CoRE Link Format can be used to build Resource Directories
• Nodes POST (register) their link-format to an RD
• Nodes PUT (refresh) to the RD periodically
• Nodes may DELETE (remove) their RD entry
• Nodes may GET (lookup) the RD or resource of other nodes
56
58. Using CoRE in Real Applications
• Resources need meaningful naming (rt=)
• A resource needs an interface (if=)
• Use WADL for this
• A payload needs a format (EXI, JSON, etc.)
• Deployment or industry specific today
• oBIX, SensorML, EEML, sMAP, etc.
• SenML is a promising format [draft-jennings-senml]
• What can we make universal? What should be market specific? How
do we enable innovation?
58
59. CoRE Link Format Semantics
• RFC6690 = Simple semantics for machines
• IANA registry for rt= and if= parameters
• Resource Type (rt=)
• What is this resource and what is it for?
• e.g. Device Model could be rt=“ipso.dev.mdl”
• Interface Description (if=)
• How do I access this resource?
• e.g. Sensor resource accessible with GET if=“core.s”
• Content Type (ct=)
• What is the data format of the resource payloads?
• e.g. text/plain (0)
59
61. Open Mobile Alliance (OMA)
• Standards body which develops open standards for the mobile phone
industry
• Only standardizes applicative protocols
• Specifications are meant to work with any cellular network
technologies
• Example specifications
• MMS multimedia messaging
• Instant Messaging and Presence Service
• Lightweight M2M
61
62. Lightweight M2M
• Provide device management functionality over sensor or cellular
networks
• 6LoWPAN, WiFi, ZigBee, any IP based constrained devices/networks
• Transfer service data from the network to devices
• Can be extended to meet the requirements of any application
• Supported in OneM2M and integrated with ETSI M2M
62
63. LWM2M 1.0 Features
• Simple object based resource model
• Global registry and public lookup of all objects
• Standard device management objects already defined by OMA
• Resource operations of creation/retrieval/update/deletion/configuration of attribute
• Resource observation/notification
• TLV/JSON/Plain Text/Opaque data format support
• UDP and SMS transport layer support
• DTLS based security
• Pre-shared and Public Key modes, Provisioning and Bootstrapping
• Queue mode for NAT/Firewall environment
• Multiple server support
• Basic M2M functionalities
• Access Control, Device, Connectivity, Firmware Update, Location, Connectivity Statistics
63
65. IoT Standardization
• IETF
• 6LoWPAN Working Group (IPv6 anywhere)
• ROLL (Routing Over Low-power Lossy Networks) WG
• CoRE WG (REST for IoT, CoAP, Resource Directory etc.)
• TLS WG (DTLS)
• OMA
• Lightweight M2M Enabler Standard (CoAP/DTLS based)
• Device Management 2.0 Enabler Standard (HTTP/TLS based)
• ETSI / OneM2M
• Ongoing work on M2M system standardization (CoAP, HTTP binding)
• W3C
• Efficient XML Interchange (EXI) standardization
• ZigBee IP
• An open-standard 6LoWPAN stack for e.g. Smart Energy 2.0
65
71. MQTT Client Subscription Example
import paho.mqtt.client as mqtt
# The callback for when the client receives a
CONNACK response from the server.
def on_connect(client, userdata, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose
the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("$SYS/#")
# The callback for when a PUBLISH message is
received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("iot.eclipse.org", 1883, 60)
# Blocking call that processes network traffic,
..dispatches callbacks and handles reconnecting.
# Other loop*() functions are available that give
.. a threaded interface and a manual interface.
client.loop_forever()
71
72. Other MQTT Client Examples
• https://github.com/eclipse/paho.mqtt.python/tree/master/examples
72
73. CoAP GET Request Client Example
• https://github.com/mwasilak/txThings/blob/master/examples/clientGET.py
73