When push comes to shove
MQTT shines in push-based communication
MQTT - a lightweight M2M messaging protocol - is something we’re going to be hearing a lot more about inthe months to come as IoT hype reaches fever pitch. But why is it catching on? And what’s driving companies to adopt it? In a special series, JAXenter reporter Diane Kupfer aims to get under the skin of MQTT. Here, in an interview which took place at EclipseCon Europe, she speaks with Christian Götz and Dominik Obermaier - co-founders of dc-square GmbH - about MQTT security, the community around it, and what drove them to adopt this protocol.
JAXenter: How did you come across MQTT?
Christian Goetz: At dc-square we focus on applications in the field of Machine-to-Machine communication, above all MQTT. We learned about MQTT through customer projects and then started making contact with the community on Twitter and mqtt.org.
This is how learned about the advantages of MQTT. We use this knowledge in a targeted way in projects that are brought to us by customers.
How do you explain this hype around MQTT - what exactly are the benefits?
Goetz: It has only been open-source and license-free since 2010, which is when it started spreading. In my opinion, the explosion of smartphones and mobile devices in general contributed significantly to the hype.
The numbers are quite impressive. For example, about 80 units per second are connected to the Internet. The hype is also due in large part to the fast emergence open hardware platforms such as the Rasberry Pi and Arduino.
Easier access to hardware-specific programming is something many people are grateful for. Thus, the need for solutions for communication over unreliable networks such as the cellular network is growing.
The requirements that were identified for a protocol for monitoring oil pipelines via satellite in 1999 are consistent with those identified for mobile communication in the 21st century. MQTT is a lightweight and resource-efficient data transfer. It does not specify a particular data format and provides the security that all messages are transmitted, even if the connection breaks off briefly. Nevertheless, it has been kept relatively simple, which is important for use on devices with limited resources. MQTT has a clear focus on the mobile sector because it helps solve problems that arise as a result of unreliable communication.
“Focus on the mobile sector” means that it is not the protocol for all applications. Dominik, you also briefly discussed HTTP and CoAP [ Constrained Application Protocol] in your EclipseCon talk. What are the advantages and disadvantages of CoAP and HTTP over MQTT?
Dominik Obermaier: HTTP is as old as the World Wide Web itself, i.e. it is a very stable protocol. There is an HTTP library for each programming language. It is standard and also readable by humans, which means it is not a binary protocol such as CoAP and MQTT.
If many different systems need to be integrated, it plays off its strengths. But it also has disadvantages, especially in the M2M sector. One of them is that it is not binary, hence the bandwidth usage is not optimized. Apart from this, HTTP generally has a lot of overhead - the HTTP header, for example - which is not necessarily required for the application.
The protocol is request-response-based, so you ask the server for data and wait for a response. For real push, HTTP communication is not suitable. Nevertheless, HTTP will not disappear from the scene. It is here to stay and will play an important role in the long run. One simply can’t afford not to provide HTTP support for a REST API.
CoAP is a relatively new protocol. It still has a very small community and there are very few implementations. Generally, CoAP, similar to HTTP, is a RESTful protocol. It is optimized for machines and in a way compatible with HTTP. As already mentioned, its disadvantages are a small community, the fact that it is difficult to debug because it is binary, and the lack of support for tools and libraries .
But the community is growing, right?
Obermaier: Exactly. I think that in the long run Internet of Things platforms will also have CoAP interfaces in addition to an HTTP interface - for example, applications that require an efficient mobile communications.
You've previously talked about Quality of Service Levels (QoS), which MQTT has and HTTP doesn’t. Can you briefly explain what this is?
Obermaier: In MQTT there are three QoS levels: 0, 1 and 2. QoS 0 is basically fire it up and forget it. Since MQTT is based on TCP, a connection via a wired network is relatively stable. It’s mostly enough to rely on the mechanisms provided by TCP when sending a message to all the clients.
In a mobile scenario with unstable mobile networks, it may happen that messages are lost in spite of TCP. With QoS 0, this means that the message does not arrive. The transmitter is not informed about it.
In QoS Level 1 it is ensured by the protocol itself that the message is delivered at least once. As a consequence, the recipient might receive duplicates under certain circumstances - for example, if the reply is lost and the client sends the message again. In principle, this means, however, that the message has arrived. The highest warranty gives QoS level 2. An extended protocol flow ensures that the message is delivered to the recipient exactly once .
If you use these three protocols for different scenarios, are there still larger gaps to fill? Are there any important use cases which are not covered and where you would need to use other protocols such as AMQP?
Obermaier: That depends on the use case. If one develops applications specifically for mobile communication, you can already cover a lot. The popular Internet of Things platforms implement these three protocols, and so far there are no major gaps. This is also due to the different paradigms on which the protocols are based: Publish/Subscribe in the case of MQTT and Request/Response in HTTP and COAP.
So they complement each other pretty well?
Obermaier: MQTT shines when you need push-based communication. In request-based communication HTTP or CoAP are very good. That's why I currently see no need for additional protocols.
You just talked about Push, where WebSocket is surely a great addition, and have also developed a very interesting solution for HiveMQ, combining WebSocket with MQTT. How does this work?
Obermaier: The MQTT broker HiveMQ which is developed by us can natively handle WebSocket. MQTT itself is based on TCP. Thanks to the native support, MQTT messages can also be sent to the broker via WebSocket. Thus virtually every browser can be a MQTT client.
What are use cases scenarios for this combination?
Obermaier: Online chats are a good example, and generally everything that needs push. News tickers are another one. The interest here is very great, especially in the community. But there are many other people who develop solutions to problems of which one hasn’t even thought of yet.
When one uses this approach, each browser and any smartphone using a browser can be an MQTT device. You get true push notifications without having to develop your own solutions, which makes it possible to exchange messages over WebSocket .
Let’s talk about security, which is obviously a big field, but also very important in the Embedded/M2M/IOT space. How can MQTT be made secure?
Götz: In general there are several ways, or actually, several levels, you need to consider. The protocol requires that when the client starts connecting, a user name and a password are transmitted. This is not secure at first, because the username and password are transmitted in plain text.
This is avoided by establishing a SSL connection between client and broker, which ensures that the message content and the user name and the password is not read by third parties or manipulated. Of course, it should be noted that SSL is a protocol with an overhead that is not small. Encrypting the actual message might be an alternative.
In some scenarios, it also makes sense, to protect the communication via MQTT through an SSH tunnel or VPN connections. In that case, the SSH or VPN connection guarantees security.
Of course, it’s not enough to provide sufficient security at the protocol level, but also at the operating system level of the server on which the broker is running. This falls more into the area of operations .
Depending on the application, it might also make sense to protect topics individually at the application level. A common security measure is to limit the privileges of each client to certain topics. For example, only some clients could be allowed to send messages [publish] or receive them [subscribe]. This is not a functionality of the protocol, but relies on the implementation of the broker.
Due to the publish-subscribe pattern, the clients do not know each other. Doesn’t this give MQTT a degree of safety?
Götz: Exactly. The client must not know each other, which is a central aspect. However, that does not directly prevent tampering or interception of data that is to be transmitted. A potential attacker could, provided he has the username and password of a client, listen to messages or send messages that cause damage. Also the assumption that clients do not know is true only so far as we do not address directly with identifiers. There also has to be a certain level of knowledge, because MQTT also data-agnostic, i.e. the protocol does not rely on fixed structures when it comes to transferring data, but leaves this to the developer. All clients must follow this in the same way. A general assumption is that only trusted clients are located in the MQTT network.
Nevertheless, you should only allow the clients to exchange the necessary messages and do not give access to everything, according to the principle of least privilege. This is especially important for devices that are publicly accessible. One must ensure that a potential attacker who has physical access to, for example, a temperature sensor, cannot send control commands and thus influence other devices.
This sounds like one already has the most essential tools for the Internet of Things. Why are still so many proprietary technologies in use, then?
Goetz: There are several reasons for this. One of them is that while the MQTT community grows every day, it just takes time to bring MQTT into companies. Development play a significant role in this. Developers are made aware of MQTT by the developer community and conferences such as EclipseCon or the JAX, and they carry this knowledge into their companies. After all, no one is informed about this by push notifications on his cell phone [laughs].
In general, the openness of the protocol, the work of the community and the open tools and libraries contribute to building a stable ecosystem, which can be used sustainably in companies. However, apart from a few exceptions, in many firms the idea to implement everything yourself, is still valued more than a reliance on open de-facto standards.
What can be done to change this ingrained corporate culture?
Götz: This is not an easy task. But as I said, the developers play an important role, since they bring open technologies into the company, making them more visible. It often takes a lot of persuasion, for example, when laying out the benefits and comparing them to established technologies. This takes time. The community is on the right track, and there is a lot of commitment. However, these things do not change overnight.
Obermaier: It is also a question of cost: If you develop your own proprietary protocol for each project, you always have a high development effort. However, if you use something that has been in the market for a while, something that is production-ready and stable, you often have a smaller risk and cost.
Goetz: The most proprietary protocols are tailored to specific applications. This is certainly not the right way. If you use open protocols, you benefit from the ecosystem around it. In addition, a higher level of maturity is attained faster because a protocol is often used in various applications and by different developers .
So higher productivity requirements will ultimately bring about change?
Obermaier: Yes, exactly.
Anything else you’d like to add about MQTT?
Goetz: In general, we want to encourage everyone to try MQTT. Don’t be put off by the different paradigm - and include it in your technology stack! We do not necessarily use it in every project. But there are more and more use cases for which it is advantageous to have a suitable solution up your sleeve.
Are there any MQTT focused community meetings or conferences?
Goetz: I know nothing of MQTT-specific conferences. There are special panels at JAX and EclipseCon where M2M and MQTT are covered in depth. These obviously foster exchange, and help grow the community.
Image by Risto Kuulasmaa