Traditional messaging solutions must evolve to address the microservices wave
The shift in application development focus has led to the development of new data handling products to support changing business application development requirements. In this article, Sean Bowen explains why the Pub/Sub model is a natural fit for a microservice architecture and why traditional messaging solutions need to change.
In today’s global market, business software must be efficient, adaptable, and easy to use. Microservices deliver simple solutions to complex problems and are currently vogue within the developer community. What are they?
Microservices are atomic, self-contained services that perform a single operation on a back-end system. The beauty of Microservices is that they are easy to create, deploy and share. This architectural approach has spawned a new generation of app creation and delivery which has made it more straightforward for Enterprises and software vendors to accelerate new applications.
If uptime is critical to your business and you have complex, backend infrastructure and geographically distributed teams, a Microservices architecture, as opposed to monolithic single applications, is now seen as the most viable approach.
A shift in development focus
The inherent scalability of Microservices makes it ideal for supporting a wide range of platforms and devices – spanning web, mobile, IoT, and wearables – or when future device support needs are unclear. Decomposing an application into discrete smaller services improves modularity and makes the application easier to understand, develop, and test.
The shift in application development focus has led to the development of new data handling products to support changing business application development requirements. For example, Apache Kafka, which mixes traditional Message Queue (MQ) functionality within a Pub/Sub model, is a popular tool for handling Event Processing/Distribution due to its ability to decouple producers and consumers of data.
The application development objective
The overarching objective for modern application development teams is to establish, monitor, and manage data pipelines that extend Enterprise back-end systems through to the constantly proliferating array of end-user devices (connected over a variety of unreliable networks). To satisfy business mandates, the applications must be consistent, reliable, high performant and operate in a cost-effective manner.
Traditional messaging assumptions
Most MQ products are designed with the following assumptions:
- Bandwidth will always be sufficient and inexpensive.
- Latency will be reasonably consistent.
- Network interruptions will be quickly resolved.
- Per-node traffic volume will remain relatively low and consistent.
- Target platforms are primarily back-end systems.
Unfortunately, the huge consumer demand for web and mobile applications, coupled with the explosion of IoT data, invalidates these assumptions, resulting in substandard application performance and added work for development teams.
The limitations of traditional messaging solutions
MQ products typically do not handle extremely high load – such as websites with hundreds of thousands of concurrent users – without degrading performance or requiring large server cluster infrastructure. Many MQ products are not designed with a web/mobile-first approach. This impacts on application responsiveness and user engagement when bandwidth is constrained, or networks congested. In addition, MQ products often lack purpose-built web or mobile-specific SDKs, complicating development projects and lengthening time-to-market.
To overcome MQ product challenges, developers are faced with the time-consuming task of implementing additional functionality on top of the core MQ product and increasing the complexity of their infrastructure. This results in increased cost, application development and infrastructure complexity, and additional delivery time for development projects.
The messaging solution for microservices
I believe that today’s application development challenges – high load, variable traffic, unreliable connections and complex implementations – are better served by higher-level paradigms such as Pub/Sub and Point-to-Point messaging to deliver reliable, high quality and consistent application performance.
This is where I introduce Diffusion, our Intelligent Data Platform. Diffusion’s real-time Pub/Sub data streaming is optimized for delivery speed and network, bandwidth, and infrastructure efficiency:
- Diffusion only sends data to subscribers when the value of a topic changes. This avoids clients being forced to continually poll for new data, reducing both code complexity and network usage.
- Diffusion only sends the differences (deltas) between topic updates, with automatic value reconstruction at the client side, drastically reducing the amount of data sent over the network.
Avoiding the MQ pitfalls
Diffusion’s delta-data steaming eliminates many MQ product challenges and provides substantial benefits:
- Fewer and smaller messages sent over the network results in a low and consistent latency profile – the data gets to consumers faster, especially on unreliable networks such as mobile or satellite.
- Less bandwidth usage means that sending data costs less.
- With low network overhead per connection, Diffusion can support vast numbers of concurrent clients.
- Topic updates are queued on a per-client basis, adapting to congested networks and allowing full recovery in the event of disconnections.
- Newly connected subscribers automatically receive the latest values for subscribed topics, reducing system complexity.
The Pub/Sub model is a natural fit for Microservice architectures because it explicitly decouples producers and consumers of data, making it ideal for both broadcasting and ingesting data to and from thousands of applications or devices. Traditional messaging products needs to change with the times as they are currently not fit for purpose.