Power up Prometheus with M3: Uber’s large-scale metrics platform
Another open sourced project from Uber? Sign me up. M3 is a large-scale metrics platform for Prometheus. Come check it out and see how scalability can improve Prometheus.
To say there’s a lot going on in Uber’s back-end systems is an understatement. Despite recent news limiting its usage in New York City, Uber continues to grow, both on the road and in their engineering department. The newest open source technology by Uber is M3: a large-scale metrics platform for Prometheus.
Alongside of M3, the ridesharing giant also open sourced M3 Coordinator, a Prometheus sidecar that coordinates reads and writes between upstream systems. (For those interested, check out this presentation to discover more about how to integrate long-term storage with Prometheus.)
Come and reap the benefits from their engineering success and see what these technologies can do!
What is M3?
Uber grew at a rapid pace and needed a metrics platform that could keep up with its expansion. The company had five goals in mind when developing a new system: improved reliability and scalability, capability for queries to return cross-data center results, low latency service level agreements, first-class dimensional “tagged” metrics, and backwards compatibility.
With these goals in mind, M3 was born.
From Uber’s engineering blog: “M3 reliably houses large-scale metrics over long retention time windows…As its documentation states, Prometheus’ scalability and durability is limited by single nodes. The M3 platform aims to provide a turnkey, scalable, and configurable multi-tenant store for Prometheus metrics.”
Since its creation in 2015, M3 houses a staggering 6.6 billion time series and aggregates 500 million metrics per second. You can’t argue with those numbers.
- A lean memory footprint helps avoid the dreaded bottleneck.
- Out of the box support for multiple clusters.
- Optimization of the metrics pipeline.
- Highly compressed data allows for a low hardware footprint, allowing for the best amount of storage possible. This complex algorithm is inspired by Gorilla TSZ.
- The native design does not require a high write volume or constant operational attention.
- Switch precision with any write.
M3 provides a single, global view of all metrics with its single pane view. The single pane view enables users to easier sum and query metrics from across all regions. Using this feature, developers can get a better sense of operations of a specific type globally its trends.
What does the future of M3 hold? According to the Uber engineering blog, “In future iterations we’d like to engineer M3 to push query aggregations to remote regions to execute before returning results, as well as to the local M3DB storage node where possible.” Its engineering team is also working out bringing M3 Coordinator and M3DB’s reverse index out of beta, adding turnkey support for StatsD and Graphite, and releasing m3ctl UI.
Test it out
SEE ALSO: Prometheus monitoring: Pros and cons
Scan the documentation to learn more about M3, including the details behind its architecture, its current limitations, and the alpha development of M3DB on Kubernetes.
Go on and start a node:
# to build a local m3dbnode process make m3dbnode # run it with the sample configuration ./bin/m3dbnode -f ./src/dbnode/config/m3dbnode-local-etcd.yml
A Dockerfile is included for development and production deployment. Don’t forget to join the chat on gitter and get talking.
As with any open source project, we at JAXenter are always grateful for tech giants opening up their creations for all to enjoy and contribute. Bravo, Uber, and we will see you on the road!