days
-1
-6
hours
0
-5
minutes
-3
-8
seconds
-2
-5
search
Scalability and durability

Power up Prometheus with M3: Uber’s large-scale metrics platform

Sarah Schlothauer
uber
© Shutterstock / photomaster

Another open sourced project from Uber? Sign me up. M3 is a large-scale metrics platform for Prometheus. Come check it out and see how scalability can improve Prometheus.

To say there’s a lot going on in Uber’s back-end systems is an understatement. Despite recent news limiting its usage in New York City, Uber continues to grow, both on the road and in their engineering department. The newest open source technology by Uber is M3: a large-scale metrics platform for Prometheus.

Alongside of M3, the ridesharing giant also open sourced M3 Coordinator, a Prometheus sidecar that coordinates reads and writes between upstream systems. (For those interested, check out this presentation to discover more about how to integrate long-term storage with Prometheus.)

Come and reap the benefits from their engineering success and see what these technologies can do!

What is M3?

Uber grew at a rapid pace and needed a metrics platform that could keep up with its expansion. The company had five goals in mind when developing a new system: improved reliability and scalability, capability for queries to return cross-data center results, low latency service level agreements, first-class dimensional “tagged” metrics, and backwards compatibility.

With these goals in mind, M3 was born.

From Uber’s engineering blog: “M3 reliably houses large-scale metrics over long retention time windows…As its documentation states, Prometheus’ scalability and durability is limited by single nodes. The M3 platform aims to provide a turnkey, scalable, and configurable multi-tenant store for Prometheus metrics.”

Since its creation in 2015, M3 houses a staggering 6.6 billion time series and aggregates 500 million metrics per second. You can’t argue with those numbers.

SEE ALSO: Uber joins the framework race with Fusion.js

M3 features

  • A lean memory footprint helps avoid the dreaded bottleneck.
  • Out of the box support for multiple clusters.
  • Optimization of the metrics pipeline.
  • Highly compressed data allows for a low hardware footprint, allowing for the best amount of storage possible. This complex algorithm is inspired by Gorilla TSZ.
  • The native design does not require a high write volume or constant operational attention.
  • Switch precision with any write.

M3 provides a single, global view of all metrics with its single pane view. The single pane view enables users to easier sum and query metrics from across all regions. Using this feature, developers can get a better sense of operations of a specific type globally its trends.

What does the future of M3 hold? According to the Uber engineering blog, “In future iterations we’d like to engineer M3 to push query aggregations to remote regions to execute before returning results, as well as to the local M3DB storage node where possible.” Its engineering team is also working out bringing M3 Coordinator and M3DB’s reverse index out of beta, adding turnkey support for StatsD and Graphite, and releasing m3ctl UI.

Test it out

You can check out M3 for yourself on GitHub and then test it out with the single node how-to guide.

SEE ALSO: Prometheus monitoring: Pros and cons

Scan the documentation to learn more about M3, including the details behind its architecture, its current limitations, and the alpha development of M3DB on Kubernetes.

Go on and start a node:

# to build a local m3dbnode process
make m3dbnode

# run it with the sample configuration
./bin/m3dbnode -f ./src/dbnode/config/m3dbnode-local-etcd.yml

A Dockerfile is included for development and production deployment. Don’t forget to join the chat on gitter and get talking.

As with any open source project, we at JAXenter are always grateful for tech giants opening up their creations for all to enjoy and contribute. Bravo, Uber, and we will see you on the road!

Author
Sarah Schlothauer

Sarah Schlothauer

All Posts by Sarah Schlothauer

Sarah Schlothauer is an assistant editor for JAXenter.com. She received her Bachelor's degree from Monmouth University in Long Branch, New Jersey and is currently enrolled at Goethe University in Frankfurt, Germany where she is working on her Masters. She lives in Frankfurt with her husband and cat.

Leave a Reply

Be the First to Comment!

avatar
400
  Subscribe  
Notify of