Out of the incubator

Cluster manager Mesos gets top billing at Apache

Chris Mayer

Twitter and AirBnB are already using the distributed resource manager – are more set to follow suit?

Open source cluster manager Apache Mesos has risen to Top-Level Project status at the Apache Software Foundation.

The news, announced today, sees the project leave the Apache incubator after almost three years, signifying that it meets the open source foundation’s guidelines. In that time, the project has had four major releases and established a solid community.

Apache Mesos is essentially an efficient cluster manager, capable of providing resource isolation across distributed applications. It can also run a number of other frameworks, notably big data processor Apache Hadoop, continuous integration server Jenkins and distributed realtime system Storm.

Originally beginning life at the University of California’s Berkeley AMPLab, Mesos has since found its way into the production environments at Twitter and AirBnB, as the ideal solution forl their cluster management needs.

“Mesos is the cornerstone of our elastic compute infrastructure,” explained Chris Fry, Senior Vice President of Engineering at Twitter. “It’s how we build all our new services and is critical for Twitter’s continued success at scale … one of the primary keys to our data infrastructure efficiency.”

Mesos dovetails Hadoop to some extent, being able to run multiple instances of Hadoop (of different versions) on the same cluster. This might sound familiar to onlookers of the big data project, with YARN, the cornerstone of Hadoop 2.0, offering something comparable. Though there are distinct differences. YARN is more of an application scheduler than a framework scheduler like Mesos. YARN is of course written while Mesos is coded in C++.

Hortonworks co-founder and Hadoop PMC Arun C Murthy said in a Quora post from 2011, comparing the two projects, that “it’s great to have multiple flavours”.

“We, the Yahoo/Hortonworks MapReduce team driving YARN, have known and provided feedback to the Mesos guys for a very long time and continue to share ideas with them.”

The project contains a fault-tolerant master, which is controlled by fellow Apache project Zookeeper, and capable of scaling up to 10,000s of nodes. Mesos has Java, Python and C++ for developing parallel applications and a UI for viewing cluster state.

Benjamin Hindman, Vice President of Apache Mesos, says it was the team’s collective goal to see the project become “a kernel of the infrastructure stack of the future.”

“The project’s graduation from the Apache Incubator is recognition that the software is mature and has brought together a diverse community to sustain it in the future.”

Now that Apache Mesos has that rubber stamp from the ASF, it should go onto to pick up more corporate admirers and developers keen to give the project a whirl.
Inline Feedbacks
View all comments