Out of the incubator

Cluster manager Mesos gets top billing at Apache

Chris Mayer
Apache-Mesos1

Twitter and AirBnB are already using the distributed resource manager – are more set to follow suit?

Open source cluster manager Apache Mesos has risen to Top-Level
Project status at the Apache Software Foundation.

The news, announced
today
, sees the project leave the Apache
incubator after almost three years, signifying that it meets the
open source foundation’s guidelines. In that time, the project has
had four major releases and established a solid
community.

Apache Mesos is essentially an efficient cluster
manager, capable of providing resource isolation across distributed
applications. It can also run a number of other frameworks, notably
big data processor Apache Hadoop, continuous integration server
Jenkins and distributed realtime system Storm.

Originally beginning life at the University of
California’s Berkeley AMPLab, Mesos has since found its way into
the production environments at Twitter and AirBnB, as the ideal
solution forl their cluster management needs.

“Mesos is the cornerstone of our elastic compute
infrastructure,” explained Chris Fry, Senior Vice President of
Engineering at Twitter. “It’s how we build all our new services and
is critical for Twitter’s continued success at scale … one of the
primary keys to our data infrastructure efficiency.”

Mesos dovetails Hadoop to some extent, being
able to run multiple instances of Hadoop (of different versions) on
the same cluster. This might sound familiar to onlookers of the big
data project, with YARN, the cornerstone of Hadoop 2.0, offering
something comparable. Though there are distinct differences. YARN
is more of an application scheduler than a framework scheduler like
Mesos. YARN is of course written while Mesos is coded in
C++.

Hortonworks co-founder and Hadoop PMC Arun C
Murthy said in a
Quora post
from 2011, comparing the two projects,
that “it’s great to have multiple flavours”.

“We, the Yahoo/Hortonworks MapReduce team
driving YARN, have known and provided feedback to the Mesos guys
for a very long time and continue to share ideas with
them.”

The project contains a fault-tolerant master,
which is controlled by fellow Apache project Zookeeper, and capable
of scaling up to 10,000s of nodes. Mesos has Java, Python and C++
for developing parallel applications and a UI for viewing cluster
state.

Benjamin Hindman, Vice President of Apache
Mesos, says it was the team’s collective goal to see the project
become “a kernel of the infrastructure stack of the
future.”

“The project’s graduation from the Apache
Incubator is recognition that the software is mature and has
brought together a diverse community to sustain it in the
future.”

Now that Apache Mesos has that rubber stamp from the ASF,
it should go onto to pick up more corporate admirers and developers
keen to give the project a whirl.

Author
Comments
comments powered by Disqus