At the summit

Hortonworks show YARN future Hadoop with HDP 2.0 preview

Chris Mayer

The Palo Alto company look in rude health following Hadoop Summit, thanks to a welcome update for HDP 2.0


Last week’s Hadoop Summit has been labelled a resounding success by analysts, signalling the big data framework’s maturity in the enterprise, as well as being a thriving hub of innovation with many contenders.

The company that appears to have come out on top from the event is Hortonworks. Aside from revealing a new hefty $50m round of funding, the Palo Alto company has offered a community preview for Hadoop Data Platform 2.0, containing the highly anticipated YARN.

The framework, which has been four years in the making, is considered the crux of the next open source release as it broadens Hadoop’s processing options and architecture. In simpler terms, the introduction of YARN allows MapReduce to focus on processing alone, with YARN itself acting as the general purpose resource management tool. Through this approach, Hadoop developers can now create batch or nonbatch jobs while running interactive SQL queries in parallel.

While the batch approach of MapReduce ultimately made Hadoop’s name, the inability to handle several applications (and glean insights from them at close to real time) has proven to be a stumbling block for some.

Speaking to IDG News Service’s Joab Jackson, Hortonworks founder Arun Murthy hoped that the introduction of YARN would “fuel a whole new wave of innovation” in Hadoop. This appears to hold substance, with Continuuity and Yahoo unveiling YARN projects Weave and Storm-YARN in the past month respectively. The likes of Microsoft, Red Hat and Splunk are among 15 companies to partner up with Hortonworks, to help preach the benefits of YARN.

Also included in the HDP 2.0 preview is Tez, an MapReduce generaliser framework, and Stinger, Hortonworks’ internal initiative to boost the speed of SQL queries against Hadoop.

The community preview of HDP 2.0 is available for VMware and Oracle Virtual Box environments, with a full preview version expected in the next few days.

Image courtesy of Hortonworks,

Inline Feedbacks
View all comments