A different CaffeOnSpark

Yahoo open-sources TensorFlowOnSpark, brings deep learning to the masses

Gabriela Motroc

It’s been one year since Yahoo open-sourced CaffeOnSpark so the tech giant has found a way to celebrate it — by open-sourcing TensorFlowOnSpark, its latest open source framework for distributed deep learning on big data clusters.

Yahoo has just open-sourced TensorFlowOnSpark, a framework which enables distributed TensorFlow execution on Spark and Hadoop clusters. TensorFlowOnSpark is designed to work along with SparkSQL, MLlib, and other Spark libraries in a single pipeline or program, according to Yahoo’s blog post.

The framework supports all types of TensorFlow programs, enabling both asynchronous and synchronous training and inferencing. It supports model parallelism and data parallelism, as well as TensorFlow tools such as TensorBoard on Spark clusters.

What problem does TensorFlowOnSpark solve?

Yahoo has discovered that, in order to gain insight from massive amounts of data, they need to deploy distributed deep learning. “Existing DL frameworks often require us to set up separate clusters for deep learning, forcing us to create multiple programs for a machine learning pipeline. Having separate clusters requires us to transfer large datasets between them, introducing unwanted system complexity and end-to-end learning latency.”

“In October 2016, TensorFlow introduced HDFS support. Outside of the Google cloud, however, users still needed a dedicated cluster for TensorFlow applications. TensorFlow programs could not be deployed on existing big data clusters, thus increasing the cost and latency for those who wanted to take advantage of this technology at scale,” Yahoo explained before revealing the purpose of TensorFlowOnSpark.

TensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark clusters. By combining salient features from deep learning framework TensorFlow and big-data frameworks Apache Spark and Apache Hadoop, TensorFlowOnSpark enables distributed deep learning on a cluster of GPU and CPU servers.

Any TensorFlow program can be easily modified to work with TensorFlowOnSpark. Typically, changing fewer than 10 lines of Python code are needed.


Remembering CaffeOnSpark

Last year, Yahoo open-sourced CaffeOnSpark, a framework which allows deep learning training and testing to be embedded into Spark applications. Since then, CaffeOnSpark has been upgraded with LSTM support, a new data layer, training and test interleaving, a Python API, and deployment on docker containers.

Even though CaffeOnSpark has gathered positive reactions from users, the question remains: “What about those who use the deep learning framework TensorFlow? We’re taking a page from our own playbook and doing for TensorFlow for what we did for Caffe,” Yahoo announced.

Although Yahoo has just open-sourced TensorFlowOnSpark, the tech giant will continue to work on advancing this framework as they continue to do with CaffeOnSpark. You can find TensorFlowOnSpark on the Yahoo GitHub.


Gabriela Motroc
Gabriela Motroc was editor of and JAX Magazine. Before working at Software & Support Media Group, she studied International Communication Management at the Hague University of Applied Sciences.

Inline Feedbacks
View all comments