Processing platform for ML workflows

Lyft open sources Flyte for developing machine learning workflows

Maika Möbus
machine learning
Alexander Oganezov /

Ridesharing company Lyft has open sourced Flyte, its distributed processing platform for machine learning workflows that is being used in different Lyft teams including Pricing, Data Science and Fraud. Let’s see how the open source tool can benefit ML workflows.

Lyft announced in a Medium blog post that they are making Flyte publicly available. Flyte is a structured programming and distributed processing platform for developing, executing and monitoring scalable machine learning workflows.

SEE ALSO: Meet Manifold: Uber’s machine learning model debugging tool goes open source

Built on Kubernetes

Flyte is built on Kubernetes to provide benefits such as portability, scalability and reliability. The platform consists of tasks that can be linked together in the form of directed acyclic graphs. It is built to scale and Lyft has been using it in production over various teams.

The following graphic shows how Flyte is designed:

Flyte’s architecture. Source.

Developing, executing and monitoring ML workflows

A graph is needed for creating a workflow, and Flyte offers developer tools to help ensure it meets the specifications. The documentation claims that, in only 20 lines of code, “a user can define a custom workflow, submit the graph to Flyte, and snooze while Flyte fulfills the execution on a schedule, or on-demand.”

Workflow execution is carried out using docker containers. Flyte supports distributed workflows, automatic retries and custom notifications.

Monitoring takes place throughout the workflow’s lifecycle, and the current status can be tracked via Flyte web interface. The web interface lets you take a closer look at the workflow steps and receive detailed error messages.

As an example, Lyft showed this workflow by Tim Wang on the Rider Pricing team as displayed in the Flyte UI:

Pricing Workflow by Tim Wang in Flyte UI. Source.

SEE ALSO: What is Kubernetes and how does it relate to Docker?

Flyte is now available on GitHub, and the Python SDK Flytekit for authoring, testing, deploying and interacting with workflows and tasks is on board as well.

The official website and the Medium blog post offer further details.

Maika Möbus
Maika Möbus has been an editor for Software & Support Media since January 2019. She studied Sociology at Goethe University Frankfurt and Johannes Gutenberg University Mainz.

Inline Feedbacks
View all comments