Lyft open sources Flyte for developing machine learning workflows
Ridesharing company Lyft has open sourced Flyte, its distributed processing platform for machine learning workflows that is being used in different Lyft teams including Pricing, Data Science and Fraud. Let’s see how the open source tool can benefit ML workflows.
Lyft announced in a Medium blog post that they are making Flyte publicly available. Flyte is a structured programming and distributed processing platform for developing, executing and monitoring scalable machine learning workflows.
Built on Kubernetes
Flyte is built on Kubernetes to provide benefits such as portability, scalability and reliability. The platform consists of tasks that can be linked together in the form of directed acyclic graphs. It is built to scale and Lyft has been using it in production over various teams.
The following graphic shows how Flyte is designed:
Developing, executing and monitoring ML workflows
A graph is needed for creating a workflow, and Flyte offers developer tools to help ensure it meets the specifications. The documentation claims that, in only 20 lines of code, “a user can define a custom workflow, submit the graph to Flyte, and snooze while Flyte fulfills the execution on a schedule, or on-demand.”
Workflow execution is carried out using docker containers. Flyte supports distributed workflows, automatic retries and custom notifications.
Monitoring takes place throughout the workflow’s lifecycle, and the current status can be tracked via Flyte web interface. The web interface lets you take a closer look at the workflow steps and receive detailed error messages.
As an example, Lyft showed this workflow by Tim Wang on the Rider Pricing team as displayed in the Flyte UI:
Flyte is now available on GitHub, and the Python SDK Flytekit for authoring, testing, deploying and interacting with workflows and tasks is on board as well.