Netflix adds to open source with Mantis and Polynote
We love open source! And now, two new recently open sourced pieces of software from Netflix have arrived. Mantis allows you to build realtime stream processing applications and Polynote is an IDE-inspired polyglot notebook. See what both of these can do, some of their use cases, and what important features set them apart.
148 million video streaming subscriptions don’t run themselves. The Netflix tech department develops internal tools and engineering systems to help the company juggle its massive reach. The video streaming giant recently announced that they are making two of their platforms open source. Previously the company’s tech department open sourced a microservices orchestration engine, a cloud gateway, and a container management platform, just to name a few.
Give a warm welcome to Mantis, an open source platform that helps build realtime stream processing applications and Polynote, an IDE-inspired polyglot notebook.
Take a break from binge-watching Bojack Horseman and take a look at these open source tools and their capabilities.
Mantis has been in production since 2014 and according to its documentation, it “processes trillions of events and peta-bytes of data every day”. This high amount of data processing means that it’s perfect for high volume cases. It offers stream processing-as-a-service in a self-contained platform.
From the documentation:
Similar to micro-services deployed in a cloud, Mantis applications (jobs) are deployed on the Mantis platform. The Mantis platform provides the APIs to manage the life cycle of jobs (like deploy, update, and terminate), manages the underlying resources by containerizing a common pool of servers, and, similar to a traditional micro-service cloud, allows jobs to discover and communicate with other jobs.
What can you use Mantis for?
According to the documentation, some of Mantis’ use cases include:
- Realtime monitoring: At Netflix, Mantis monitors the massive amounts of Streams per Second in order to determine the current health of the streaming service. If the number of Streams per Second dips below a certain amount or shows an abnormal change, Mantis detects the change in seconds.
- Contextual alerting: On a large scale, Netflix analyzes interactions between microservices in order to determine changes and anomalies. According to Netflix, Mantis’ realtime nature cuts down the time of detecting anomalies to seconds.
- Chaos testing: What’s the worst thing that can happen? Test it! Chaos testing and experimental is an exercise of system resiliency in apps. Mantis monitors chaos testing so that teams can analyze the data and determine what is causing anomalies.
- Alerting on Log: Use Mantis to create alerts attached to certain patterns in application logs. Mantis will then watch in realtime for these patterns alerts you when it detects them in application logs.
Polynote is an open source polyglot notebook that supports Scala, Apache Spark integration, and multi-language interoperability with Scala, Python, and SQL. It focuses primarily on reproducible notebooks and better visibility. For example, it has no hidden state, unlike other notebooks.
Inside Netflix, Polynote allows machine learning researchers to integrate their JVM-based Scala-heavy machine learning platform with Python machine learning libraries. If you’ve ever watched a recommended movie from Netflix, then you’ve experienced the work of Polynote.
Important features of Polynote
- IDE-like features: Interactive auto-complete works as you type, with a rich text editor, LaTeX equation support, and integration with the Monaco editor. Polynote also highlights errors in the code and provides parameter hints.
- Seamless multi-language support: Mix programming languages in polyglot notebooks. The same notebook can support different languages in each cell. For instance, use Python and matplotlib in combination. For further information about how language mixing works and a list of some common tasks, read this section of the documentation.
- Runtime insights: Polynote displays what the kernel is doing in the tasks area, so you can keep track of everything easily.
- Reproducible: With its immutable data model, Polynote takes each cell’s position into consideration when executing it, allowing for reproducibility. Sharing data between notebooks works seamlessly.
- Data visualization: It can integrate with matplotlib and the Vega visualization language to create rich data visualizations.
- Apache Spark integration: This is an optional feature and by default, Polynote runs without Apache Spark. However, if you have Apache Spark installed and wish to use it with Polynote, you can do so and use it with kernel isolation.
Right now, Polynote has only been tested on Linux and MacOS using Chrome. Thus, users should take this into consideration when trying it on a different OS or browser.