Uber engineer: “You don’t have to reinvent the wheel to solve common infrastructure problems”
Many companies still use expensive proprietary hardware and software to provide load-balancing and routing for their users and services. We talked to JAX DevOps speaker Andrey Sibiryov about how IPVS can be used to automatically configure load balancing and routing for Docker containers using a simple Go daemon and a Docker plugin.
In this interview, Andrey Sibiryov, a Senior Infrastructure Engineer at Uber Technologies and speaker at the upcoming JAX DevOps, is talking about IPVS —an open-source technology which has been a part of the mainline Linux kernel for over a decade and can be at least as good as the expensive proprietary hardware and software used by several companies to provide load-balancing and routing for their users and services.
JAXenter: During your JAX DevOps session, you will present an open-source technology that has been a part of the mainline Linux kernel for more than 10 years. Why do you think companies are still using expensive proprietary hardware and software to provide load-balancing and routing for their users and services even when they have another option?
Andrey Sibiryov: Implementing own load-balancing and routing solutions in terms of IPVS or any other open-source technology requires world-class engineers with a specific set of skills, an engineering culture where innovation in the infrastructure is rewarded, and time. Many companies focus almost exclusively on the product while accumulating substantial technical debt in infrastructure. Unfortunately, it inhibits building the operational expertise required to run complex distributed systems which eventually surfaces in the interactive part of the business in a variety of forms ranging from annoying lag to service outages and a complete downtime.
JAXenter: What is IPVS and how can it make our lives easier?
Andrey Sibiryov: IPVS is a software load balancer implemented as a Linux kernel module, operating entirely in kernel space. It has been a part of mainline Linux kernel for many years and is used in production by quite a few well-known companies such as Google, Facebook, LinkedIn, Dropbox, GitHub, Yandex, Alibaba and more. Load balancing is an essential part of virtually any infrastructure and having a built-in, production-tested, performant solution available with no strings attached sounds like something that many engineers would appreciate.
JAXenter: What has changed about IPVS in its decade of existence?
Andrey Sibiryov: This technology is actually pretty stable. The general problem of load balancing requests is rather straightforward and there’s not much to add there. Nevertheless, over the course of many years, IPVS improved off a steady stream of contributions from the community: it got many new scheduling algorithms in addition to classic ones like round-robin and least-connections, IPv6 support, DNS load balancing via “one packet mode”, netlink interface in place of obscure syscalls for configuration and lots of smaller improvements & bug fixes.
Many companies focus almost exclusively on the product while accumulating substantial technical debt in infrastructure.
JAXenter: What is the key message of your session at DevOpsCon that every participant should retain after visiting your talk?
Andrey Sibiryov: There’s a simple, effective and straightforward solution to the majority of common infrastructure problems that doesn’t involve reinventing the wheel.
JAXenter: What differentiates IPVS from other load-balancing software?
Andrey Sibiryov: I’d say three main things: first of all, IPVS is built-in and operates entirely inside the kernel. This removes the requirement to make expensive memory copies of incoming and outgoing packets to and from userspace and avoids costly context switches – other load balancers simply cannot achieve this kind of performance. Secondly, IPVS supports zero-downtime dynamic reconfiguration of virtual services and backends. For example, other solutions often introduce a brief downtime that requires either complicated setups or custom kernel patches to be eliminated. Thirdly, IPVS supports UDP load balancing. While it’s not a common thing to do, when it’s needed – IPVS is one of the very few straightforward ways to do it.
Thank you very much!
Andrey Sibiryov will be delivering one talk at JAX DevOps which will focus on how how IPVS can be used to automatically configure load balancing and routing for Docker containers using a simple Go daemon and a Docker plugin.