How YouTube are using Go to augment MySQL
Cutting-edge meets old school in YouTube with Vitess, which allows MySQL to be run at Google-size scale.
With so much talk of the exciting potential of NoSQL databases, it’s easy to forget that MySQL is still the dominant player.
YouTube love MySQL so much that they have augmented it to keep
up with the video site’s enormous traffic in an open-source project
The first product to emerge is Vtocc, which is designed to efficiently multiplex large numbers of incoming connections. At Usenix LISA conference this week, Sugu Sougoumarane and Mike Solomon from YouTube spoke about the project and their experiences of working with Go.
MySQL, the unfashionable open-source relational database, is preferred at YouTube for its reliability and simply being a known quantity. However, its major drawback is scaling to a necessary size. After all, it takes some serious engineering talent to serve Gangnam Style 970 million times.
“The major problem with MySQL is that once you get to a certain point, you spend a lot of time managing hardware and how many instances you have,” said Solomon. “We want to automate that chunk. We want to take every action that is complicated and error prone and make it heal itself.”
Unwilling to mess with MySQL itself, however, they opted to write a new product, Vitess, that runs in conjunction with MySQL. Its first branch, Vtocc, has “been very steady, and it has all the necessary tools for you to use in a production environment,” said Sougoumarane. It allows thousands of SQL queries to be consolidated into batches for efficiency, caching results to deal with duplicate requests.
With YouTube sticking doggedly to old hand MySQL, it’s perhaps surprising that they chose to use Google’s own server-side language, which only just celebrated its third birthday, to build Vitess. Though now offered on Google App Engine, it still has yet to find mainstream adoption – except among Google.
Sougoumarane described the language’s features as “well-thought-out”, adding: “They help you compose things in a much more elegant way than traditional languages.” Go’s thread management features are particularly useful, they said, though admitting that its error handling, scheduling and garbage collection is currently left wanting.