Shutterstock’s polyglot approach
Being a multilingual, multicultural company doesn’t just bring benefits on a level of corporate culture. Shutterstock search engineer Chris Becker explains why enterprises need to stop speaking just one language.
A technology company must consider the technology stack and programming language that it’s built on. Everything else flows from those decisions. It will define the kinds of engineers who are hired and how they will fare.
During Shutterstock’s early days, a team of Perl developers built the framework for the site. They chose PERL for the benefits of CPAN and flexibility that language would bring. However, it opened up a new problem – we could only hire those familiar with Perl. We hired some excellent and skilled engineers, but many others were left behind because of their inexperience with and lack of exposure to Perl. In a way, it limited our ability to grow as a company.
But in recent years, Shutterstock has grown more “multilingual.” Today, we have services written in Node.js, Ruby, and Java; data processing tools written in Python; a few of our sites written in PHP; and apps written in Objective-C.
Developers specialize in each language, but we communicate across different languages on a regular basis to debug, write new features, or build new apps and services. Getting from there to here was no easy task. Here are a few strategic decisions and technology choices that have facilitated our evolution:
Service Oriented Architectures
First, we built out all our core functionality into services. Each service could be written in any language while providing a language-agnostic interface through REST frameworks. It has allowed us to write separate pieces of functionality in the language most suited to it.
For example, search makes use of Lucene & Solr, so Java made sense there. For our translation services, Unicode support is highly important, so Perl was the strongest option.
Between languages there are numerous frameworks and standards that have been inspired or replicated by one another. When possible, we try to use one of those common technologies in our services.
All of our services provide RESTful interfaces, and internally we use Sinatra-inspired frameworks for implementing them (Dancer for Perl, Slim for PHP, Express for Node, etc). For templating we use Django inspired frameworks such as Template::Swig for Perl, Twig for PHP, and Liquid for Ruby. By using these frameworks we can help improve the learning curve when a developer jumps between languages.
Often, the biggest obstacle blocking new developers is technical bureaucracy needed to manage each runtime — managing library dependencies, environment paths, and all the command line settings and flags needed to do common tasks. Shutterstock simplifies all of that with Rockstack. Rockstack provides a standardized interface for building, running, and testing code in any of its supported runtimes (currently: Perl, PHP, Python, Ruby, and Java).
Not only does Rockstack give our developers a standard interface for building, testing, and running code, but it also supplies our build and deployment system with one standard set of commands for running those operations as well for any language. Rockstack is used by our Jenkins cluster for running builds and tests, and our home-grown deployment system makes use of it for launching applications in dev, QA, and production.
In order to create a standardized method for testing all the services we have running, we developed (and open sourced!) NTF (Network Testing Framework). NTF lets us write tests that hit special resources on our services’ APIs to provide status information that show the service is running in proper form. NTF supplements our collection of unit and integration tests by constantly running in production and telling us if any functionality has been impaired in any of our services.
We also want our developers to learn and evolve their skillsets. We host internal meetups for Shutterstock’s Node developers, PHP Developers, and Ruby developers where they can get fresh looks and feedback on code in progress.
These meetups are a great way for them to continue professional development and also to meet others who are tackling similar projects. There’s no technology to replace face-to-face communication, and what great ideas and methods can come from it.
We post all the code for every Shutterstock site and service on our internal Github. Everyone can find what we’ve been working on. If you have an idea for a feature, you can fork off a branch and submit a pull request to the shepherd of that service. This openness goes beyond transparency; it encourages people to try new things and to see what others are implementing.
Our strategies and tools echo our mission. We want engineers to think with a language-agnostic approach to better work across multiple languages. It helps us to dream bigger and not get sidelined by limitations. As the world of programming languages becomes much more fragmented, it’s becoming more important than ever from a business perspective to develop multilingual-friendly approaches.
We’ve come a long way since our early days, but there’s still a lot more we can do. We’re always reassessing our process and culture to make both the code and system more familiar to those who want to come work with us.
This has been adapted from a post that originally ran on the Shutterstock Tech blog. You can read the original here.