Netflix unleashes Chaos Monkey
Netflix continues its open source pledge, bringing an AWS drill sergeant to make sure your infrastructure is ready for blackout.
Over the past few months, video streaming behemoth Netflix have
been giving back to the developer community, open sourcing a number
of interesting tools, namely two Cassandra helpers
Astyanax and Priam.
Now the technical team say it’s the right time to uncage the leader
of their so-called ‘Simian
Army’, Chaos Monkey, a veteran that tests how malleable
their Amazon Web Services infrastructure is during times of
Available through Github,
Chaos Monkey hunts down groups of systems and randomly
terminates virtual machine instances in applications, to simulate
what would happen in a disaster scenario. Writing in the Netflix
technical blog, Cory
Bennett and Ariel
Tseitlin tell us why Chaos Monkey should be present in
We have found that the best defense against major unexpected
failures is to fail often. By frequently causing failures, we force
our services to be built in a way that is more resilient.
Failures happen and they inevitably happen when least desired or
expected. If your application can’t tolerate an instance failure
would you rather find out by being paged at 3am or when you’re in
the office and have had your morning coffee? Even if you are
confident that your architecture can tolerate an instance failure,
are you sure it will still be able to next week? How about next
month? Software is complex and dynamic and that “simple fix” you
put in place last week could have undesired consequences.
Whilst this might sound like some sort of mercenary taking out
your infrastructure, the benefits of having Chaos Monkey there are
numerous. It makes sure engineers are responsive and alert when
it’s all hands to the pump, making sure that your infrastructure
can cope and has a battleplan in Code Red scenarios.
If anyone is well placed to deal with situations like this, it’s
Netflix with their massive cloud infrastructure. Crucially, with
scrutiny on Amazon Web Services growing with more blackouts, it
always seems to be Netflix who come out of it unscathed, no doubt
in part due to their Simian Army – a ruthless squadron of tools
that puts the Netflix architecture through its paces.
The stats are impressive too:
There are many failure scenarios that Chaos Monkey helps us
detect. Over the last year Chaos Monkey has terminated over 65,000
instances running in our production and testing environments. Most
of the time nobody notices, but we continue to find surprises
caused by Chaos Monkey which allows us to isolate and resolve them
so they don’t happen again.
With some tweaking to Chaos Monkey’s REST
API, you can initiate a testing scheme to make sure you
aren’t caught out. The great news is Netflix intends to deploy its
entire army moving forward, with environment tidier Janitor Monkey
and something they call Chaos Gorilla, which simulates an entire
AWS zone outage. Get this tool now to make sure AWS doesn’t make a
monkey out of you.
Image courtesy of FHGitarre on