DevOps, Cloud and Microservices: “All Hail the Developer King/Queen”
Microservices are mainstream, security needs to be talked about more and development and operations should be best friends. Daniel Bryant, Principal Consultant at OpenCredo, shares the lessons he’s learnt from attending the JAX London 2015.
Last month Steve Poole and I were once again back at the always informative JAX London conference talking about DevOps and the Cloud. This presentation built upon our previous DevOps talk that was presented last year, and focused on the experiences that Steve and I had encountered over the last year (the slides for our 2014 “Moving to a DevOps” mode talk can be found on SlideShare, and the video on Parleys).
I always enjoy presenting with Steve, as his experiences are typically complementary with mine. Steve works as a software engineer at IBM, and leads the DevOps efforts in creating a shared development/operations platform, and also driving the cultural changes that accompany the DevOps philosophy. He has in-depth knowledge of the IBM efforts and also lots of experience with the changes over time in one organisation. In my role as a principal consultant at OpenCredo I’m in the fortunate position to work with many different organisations (that come in all shapes and sizes), and I get to drive the transformation, disruption and evolution of software delivery on a daily-basis that come along with implementing a DevOps practice.
So what did we both learn over the past year when working with DevOps and the cloud?
Part 1 – Painful lessons learnt
- Microservices are mainstream, and are causing many challenges as organisations adjust to this new way of creating (and delivering) software
- Microservices increase operational complexity
- Observing and debugging microservices is often very challenging
- Sharing datastores sounds like it saves effort, but introduces coupling between applications
- Big-bang versioning of applications means putting existing unchanged use cases at risk
- Scaling is challenging when you try to duplicate whole systems (see the great ‘Art of Scalability‘ book)
- Infrastructure – it’s much more important than we realised
Part 2 – BYOP: Build Your Own Platform?
- As an industry, we are beginning to agree on the components needed within a microservice platform, which is being driven by the likes of Adrian Cockcroft
- Cloud definition languages and automated tooling are becoming mature e.g. Ansible, CloudFormation and HashiCorp’s Terraform
- There is a lot (too much?) of technology that promises to solve the problems with specific parts of the platform
- There are recipes available on the Internet that explain how to build a microservices platform. People like Joe Beda have built these things to run at large (Google) scale – we recommend listening to them!
- The key question is should we actually try and build a platform?
- Don’t dismiss the benefits offered by PaaS, such as CloudFoundry or Deis
Part 3 – Safety first!
- Safety and security are often the ‘elephant in the room’ that developers (and the supporting business team) are reluctant to talk about
- The introduction of cloud and container technology means that you, as a developer, are responsible for security and a lot more ‘ilities’ (throughout the application/operation stack)
- You must understand the basics of security and compliance – for example, where is the data in your systems of record (SoR) currently located? Where should it be?
- As a developer, how do you test your services, containers and SaaS products for security and compliance?
- As a developer, the operations team should be your best friends. This is what they do – make sure you work closely with them
Part 4 – Sharing is caring
- Cultural is still critical for the successful implementation of DevOps (people and process should drive tooling, not the other way around)
- The Puppet Labs 2015 State of DevOps and Nicole Forsgren’s analysis of this make for essential reading for those working at the development/operations coalface, all the way through to the C*O level
- Building a generative, supportive and empathetic culture is vital for sustainable success when developing software
- DevOps is about sharing responsibility – both the successes and pains
- As a developer, you must take responsibility for the delivery and operation of the software you write
- Small cross-functional ‘product’ teams appear to be the best way to build and operate microservices
- Monitoring, and the associated response by people, is vital for handling the inevitable issues that occur when running complex software
- Practice dealing with real failure scenarios, for example, by running ‘game days’ or ‘disaster in recovery training (DiRT)’
Part 5 – The Future of ‘DevOps’
- We asked “DevOps – false start, dead, or alive and kicking?”
- We see quite a few ‘DevOps’ teams that may actually be operation teams using Puppet or Chef
- Google and other ‘Unicorns’ are increasingly not ‘doing’ DevOps, but are creating teams with operationally-savvy fullstack developers, platform/tooling specialist and site reliability engineers (SREs). Is this similar to ‘doing’ Agile versus ‘being’ agile?
- Cloud operations increasingly involves programming. For example, “programmable infrastructure” languages and tooling such as Ansible, CloudFormation and Terraform are all very powerful solutions. Developers must work closely with operations, and share programming good practice such as modularisation, cohesion and coupling (otherwise we are doomed to see bad programming practices emerge from operations)
- Serverless infrastructure is emerging… (is ‘winter’ coming for operations?)
Part 6 – Conclusions
- Consider PaaS, as opposed to building your own platform…
- … but be wary of vendor lock-in (create abstraction and anti-corruption layers as appropriate)
- Containers are here to stay – they are awesome, but you need to learn how best to create ‘container-native’ applications i.e. discover good practices for ‘microservices’
- Think ‘safety first’! Security and the other ‘ilities’ are now the responsibility of the developer, as much as they are the operator
- The rise of the professional cyber criminal is real, and it is only a matter of time before you (and your organisation) are affected. Deal with it!
- Development and operations should be best friends
- Golang appears to be dominating container, infrastructure and tooling
- Learn about clustering and scheduling (this is the future, and the unicorns have been working with this technology for quite some time!)
- Everything in the cloud is on a network. Learn about networking – virtualisation, overlays, the OSI model etc
- Learning about multi-cloud technologies appears to be a solid bet. The rise of the ‘cloud broker ecosystem’ is fast becoming a reality, and tooling such as OpenStack, Terraform and Ansible make this easy to implement
You can find the slides for the talk on SlideShare:
“Last year we talked about DevOps, what it was, why it was important and how to get started. Boy, was it scary. Now we’re wiser. More battle-scarred. The scale of the challenge for application writers exploiting cloud and DevOps is clearer, but so is the path forward. Understanding the DevOps approach is important but equally you must understand specific deployment technologies. How to exploit them and how they affect the design of applications. Whether creating simple applications or sophisticated microservice architectures many of the challenges are the same.”
Anyone who knows me knows that I love to read! Here is my reading list for this topic:
- The Practice of Cloud System Administration by Limoncelli et al
- Threat Modelling: Designing for Security by Adam Shostack
- DevOps Troubleshooting: Linux Server Best Practices by Kyle Rankin
- The Phoenix Project by Kim et al
- The Lean Enterprise by Humble, Molesky and O’Reilly
- Drift into Failure by Sidney Dekker
- The Art of Scalability by Martin Abbot and Michael Fisher
- Building Microservices by Sam Newman
- Continuous Delivery by Humble and Farley
- More Agile Testing by Janet Gregory and Lisa Crispin