How to catch critical bugs before your users do
We’ve been really enjoying JAX DevOps in London this week. Stefanos Zachariadis, one of our speakers, was kind enough to sit down and answer a few questions about the importance of data testing, how to keep end users from getting bugs, and why you should invest in a data “cleanser”.
His talk, “After Acceptance: Techniques for making sense of what your system is producing and detecting issues before customers do” focuses on how to sanity check a live system using end to end testing, limiting interference with real interactions. It also suggests ways to integrate real production data into a continuous delivery pipeline and assert on the validity of the production outputs.
JAXenter: What are the techniques for making sense of what your system is producing?
Stefanos Zachariadis: Agile development with end to end tests is great and superior to what came before it — but it’s not perfect. In most agile environments, testing stops at the point your code is released to production. This is the point that your code will come into contact for the first time with the data and state that previous releases of your software have produced. This data is usually very different to the data generated in a test environment.
Acceptance tests usually represent only a simplified version of user interactions. In production, data will almost certainly be generated by code paths that have not been fully exercised in tests. And production data usually exists in far greater volumes than testing data. If the current version of the system is stateful, how do we know that future versions will consider that state valid and act upon it appropriately? This problem becomes harder in a microservices environment where there may be different versions of a service running at the same time.
The solution to this is data testing. We define data testing as a type of integration testing that happens after the acceptance testing phase and that integrates data with code prior to it being released to production.
JAXenter: How can you detect issues before customers do?
Stefanos Zachariadis: Data testing allows us to write tests that detect a broad category of potential bugs before the code hits production and customers detect them. It allows us to write tests around the validity and invariance of the data. Data testing also allows us to test the code with the actual space complexity of the data. That means we can test our data migrations before they hit production. For more details, come to my session :-).
JAXenter: How can one integrate real production data into a continuous delivery pipeline and assert on the validity of the production outputs?
Stefanos Zachariadis: I wouldn’t want to give everything away so early, but there are two key items that are required:
- Data is owned by the application and therefore data migration is integral to it. This means that the migration process is also owned by the application and new migrations should ship with the application and be performed with the deployment of every new session.
- A data sanitisation service, a cleanser if you will, is a service that we’ll define that integrates production data early in the continuous delivery pipeline. Why is it called a cleanser? Come to the session :-)
JAXenter: What should participants learn from your session?
Stefanos Zachariadis: Your data is a rich source of information that’s there waiting to be mined. Acceptance testing is enough to tell us that a feature is done or not but it leaves wide open a whole category of data related issues that your code may suffer from. This is not to say that acceptance testing should deal with it – no, that’s not its purpose. But by bringing your production data into your continuous delivery pipeline you can catch a large number of critical bugs before your users do, and enjoy continuous releases with fewer unexpected surprises. I hope that the session helps participants identify the particular issues that data testing can catch *after* the acceptance testing phase but before users do.
Please complete the following questions:
Dev and Ops work best together if … if they are the same team – not different teams managed differently.
The biggest obstacle for DevOps is … a strict separation between Dev and Ops, i.e. “let’s ask ops to do this” even though dev is perfectly capable of doing it themselves. After all, we’re all on the same team trying to achieve the same result!
What promotes employee satisfaction is … finding meaning in your work. And the best way to find meaning is to continuously get your work in the hands of users, i.e. continuous delivery!
The biggest advantage of autonomously-working teams is … speed of delivery. Team ownership. Happiness.
It is important for a positive company culture to … acknowledge good work when that happens. As a dev, it feels great to get your work out. An occasional acknowledgement (or even a beer!) to recognize good work is important for forming a positive company culture.
Stefanos Zachariadis will be delivering one talk at JAX DevOps which will focus on how to avoiding those pesky acceptance errors and how to catch them before your users find them.