What is “Continuous Testing” and How is it Even Possible?
Continuous Integration calls for code to be always working. Continuous Delivery and Continuous Deployment just up the stakes, saying that we must be able to roll out a working configuration to production at any time. The only way to prove code and configurations are working is to test them. That’s why we need Continuous Testing.
Before you ask yourself how you could automate all this with approaches from the past, consider that as DevOps and autonomous “two pizza” sized development teams take off and big apps are broken down into smaller independent components, things are not just happening faster, but there are more things happening. How do you keep up with that? You can just push untested code into production and let some subset of your users test for you if you like the excitement of waiting until users are impacted before fixing things, or you can find a new way to test before they are impacted.
Continuous Testing is that new way. Face it: you’ve got a large and ever growing list of “application endpoints” that must be “always working” in order for you to achieve Continuous Delivery. Where are all of these new tests going to come from if everything is becoming “continuous”? Is it even possible to create tests as often as every few minutes, day in and day out, if needed?
Don’t just shift left. Shift all the way left to those who know your code best.
Development shops that shift testing “all the way left” know the answer. Today’s software engineers live in a world where DevOps is becoming the norm and bugs in production are a developer’s personal problem. Dev’s are highly motivated to prove their code does what it is supposed to. They want to know immediately (and long before any deploy to production) if code changes they are working on break something.
So what does shifting “all the way left” mean? It means that tests come from the same place that the units of code do. This can be the developer, or an agile tester working on the same small team. The secret is to reduce tests to simple but effective human readable code, which we call “test blueprints.” Development teams test locally for instant feedback while coding and then check both code and test blueprints into source. Never again run a build only to find that the code works but some existing set of (downstream built) tests are now broken.
In organizations making this transition, performance engineers and “engineer in test” automation experts shift from minding a backlog of test-building work to facilitating testing by all. If you want more context for this shift, consider viewing our webinar Moving from LoadRunner to Open Source Testing Tools: Shifting Left & Democratization.
Always have a working test. Run any test in seconds when you need it. Never wait.
What’s becoming clear as client after client embarks on this new journey is that Continuous Testing is not only possible, it is the “new normal” when developers are equipped with low-friction testing capabilities and expected to check in working test blueprints with any new or updated code. With testing shifted all the way left, Continuous Delivery organizations are always just seconds away from proving code is ready and working to acceptable service levels.
Deciding what to test, assembling the test and executing it with no human intervention is just basic CI once you shift left. Whether the scope is one code module or an entire release, the right test is composed on the fly by tracing dependencies, just as you do when building the code itself. Better still, humans are only involved and assigned new work if the test fails. This all happens programmatically within seconds of a code check-in or a build or trial deployment.
“Within seconds” by the way, is the only sort of “performance testing backlog” you can afford in Continuous Delivery.
At first glance it may appear too hard. Look again. Always look again (Mary Anne Radmacher)
So, welcome to the Continuous Testing era. Continuous Integration itself may have sounded impossible to many when it first emerged, but long ago “always having a working build” became the new normal. Continuous Testing will follow the same path, only faster. You may still be wondering, “Is this still just science fiction at this point?” Based on what we already see with our clients, William Gibson was right when he said, "The future is already here — it's just not very evenly distributed."
Have a look at our mini-video and white paper on Continuous Testing to speed your own transition.
You might also be interested in viewing our on-demand webcast on Continuous Testing for Containerized Applications, featuring special guest Codefresh.