Hi Rosie,
There’s a lot of really good material that’s already been provided by others in this thread but I wanted to chime in with my thoughts and provide some material. Continuous testing came about as organizations are struggling to understand the level of risk they were introducing into their applications during the software development process. In the early days continuous testing represented a form of test automation that encompassed all of the different testing activities (i.e. static analysis, unit testing, API contract testing, and functional testing, regression testing, and even nonfunctional testing like security and performance) and running them in an automated way. The key though was not simple automation but the feedback process that Ashley was referring to.
Fast feedback enables meaningful actions to be taken as a part of the testing process. I’ve always use the term “policy gates”. The idea is that every night when I run my tests I will have certain functional violations that will trigger. But I set up a threshold for risk. An example would be only for severity one violations and up to three functional regression tasks would allow the application to be deployed into a staging environment. If those thresholds were not met then I would stop the deployment promotion process and deliver the relevant tasks back to the initial testers or developers there were identified as the authors of the code or test.
I actually think these principles involved from test automation to continuous testing as agile and DevOps became key initiatives. Agile (agile testing not agile development) helps us focus on new functions and features that development is introduced into the product so that we can get meaningful information back to our stakeholders about the level of risk were introducing and stakeholders can make a decision about prioritization of tasks prior to release. This ultimately helps development and product management make a more informed decision about where they’re going and actually scope things in a meaningful way. DevOps says we need to be able to get these artifacts deployed in a much faster way so that we can get the second type of feedback which is stakeholder and customer feedback. So let’s start building infrastructure as code so that we can deploy artifacts a an automated way. The problem with all of this was that without thoroughly validating the application as it sped its way down the DevOps delivery pipeline. We ran the risk of defects leaking into production.
I think that’s where continuous testing really hit its stride. Because it evolved from nightly based test execution to continuous test execution where the different types of test cases could be executed as a function of code check-in. In the earlier stages of development. There’s technologies out there that help us understand what the relevant tests are to execute and by aligning that test execution to the initial requirement we get immediate feedback into the impact of development code change on our overall application much much earlier than when the application is deployed.
Now I have to be upfront with you, I do work for a testing tools company so you’ll have to take my input with a grain of salt but I explored this topic in much greater detail in a blog I wrote call continuous testing for DevOps
I hope this helps
