So, what is continuous software testing?

It feels like a hot topic at the moment. But when I do a search on it, I get lots of fancy diagrams that don’t mean much, or articles that are shallow.

What are some good resources for learning about how to do continuous software testing? Where would testers go to learning how to do testing better in a dev ops environment?

I’d also love to understand how people are defining continuous testing, and what actually makes it different?

Katrina Clokie’s book is a given - https://leanpub.com/testingindevops

3 Likes

To me, it’s about fast feedback at various points in your pipeline. Instead of an all encompassing regression suite, how can I break that up into meaningful, fast test suites that may be run upon commit, or merge, or deploy (just examples, not limiting it to those points) - and in some cases re-run. Not saying a regression suite isn’t necessary, but is it something we need until I’ve identified other opportunities to test earlier and faster? I like to look at the tenants of DevOps of Collaboration, Iterative Improvements, and Continuous Learning - how do we iterate on our testing, how do we improve upon it, and what are we learning from it - collaborating as a team to understand.

Working on writing more about it, but a lot of what is in my head is based on several books (of course including Katrina’s book you already mentioned), though I’m making a start at https://github.com/ahunsberger/TestSuiteDesign as I make my company work a bit more generalized and write more in depth about this. This is also very much based on Katrina’s test automation canvas (http://katrinatester.blogspot.com/2017/07/test-automation-canvas.html), and have adopted her lean canvas to areas that I’ve found useful in exploring with teams. (like I said, this is a massive work in progress!)

  • A Practical Guide to Testing in DevOps (Katrina Clokie)
  • Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation (Jez Humble, David Farley)
  • Accelerate: Building and Scaling High Performing Technology Organizations (Dr. Nicole Forsgren, PhD; Jez Humble, and Gene Kim)
  • How Google Tests Software (James Whittaker, Jason Arbon, Jeff Carollo)
5 Likes

Hello!

I would like to just state here my opinion:
From what I see, I would divide Continuous testing in 4 pillars, but, before that, I want to say that continuous testing should be looked at as a mindset and not a framework! I see the use of continuous testing as a way to achieve a continuous quality state.
With that said, here it goes my four pillars explained:

  1. Code Quality
  • Ensure the code is built correctly and tools are scanning that code and providing feedback on how good, secure and performant it is.
  • Question to ask to understand if you are using this pillar: Did developers created the application code correctly?
  1. Pipeline automation
  • Ensure the good code that was built can flow across environments without manual intervention.
  • Question to ask to understand if you are using this pillar: Can the application code flow across environments without manual intervention?
  1. Application Quality
  • Everyone in the team must have a common understanding of what must be developed, tested and released to users.
  • Question to ask to understand if you are using this pillar: Did developers created the correct feature? or Does everyone in the team has a close view of how the product works?
  1. Customer experience
  • You should always be in the look for your costumer opinion! You can have the best app and with most beautiful code but if you don’t have users, that you need to change something
  • Question to ask to understand if you are using this pillar: Are users perceiving value in the application delivered?

If you need further help, contact me and we can discuss this!

Cheers

2 Likes

Try Tom Gilb and Evolutionary Delivery

So good to come across this post. I am also a bit confused by this new buzzword :slight_smile:

We already have the terms continuous integration, continuous delivery and continuous deployment. In each of them automated tests are included. So what’s new about contiuous testing? Is it just a new definition which only focuses on test activities of continuous integration, delivery and deployment? Or does it bring something new on the table?

P.S. As if that weren’t enough already, I have heard “continuous test integration” recently. Anyone has an idea what it is? It sounds made-up to me.

2 Likes

Do you mean relating to this? http://leansoftwareengineering.com/2007/12/20/tom-gilbs-evolutionary-delivery-a-great-improvement-over-its-successors/

Can you explain why? (To save people trying to figure out).

1 Like

Indeed I do, the book “Principles of Software Engineering Management” contains details of EVO (Evolutionary Delivery) which requires detailed planning but also requires software to be delivered with usable functionality at each stage allowing for both continuous development along with continuous testing.

I tried many years ago after spending time with Tom, to influence the company I worked for to introduce the idea of EVO with a view to break it down to one change, one test cycle, place on shelf ready for release. I failed as they were not ready for the practical introduction of such a novel process.

The issue in reality is the need for continuous development of automated tests as well (I am a believer that test automation of functionality can only be done successfully after the development has been tested and stable).

Perhaps have a look at - https://www.gilb.com/blog/how-well-does-the-agile-manifesto-align-with-principles-that-lead-to-success-in-product-development

2 Likes

The DORA State of DevOps report (see at https://cloudplatformonline.com/2018-state-of-devops.html) has a fairly good high-level explanation of what makes continuous testing different from other test approaches:

Continuous testing includes these practices, with some important additions:
• Continuously reviewing and improving test suites to better find defects
and keep complexity and cost under control
• Allowing testers to work alongside developers throughout the software development
and delivery process
• Performing manual test activities such as exploratory testing, usability testing,
and acceptance testing throughout the delivery process
• Having developers practice test-driven development by writing unit tests
before writing production code for all changes to the codebase
• Being able to get feedback from automated tests in less than ten minutes
both on local workstations and from a CI server

2 Likes

For me Continuous Testing is doing testing in all phases of development right from requirement phase to production. Here is the article detailing that. https://amtoya.com/2018/09/19/continuous-testing-for-delivering-continuous-value

Good Reads are:
Specification By Example - Gjoko Adzic
Testing in Production - Charity Majors

3 Likes

Hi Rosie,

There’s a lot of really good material that’s already been provided by others in this thread but I wanted to chime in with my thoughts and provide some material. Continuous testing came about as organizations are struggling to understand the level of risk they were introducing into their applications during the software development process. In the early days continuous testing represented a form of test automation that encompassed all of the different testing activities (i.e. static analysis, unit testing, API contract testing, and functional testing, regression testing, and even nonfunctional testing like security and performance) and running them in an automated way. The key though was not simple automation but the feedback process that Ashley was referring to.

Fast feedback enables meaningful actions to be taken as a part of the testing process. I’ve always use the term “policy gates”. The idea is that every night when I run my tests I will have certain functional violations that will trigger. But I set up a threshold for risk. An example would be only for severity one violations and up to three functional regression tasks would allow the application to be deployed into a staging environment. If those thresholds were not met then I would stop the deployment promotion process and deliver the relevant tasks back to the initial testers or developers there were identified as the authors of the code or test.

I actually think these principles involved from test automation to continuous testing as agile and DevOps became key initiatives. Agile (agile testing not agile development) helps us focus on new functions and features that development is introduced into the product so that we can get meaningful information back to our stakeholders about the level of risk were introducing and stakeholders can make a decision about prioritization of tasks prior to release. This ultimately helps development and product management make a more informed decision about where they’re going and actually scope things in a meaningful way. DevOps says we need to be able to get these artifacts deployed in a much faster way so that we can get the second type of feedback which is stakeholder and customer feedback. So let’s start building infrastructure as code so that we can deploy artifacts a an automated way. The problem with all of this was that without thoroughly validating the application as it sped its way down the DevOps delivery pipeline. We ran the risk of defects leaking into production.

I think that’s where continuous testing really hit its stride. Because it evolved from nightly based test execution to continuous test execution where the different types of test cases could be executed as a function of code check-in. In the earlier stages of development. There’s technologies out there that help us understand what the relevant tests are to execute and by aligning that test execution to the initial requirement we get immediate feedback into the impact of development code change on our overall application much much earlier than when the application is deployed.

Now I have to be upfront with you, I do work for a testing tools company so you’ll have to take my input with a grain of salt but I explored this topic in much greater detail in a blog I wrote call continuous testing for DevOps

I hope this helps

1 Like

Continuous testing refers to the process of testing an application precisely at every step. This kind of testing emphasizes testing early and often testing an application so that there is no loophole that can cause a glitch in the overall performance.

The conventional testing involved a lot of investment of time and manpower, which isn’t required in case of continuous software testing. The idea is to improvise a mechanism/process that can help in the precise testing of the application at every step. For all those that require adequate results without squandering their precious time; continuous testing is perhaps the best way out there!

This testing technique lends a hand in finding the risks that can be quite challenging to rectify when using traditional testing methodologies

1 Like