Capturing what has been automated

Morning,

To those who have automated tests as a core part of their strategy and who still have ā€œmanualā€ regression testing also, I have a question. (perhaps the second point is irrelevant)

Do you capture/document anywhere what the automated test pack actually consists of?

If someone wanted to know what was in the pack, how would/do they know?

3 Likes

I noted it once during development in a dedicated document (which I had to update) and nobody cared.

In no particular order, everything can be combined:
asking the automation developers (me), reading the generated reports, reading the code comments, reading the code.

The reports of the automation are one place which comes typically out-of-the-box. I like my automation to have reports that contain some prose text about the content of the cases.
But be aware of that reports and all written text, even code comments, are always a simplification. The last level of detail and the only true source is the code.
And you have to keep it up-to-date if you change the code.

You may answer this for your own and find an solution which fits your needs:
For whom do you need it?
For what do these people need this information? Just a brief overview? Coverage of different features?
How often will it be needed?

Even as main developer I donā€™t memorize all details, but I know where to look.
At first people should ask me and then we find out together. Sometimes matters also the details of the code.

4 Likes

Thanks Sebastian, great reply there.

I think overall, what I need here is perhaps for each of the features, what are the regression test scenarios/exploratory sessions and then a marker against which ones have been automated. For awareness I want to have a list of the regression coverage, then what are we doing ā€œmanuallyā€ and what is covered by the automation.

If you just have what you do manually then thereā€™s no visibility of what is automated. Who wants this? Perhaps anyone in the team, if we want to periodically review it.

I suppose the main point of my question is DO people document the automation coverage anywhere (outside of the automated tests themselves)?

3 Likes

I donā€™t get you. What do you mean?

Another thing is to have a test case management system which contains manual test cases as well as automated ones (maybe the automated ones where ones manual?). While the status of manual cases are set by people, the ones for automation can be set by imports from the automation.

By BDD and Cucumber some teams go the other way: They start with describing scenarios and then automate them.
I havenā€™t done this so far, so I cannot tell you much about this. I donā€™t know where the scenarios are stored.

1 Like

We did exactly this where I work. I use https://qase.io/. I created two projects, Manual Regression and Automated Tests. I created all the test cases in Manual Regression and then cloned (you can do this with Qase) all the test cases over into the Automated Tests project.

As tests were automated I marked them as automated in both projects. In the Manual Regression project, I created a section called ā€œDisregardā€ and would move the tests into that section as they were automated. The team could watch the automated tests grow and the Manual Regression shrink.

We stopped manually testing every feature recently and now only test blast radius which is way more useful when it comes to testing.

Is what Iā€™m describing more of what you are after? Iā€™m happy to answer any further questions.

2 Likes

The automated pack should consist of tests that can run without any intervention or change. They shouldnā€™t be ones that break frequently or are unstable. Just having automated scripts doesnā€™t make tests automated. This should be in scope of your test strategy to understand what you cover from manual execution perspective and what you execute in the automated pack. Automated packs need to be in a Jenkins Job and this needs to be run at every build.
You also need to make sure stakeholders are aware that the purpose of automated packs are not to replace all tests and testers. They are there to ensure your build hasnā€™t broken existing functionality before you start your execution.

2 Likes

Great reply thank you. Very similar to what I do at the moment.

In terms of a lifecycle, after a feature is completed, we determine the tests/sessions that we feel should remain as part of the regression testing and these are entered into the test case management tool. They are tagged as automation candidates where necessary, and once automated then they are re-tagged, and changed to P4 (in our approach P4 tests are never executed manually as part of the reg test, just for awareness). Sometimes if a test/scenario is only partly automated then we need to split out to we differentiate.

If you have exploratory sessions as your approach to reg testing, then itā€™s not as easy to say ā€˜this is coveredā€™ with the automation (in automation, only specific test conditions are covered).

But I suppose the main thing I was hoping to find out is if documenting the automated test coverage (outside of the code itself) is something people generally do.

2 Likes

I like documenting it because itā€™s something I can report on. When you set an automated status on a test case in Qase, it tracks and reports can be pulled up and shared with the team (with a paid plan). I appreciated this because when I joined our team, we had 7 automated tests, 6 of which were broken. Writing the test cases in Qase, tracking them, and sharing reports with the team was a big boost. Everyone is far more confident with our releases because of our test coverage. So, in my opinion, itā€™s helpful, but it depends on the needs of the team.

2 Likes