Sustainable Test Automation: From Garden to Ecosystem with Anaïs van Asselt

For our second talk of TestBash Home 2021, @anaisvanasselt takes to the stage to talk to us about how we can keep our gardens of automated tests green :seedling:

We’ll use this Club thread to share resources mentioned during the talk and answer any questions we don’t get to during the live session.

3 Likes

Questions we didn’t get to

  1. Steven Devonport > Do you find customers are concerned with the question ‘why should we automate our existing tests’?

  2. Sailaja Tata > Agile disagrees with time ? how do you convince your team about this time?

  3. Tehreem Fatima > How do you prioritize the tests to automate first? and how to select tools that add sustainability to the project?

  4. Stacey Armour > Should the development language/tools affect the choice of testing language/tools?

  5. Matthew Churcher > Are you testing in production! It’s super tough, got any tips?

  6. Paul Naranja > Do you have a strategy to properly scope a test that should be Unit/Integration/E2E ?

  7. Matthew Churcher > Where do you stand on the overlap between test layers. do you cover the same functionally in both API (E2E) and Unit/Integration tests? Also also, we are moving to contract testing! Contract-testing is the bee’s-knees. Any thoughts there?

  8. Kamil Brdys > Most testers write UI automation, do you also write integration and unit tests ?

  9. Armando Cifuentes González > If the customer is reluctant to change their automation framework, although you have explained all the improvements that it entails, what would you recommend?

  10. Atiqah Azlan> How do you balance between fixing the current flaky test and adding new featuresto ensure test automation always updated

  11. Becca Batchelor > At what point do you make the decision that a test is too flaky to automate, and that it is better to be manually tested?

  12. Del Dewar
    Have you tried Kent C. Dodd’s testing-library when dealing with GUI automation to move away from the traditional automation design patterns?

  13. Penny Howard > Sooo, do you garden?

  14. Jeff Johnson > How would you go about talking to your team about exploring different testing tools when the team appears to be set with a particular tool?

  15. Kashyap Mehta > Did RF work like a magic or is it also another same sort of tool that will require lots of maintenance over the period of time when regression suite builds up?

  16. Jay Chandrasekaran > Was the POCs done for one project and then used to showcase the benefits or a tiny piece of each project was identified to give the big picture !!

Resources mentioned

Enhanced Test Pyramid

Article about an automation wheel if you’d like an alternative to the pyramid An Introduction To The Automation Test Wheel | MoT

Jit’s talk with the hexagon - Mobile Test Automation at the BBC: Then, Now and Next - Jit | MoT

Robot Framework
https://robotframework.org/

1 Like

Thanks for all of your interesting questions! Hereby my first part of answers; from my point of view, based on my own experience.

1. Steven Devonport > Do you find customers are concerned with the question ‘why should we automate our existing tests’?
Definitely, and a healthy concern if you’d ask me! I believe we need to keep asking the question why should we automate a test, and if so, in what layer (unit/integration/UI). For example, if the concerning manual test is an exotic scenario that only needs to be executed once or twice, the effort to automate and maintain such a test probably wont weigh up the added value. Always think about the costs vs added value.

2. Sailaja Tata > Agile disagrees with time ? how do you convince your team about this time?
I’m afraid I don’t understand your question. What do you mean by ‘disagrees with time’, what time? The short agile iterations and that you have to create/maintain your automated tests in that time, maybe?

3. Tehreem Fatima > How do you prioritize the tests to automate first? and how to select tools that add sustainability to the project?
If there is an application without any automated tests, I start with a classical product risk analysis; i ask the business what functionality/user flows are most important and the developers where there greatest complexity is. I determine the risk and start with the high risk functionality. Unfortunately there are no tools that add sustainability to the project. But if you need a tool to automate tests, look at your context to determine the best suitable tool;

  • who will be working with the tool? Testers/developers? What are their skills?
  • what is the technology stack?
  • do you need to integrate with ci/cd? Etc
  • discuss it with your colleagues, also developers, it helps!
    The more people support the choice of tooling, the more chance you have to succeed!

4. Stacey Armour > Should the development language/tools affect the choice of testing language/tools?
In an ideal world, yes. If your automated test can ‘live’ in the same project as your application code, it will bring a lot of benefits > it will grow with the same version of the app, and it can be maintained easier. Another great benefit is that developers can and will contribute as well, since they already have experience in that context and it lives right next to the app code. The exception is the scenario I talked about during my talk; we needed a tool in a context with a lot of different technology stacks (java, .net, package software), and it had to be accessible to functional testers. We chose to work with Robot Framework, because it’s a keyword driven tool very easy to use and suitable for multiple contexts.

5. Matthew Churcher > Are you testing in production! It’s super tough, got any tips?
Unfortunately not, but can imagine it’s super tough! I’m still in the learning phase, and hope to apply it in the near future.

6. Paul Naranja > Do you have a strategy to properly scope a test that should be Unit/Integration/E2E ?
Definitely! With the whole team (!) I often discuss during the kickoff of a story what the test scenarios are, what to automate and where. Ideally you don’t want to have doubles/gaps in your coverage, you want the layers to fit together very well. I follow these guidelines:

  • Unit test > covers technical requirements, think about specific business rules, if/else statements, data validation, boundary values
  • Integration/API test > covers functional requirements, think about different kinds of input/output, also alternative scenarios (bad requets, not found etc.), simulate external services
  • E2E backend > covers connectivity/integration requirements, use real instances of services that you mocked with the integration test
  • UI tests > covers UI specific behavior, that user interaction triggers the right events, browser specific stuff, responsiveness, not more than that!

7. Matthew Churcher > Where do you stand on the overlap between test layers. do you cover the same functionally in both API (E2E) and Unit/Integration tests? Also also, we are moving to contract testing! Contract-testing is the bee’s-knees. Any thoughts there?
Check out my answer of question 6. I believe it’s not possible to completely separate the testing layers when it comes to the application code that is ‘touched’ by the test. The nuance here lies in the assertions you do. An example; if you already asserted the data validation of event A in a unit test, you won’t assert all those validations again with an integration test, but there you focus that this event A is triggered during the right functional flow.

And yes, contract testing, definitely the bee’s knees haha! I don’t have any experience with it, but heard good stories. My test colleague is doing a POC with contract testing to replace our E2E backend flows. Unfortunately no tips, I hope to learn more about it in the near future!

8. Kamil Brdys > Most testers write UI automation, do you also write integration and unit tests ?
Yes, definitely! In order to fit the test automation layers together very well, I found it very useful to take an interest for these tests. This interest turned into hands on experience. By pairing with developers, I learned a lot about these tests and programming. It also helps to get more feeling about the application, where the complexity lies, so you can better advise what and where to test.

9. Armando Cifuentes González > If the customer is reluctant to change their automation framework, although you have explained all the improvements that it entails, what would you recommend?
I experienced it as a test automation coach. For months I spoke with managers and presented fancy slide decks in the hope to convince them to make test automation a priority and improve their framework. Then I visited the teams themselves with the ‘walk in the park’ sessions I mentioned during my talk. The teams were part of the improvement process, and their enthusiasm reached the management so much faster than a slide deck :slight_smile: You do your best stating the speed, added value vs costs and risk. However, at the end of the day you are the employee that gives advice regarding test (automation). If your ‘customer’ is still reluctant, so be it, and try to make the best out of it :slight_smile:

10. Atiqah Azlan> How do you balance between fixing the current flaky test and adding new features to ensure test automation always updated
If you find a flaky test during a story of adding a new feature, you ideally immediately fix the test. However, youmay come to a point that the progress of that feature get’s blocked by that flaky test. Discuss it with the team; you might timebox the test fix, and if it’s still not fixed, create a task/story to fix the test. Make sure it get’s priority though! If there is a greater number of flaky tests, you can create some technical debt stories to fix it with the team. Also in that case it is important that your environment understand the importance and that the testset adds less value when it remains in that situation.

11. Becca Batchelor > At what point do you make the decision that a test is too flaky to automate, and that it is better to be manually tested?
If you analyze a flaky test, you weigh the added value vs the costs to maintain the test. If the costs are too high, you might decide to ‘weed’ the test. It doesn’t mean you have to leave everything to a manual test though. You might decide to add unit/integration tests to cover parts of that functionality, in order to reduce the effort of manual testing. I often have discussions with developers to translate the ‘functional’ into the ‘technical’ to help determine what is needed.

Del Dewar. Have you tried Kent C. Dodd’s testing-library when dealing with GUI automation to move away from the traditional automation design patterns?
I know the ‘Test Automation Trophy’ of Kent C. Dodd (I wrote a blog about this 'Test Automation Shape and other shapes On Adventure Through Shapeshifting Test Automation Strategies | by Anaïs van Asselt | deTesters | Medium), however I’m not aware of his testing-library. If you have more info, i’m definitely interested! We replaced a lot of GUI tests with Angular unit/integration tests. Angular provides a so called ‘TestBed’ that enables testing in an isolated Angular context where you can cover a lot UI specific behavior. There are a lot of other strategies and methods available to push down tests, sometimes dependent on the context you work in.

Stay tuned for the rest of the answers!
Kind regards, Anais