How do you make test data considerations a natural part of your scenarios? Test data is usually a requirement for any scenario to be executable but it never seems to be an organic part of behaviour or logic models for writing tests.
Hello everyone!
Warmest regards from the rainy Budapest… 
Thank you for the good questions, I will try to answer them in this hour.
I am super excited, so let’s start…
2 Likes
For GUI automated tests I am using test cases that represent a small workflow of a functionality that needs to be tested.
How huge can be those GUI automated tests? How many steps should one test consist of?
Hi Gem,
I don’t have any very scientific tool. I think the key is to make the team engaged and interested. I usually use a technique called Example Mapping, described my Matt Wynne. That is a very lightweight meeting facilitation technique. The idea behind it is that it is easier to keep the team engaged if they “only” need to collect some rough examples on post it notes, in compare to let them write properly phrased scenarios. Once you collected these rough examples with the entire team (that is fun!), pair of people can turn them into scenarios with not so much effort. Whatever you do, you should facilitate the meeting in a way that everyone can contribute, not only the domain-expert super heroes.
I have a couple of post about Example mapping: http://gasparnagy.com/tag/example-mapping/ and also our book with Seb Rose: Discovery (http://bddbooks.com) focuses on that if you want to read more.
1 Like
I wish there would be a good book or collection of the test-related design patterns… Like the ones for the dev patterns.
I use a couple of patterns, but not all of them have an official name:
- The most important if the “driver pattern”, that is a separation of the automation aspect from the testing aspect. Pretty much the generalization of the page object pattern. Others call it adapters or glue code. That has the most positive impact on my automation codes. (See the article of David Leitner in the BDD Addict newsletter 2016 March: http://gasparnagy.com/2016/03/bdd-addict-newsletter-march-2016/)
- I also often use the object mother pattern (ObjectMother): separation of the test data or test data defaults from the other parts is very important.
- I also use a pattern, that I call the “ensure pattern” – to be able to manage implicit contexts
- I also typically separate the “current objects” that the scenarios “talk” about, like the current order, the current product, etc.
- I sometimes use the test id pattern, that is a given pseudo identifier for the objects that are hard to identify otherwise.
probably a few more, but these are the ones that come to my mind…
3 Likes
I think understanding that we cannot have prove that there are no bugs but we can only reduce the probability is already a good step. Many people still believe in the illusion of “full coverage”.
Regarding my experience, good understanding of the domain and the problem is a very good tool to reduce the probability of the bugs. So don’t underestimate the human aspect. (I posted recently a few tips: http://gasparnagy.com/2019/04/example-mapping-5-tips-to-avoid-forgotten-requirements/)
Once you have the good understanding, any tool that can exercise your code is useful: I think AI will be one of these soon, but today the entry barrier is probably too high, so it can only be applied for specific cases. But good unit tests with data variations, or property-based testing is already applicable “easily”.
Also DevOps and monitoring (together with a good continuous integration process) can be a super efficient tool.
2 Likes
The optimum level is a bit blurry, so try to reach “good enough”. Generally the primary question you need to ask is whether a particular piece of data is essential for understanding the test/scenario or not. If it not essential than it is probably better to push it down to the automation layer, where you can have much better tools to keep them separate and clean.
Make a few scenario and ask the team to review them together!
1 Like
Whenever I conduct a review, my primary goal or outcome is to find patterns. Patterns that are specific to our application or context and try to have a consensus across the team on how do we solve that.
Testing and test automation is complex, but very much pattern-based. Once you have a good collection of “how we usually address such problems” patterns the testing becomes much easier.
1 Like
Writing low-level click-click-fill-navigate style scenarios is a trap. The trick is, that it is pretty easy to build up a couple of super-reusable test steps and with those you can practically automate anything. So it is a quick win.
The problem starts when you will have 20, 50, 100, 2000 tests/scenarios like that. Suddenly you realize that they are very brittle and don’t bring much value. The value of the test is not that it can turn red or green. The value is that if it turns red, you can understand what the impact of that error for the application and get an idea where the problem might be.
Check out also my post: http://gasparnagy.com/2019/05/clean-up-bad-bdd-scenarios/
3 Likes
The primary purpose of BDD scenarios is to help the team understand the expected behaviour and to document it. The scenarios are also executable test cases, but you will probably need a larger set of executable test cases to get confidence about the quality (e.g. testing an edge case that was not necessary for understanding or testing multiple test points within the same equivalence group).
1 Like
When we write the scenarios using declarative style, we better focus on the intention. What we wanted to achieve, what context/precondition we expect or what benefits do wen want to observe.
The imperative style focuses on how those intentions can be achieved, so it is more about the mechanics.
Focusing on the intentions rather than the mechanics is generally better, because it will be easier to understand the test later and it also opens up multiple options for automating it. Consider this:
Given I have logged in
And I have registered a question with …
Here the intention was probably, that I wanted to have a question registered, because I want to check how it behaves if you for example check its details. If you just say
Given there is a question with …
than it is more brief and you can choose to automate it however it is the most efficient.
(In this post, there are a bit more details on this: http://gasparnagy.com/2019/05/clean-up-bad-bdd-scenarios/)
This is hard. I haven’t really found any very good metric.
I like to measure the time that the team needs to spend with bug-fixing and maintenance. I also like to ask the team to tell about their confidence in quality.
Coverage tools are more useful to spot out absolutely untested areas.
It sounds stupid, but the best is if you will leave it up to the team. With that they will feel better responsible for the overall quality and this will help them to find the optimum level.
Generally it is very hard to estimate the cost of automation of a new area where we don’t have automated tests yet. So I like to start with automating something easy and learn about the context to better judge how much should be automated.
If you do the implementation test-first, than the things are much easier. You can automate nearly everything in the most of the cases.
2 Likes
I think reusability, especially at the test step level is a bit overrated. The automation code has to be clean and production-quality, so there the DRY principle should be definitely applied. But for the definition of the tests (in BDD the scenarios), having a bit of redundancy - if it helps to focus on the intention - is beneficial.
So I try to keep incidental details, especially data out of my scenarios and apply meaningful defaults in the automation layer instead.
The Example Mapping technique and any other structured conversation technique that is used to discover requirement details is noting else than applying the testing principles for requirement analysis. This is essentially what some people call “shift left” in agile testing.
Exploratory testing helps you to see the product from a different view point and to find heuristics about how you can discover more of it. This practice is absolutely useful for Example Mapping as well, so I am pretty sure that as an exploratory testing practitioner, you will find better examples or examples that we would have otherwise forgot.
I would not explicitly use BDD for any of these… BDD is for understanding and documenting the problem. It’s another thing that if you make good BDD scenarios, you will “automatically” get a lot of good integration, regression and UAT tests as well.
There are 8 more questions and the power hour ends… but I will keep answering them! 
2 Likes
I don’t have much experience with Accessibility testing, so I cannot really comment on that.
Accessibility is a non-functional requirement. BDD does not directly target non-functional requirements, so usually people use other testing tools for these. (That can work well together with the functional BDD scenarios of course.)
Thank you! Great responses so far
The problem is that for manual test cases, the most expensive resource is the human who performs it. So the entire process should be optimized for getting the most out of the human efforts… Because of that, manual test cases are typically covering a longer user journey and contain a lot of verification (if you are anyway there, check is and that as well).
For automated tests the most costly part is the implementation and the maintenance. Because of that it is better to make small and focused tests, to reduce the side-effects.
Because of this difference it is typically not a good practice to automate a manual test, but rather build them up from scratch.