How to handle overlapping tests

Hello,

I have a two different test that overlap. Let me share the given-when-then statements

test_order_placement.py

# GIVEN The user is on Red pepper Pizza

# WHEN The user clicks on the Add to order button below “Opera”

# AND The user clicks on the Add to order button below “Frutti di Mare”

# THEN Opera and Frutti di Mare appears as order items in Your order

test_order_change.py

# GIVEN The user is on Red pepper Pizza

# WHEN The user adds a “Blue and White” pizza to an order

# AND The user clicks the Delete button next to the order item

# THEN the entire order is removed form Your order (both order items and order total )

Look how how test_order_change relies on the when-step to run. The “when” step in test_order_change is pretty much the same as the “then” step of test_order_placement.

Lets 'say that the first test fails because for some reason pizzas can’t be added to an order. Next the yet I replicate the actions from the first test and then expand on the user behavior to delete an order item. If I only test for the absence of an order how will I know the order was present in the first place?

What is a wise way to address this overlap? So far I thought of two options.

  • Use pytest -x flag to stop all tests after the first failure.
  • Only keep test_order_change and assert both order placement and order deletion in the same test (I am aware this is bad practice)

Happy to hear your thoughts!

2 Likes

When I am developing automated tests, I do lean heavily on the Heuristic / mnemonic SACRED, as defined by @friendlytester …

Here is (one of) his presentations on this:

For my own less-brilliant-than-his analysis, I will examine your example with regards to only a couple of the letters.

First, the state. (S) For your tests, I will assume that you have the environment under control, and you know the different states you have to work with. Starting with “GIVEN” means that you can get to that state reliably every time. In this case, the state is “Red Pepper Pizza.”

The first part of your “WHEN” statements in both tests is ALSO defining a state. That is, “User is on Red Pepper Pizza AND an order has been selected.” This is the same in both tests, though the order is different. If you are not running tests in parallel (I usually don’t), you may want to test this state in a different test before running either of these tests. (GIVEN user is on RPP, WHEN the user adds something, THEN the selection is visible).

That is, test the previous Action (A) before testing multiple actions in a list of actions. In this way, if the first action causes a problem, then you know for sure that this is a problem, and you have less investigation to do.

Now you have some code you can share with the two tests you have already defined. “GIVEN an order has been selected.”

It’s then you may put the Action-you-want-to-test into play. So in a test, you are only testing “click delete” or “Click add again”.

So in your test, you have defined a state and an action, so then you have to codify your oracles ( C).
In this case, it could be as simple as two THEN statements, THEN Opera appears in your order AND Frutti di Mare appears in your order. If there are other conditions, you may add them too (i.e. there might be a subtotal or a price or an amount or …) Since the (new) state is defined, you may have some more complicated oracles.

You may want to go through (at least) the link I gave to come up with your own ideas.

I wouldn’t stop tests because one failed, since there are other tests which may pass later in the test suite. In other words, you want to get as much information from running tests as you can get.

I wouldn’t add different types of assertions in one test (i.e. assertions related to different actions), since if this failed, it might not give a very good indication of why it failed. Or to put it more bluntly, Keep It Simple. (Or as simple as you can get in a complex system)

2 Likes

Hello @brian_seg ,

Thank you for the thoughtful response. I followed the link you gave me and after that continued with more material from om Richard Bradshaw: He has more to say in his talk Your Tests Aren’t Flaky, You Are where he expands on Sacred.

The Sacred concept does come in handy with defining the problem I see with the two overlapping tests.

. I use a conftest.py file with a pytest fixture to just open the webpage in full screen. This state is set before each test.

For test_order_placement the conftest.py works just fine. It set the state I need.

As for test_order_change I need a more deliberate state: The webpage open with an existing order item. What I have done in my code so far is just to copy the first part of the code in test_order_placement to get to that state; I use selenium to locate a menu item and use .click() to add that menu item to to the order.

Ideally I would find another way to set the browser to that state that doesn’t copy the WHEN step from test_order_placement but I don’t know how. With selenium I am doing black box testing. All it does is simulate user behavior. I did some googling just now and I realized I can use webdriver to execute JavaScript. Does that sound like a reliable way to set the state for test_order_change or am I only introducing more flakiness you think?

1 Like

Yes, that seems a lot more reliable to set certain states.

I’ll just put in a link here, because Angie Jones is a much better explainer than I am… :slight_smile:

https://testautomationu.applitools.com/setting-a-foundation-for-successful-test-automation/

I can’t remember where she talks about seams, I think it’s chapters 3 or 4. But it is worth the entire watch / listen.

1 Like

I will have to practise my long forgotten JavaScript skills then. I will also go through Angie’s course and pay attention to the seams concept.

Thank you very much for helping me think clearer about testing.

2 Likes