Ask Michaela a Question About: Hurdles of Testing Large-scale Software Systems

Our 5th talk during the TestBash Home conference is “Hurdles of Testing Large-scale Software Systems” with the brilliant @mgreiler.

I’m always fascinated when I start to think about how large systems integrate with each other, how the pieces work (or don’t work) with each other, how a code change in one place could affect so much more. It’s like all of the small projects I have experience with amplified!

Michaelas podcast https://www.software-engineering-unlocked.com/

We’ll be adding all unanswered questions from the talk here so if we didn’t get to your question, don’t worry, we will find you an answer :grin:

If you’ve got a question you’d like to ask in advance, why not ask it now?

Unanswered Questions

  • shabbir: As now a days large scale systems are built incrementally and supported and operated with a lot of baked in tests – Do you think the hurdles of testing large scale system still exist?
  • Elizabeth: Were there tests you decided to stop running while you were still compiling the data/asking questions, but before calculating the actual cost?
  • Zivan Miljkov: What do you do with new test suits that are created during the analysis time?
  • Madhuri: How often u go and check if tests are still relevant?
  • Dennis dB: Who much did the investigation into costs cost?
  • Paul Grossman the Dark Arts Wizard: Can you recommend best practices to manage source control merges with such a large team - I have nine team members practicing feature branch approach.
  • Nathan Owen: Tying in to the talk about deleting tests earlier, did you find any tests where the cost of skipping a test never exceeded the cost of running it? Have you looked at applying this to identifying tests to delete altogether?
  • Kishor Mohanty: That’s true that we can think about cost cutting, but what will happen to quality? How do we ensure that quality won’t impact while reducing/deleting some tests 

1 Like

How do you approximate bringing up a large scale system in a “test” environment (non-production) for testing? It’s not easy replicating “production” in a non-production environment for testing, in terms of scale.

My role was more as a consultant. So, I did not delete any tests. My role was really to inform the developers and testers about the effectiveness of their tests. Based on the effectiveness profiles teams did take measure and deleted/fixed/changed a lot of tests. This did not influence the analysis, as we calculated that for the time periods before.

In this approach, we did not reduce quality, because we ensured that all tests are run at least once. We might have delayed the detection of a defect. And we balanced whether we should be running more tests, or finding the defects later.

Yes, too both. Many tests were worth running, and during the analysis and talking with the teams, we discovered quite a bunch of tests that either have been improved or deleted.

We did not take those changes into consideration.

Madhuri: How often u go and check if tests are still relevant?

We tuned this as a parameter, based on the project.

shabbir: As now a days large scale systems are built incrementally and supported and operated with a lot of baked in tests – Do you think the hurdles of testing large scale system still exist?

I’m not sure I understand this question correctly. But especially for projects that tests are build up incrementally (so some are there for a long time already), I think flaky tests and non-performing tests become an issue.

Good point. We did not consider it, as it’s a one time cost that amortize over time. Given the large scale systems (and the amount) it’s a marginal cost. For smaller teams it’s definitely something to take into considerations!

I like the feature branch approach - it’s also very suited for code reviews, which I’m a big fan of. Some of Microsoft’s products also have a large engineering team behind and uses the feature branch approach.

I’m done answering all questions. I hope everything is clear. Happy to clarify further!