TestBash Spring 2023 - Tester Meets CI System - Tobias Geyer

[dianadromey]

During TestBash Spring, @the_qa_guy gave a talk on Tester Meets CI System.

We’ll use this Club thread to keep the conversation going, share resources and answer any questions we don’t get to during the live session.

[dianadromey]

Questions Answered Live

@ifs - How is manual or exploratory integrated in the CI and release process?

@goranmaksimovic - What tools and approaches you used to measure code coverage?

@lisa.crispin - Did the team run tests locally before committing changes? And if not, could you influence them to do that?

@jaimeellis - How often did something slip through because of only running a subset of the tests?

Anonymous - What kind of obstacles you faced when implementing this in your firm & your experiences about that

#(To Be Archived) Questions Not Answered Live

@fullsnacktester - Would you consider continuing to build your code skills and transition into a build engineer in the future?

@mirania - What format/forum did you find most useful in learning Gradle yourself? e.g. Pluralsight or Udemy, or Gradle support documentation itself?

@mirania - What format/forum did you find most useful in learning Gradle yourself? e.g. Pluralsight or Udemy, or Gradle support documentation itself?

@grebalexa - Could you please summarize one more time who is the owner of your CI now? Is it a devops or the entire team?

@pauladonegan - We do similar with Sonarqube, however devs most times think they have an exception and want us to override the block to go live. Has this also happened to you?

@tressaking - When you subdivided your tests into pre-commit, in-CI and post-CI what criteria did you use to move tests between the groups? was it just timing?

@tressaking - Where you didn’t already know who to talk to how did you solve this?

Resources Mentioned

• Slides: Tester meets CI System - lessons learned - Google Slides

[engineer (Robert)]

When developing gradle scripts in Jenkins I found I had to commit them to the repository. When you use Subversion all of those intermediate scripts are permanently kept. Did you find a way of experimenting without leaving a trail of intermediate gradle scripts in the repository?

[davidrutt (David)]

Fantastic talk Tobias! So very engaging, and some really pertinent content. Thanks!

[the_qa_guy (Tobias Geyer)]

@dianadromey

@fullsnacktester - Would you consider continuing to build your code skills and transition into a build engineer in the future?

I can’t let go of doing some build engineering work myself but I am a tester by heart and always will be. So I’m letting the build engineers we have now take over the work in this area and focus on testing again. For me it’s much more fun and fulfilling.

@dianadromey

@mirania - What format/forum did you find most useful in learning Gradle yourself? e.g. Pluralsight or Udemy, or Gradle support documentation itself?

I started off with a book, then used stack overflow and the gradle documentation to learn. Also getting reviews from others was such a great learning opportunity and helped me understand things a lot better.

@dianadromey

@grebalexa - Could you please summarize one more time who is the owner of your CI now? Is it a devops or the entire team?

The owner right now is a new team that I’m a part of called “Tooling Build and Test”. We’re taking care of the CI system, the auxiliary systems to that and many testing activities.
If teams need changes to the CI system they can either do them themselves and get a review from us (which doesn’t happen often) or they create a ticket with requirements for us to work on.

@dianadromey

@pauladonegan - We do similar with Sonarqube, however devs most times think they have an exception and want us to override the block to go live. Has this also happened to you?

We had this problem on the code coverage where people started gaming the system, so: yes, it’s happened. What helped to get out of this situation was to really try to understand why people did it (or in your case: want the exceptions). Then you can have a discussion and ideally come to an understanding.
In case of static code analysis our architects created a set of rules that must be obeyed, deployed it to the whole development team and collected feedback. That resulted in fine-tuning of the rules and fine-tuning of the expectations of the developers. For some (very few) things the architects simply put their foot down and the developers had to accept it.

@dianadromey

@tressaking - When you subdivided your tests into pre-commit, in-CI and post-CI what criteria did you use to move tests between the groups? was it just timing?

We started off with timing but then also had a look at what the tests do. If they didn’t cover crucial features they moved to a later stage, if they did they moved to an earlier stage.
This was also the case with test failures that got found in a later stage. Especially in the beginning we asked “should that have been found earlier” and moved the test if the answer was “yes”.

@dianadromey

@tressaking - Where you didn’t already know who to talk to how did you solve this?

I solved this by asking people who are at the company longer than me. Sometimes that meant following a few “you should ask” - “you should ask” chains.

[the_qa_guy (Tobias Geyer)]

@dianadromey

@goranmaksimovic - What tools and approaches you used to measure code coverage?

Bonus answer so that people can find it easier: We’re using Java, so we’re using JaCoCo to measure the code coverage

[the_qa_guy (Tobias Geyer)]

We made sure that we can run the gradle scripts on any local machine and on Jenkins and in 99% of the cases they produce the same output. That enables us to develop and test them locally and only commit them once they work.
For cases where that doesn’t work out we do use feature branches but the feedback loop via Jenkins is so slow and painful.