How to ignore some tests in CI with GitHub Actions?

Hi,

We have created some automated tests with Selenium and Cucumber, implemented a test runner for local running and now we would implement tests on CI (with Github Actions) to be triggered on every PR.

We have many tests so it takes about 30 minutes to execute all of them, which is clearly too much time in CI.

How can we run just some tests in order to have a quick feedback ?

On internet I have found some suggestions about tags, but I haven’t found how to clarify in the yml file to run only tests with this particular tag.

2 Likes

Assuming you’re running Cucumber from Maven, you can specify that you only want specific tags run by setting the cucumber.filter.tags system property.

You could either do this by setting the CUCUMBER_FILTER_TAGS environment variable (Github actions docs for setting env vars here) or by passing the property to Maven with the -D flag.

For ease of understanding I recommend the latter, because it’s immediately apparent that you’re asking Maven to only run some tests. If you set an environment variable, you might end up with staff confused why certain tests aren’t running.

Example
Say you only want to run the “Smoke” test. You’ve tagged a bunch of tests with @smoke. Now, in your github actions YML file:

      - name: Build with Maven
        run: mvn test -Dcucumber.filter.tags="@smoke"

But I Have Concerns
If you’re not running all your tests in CI, when are you running them? Is your intention to run some fast-feedback tests so the committer can get on with work, and then continue running the entire suite? Do you run them daily, or hourly, or only before release?

I’d suggest, if possible, trying to run multiple jobs in parallel, using the Matrix build strategy. Here’s how I’d do it:

  1. Decide how many jobs you’re going to run in parallel. It looks like free accounts can run up to 20 jobs in parallel.

  2. Randomly assign your Cucumber tests to parallel groups. Tag them with the group ID (eg, @group_1?)

  3. Edit my Github Actions yml file:

strategy:
  matrix:
    group: [1, 2, 3]
steps:
  # Cutting out details here
      # Set the group ID environment variable:
      node-version: ${{ matrix.node }}
      - name: Build with Maven
        run: mvn test -Dcucumber.filter.tags="${{ matrix.group }}"

This isn’t tested BTW! I’m just trying to show; You define a matrix of groups, then use the group ID in your Maven tags. The only downside here is that, because some tests run longer than others, you might end up with some test tags taking much longer. You could fix this by re-assigning to different groups every now and then. You also have to be vigilant to make sure that tests get added to groups.

2 Likes

Thank you very much @dylanlacey !!!

Just to make sure that I understood, when you do the following thing in the YML file :smiley:
run: mvn test -Dcucumber.filter.tags="@smoke"
→ it means that tests with the tag @smoke in the class won’t be run, right ?

If it works like this it would be great !

Otherwise I found this solution too, it sounds relevant as well :
https://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html
What do you think ?

About your concerns, I think that in CI we have to run the most important tests in order to have a quick feedback on regressions. Then, during the sprint we can run all the tests (for exemple every night). About parallel tests are quite interesting and I totally agree with you but I think that we will do it later. For now the highest priority is to set up the CI and run with few tests.

1 Like

Filtering with Cucumber is a bit weird; When you do

run: mvn test -Dcucumber.filter.tags="@smoke"

It will only run the tests tagged smoked. If you don’t specify a filter, it runs everything.

(Oh, and you can specify multiple filters at once).

I’m not too familiar with how Surefire interacts with Cucumber, but I’d say try that solution… That way, no-one has to manage which tests are tagged or not; It’ll be a bit more straightforward.

2 Likes