I’m currently working in an environment which is trying to introduce nightly build to runs of our ui automation against. Currently we run unit tests on merging to master but not UI automation. The aim is to run automation each night (takes 3-4 hours, about the same time as our unit tests ).
UI automated and manual testing is ran in sprint targeted to area’s impacted, the aim I’m I’m told is to find things that have been caused merge issues and or things that are missed by squad targeted/full automation run testing.
I’m of the view that nightly build and automation is an anti pattern - common path but a better path exists.
My view is fix the underlying issues - eg why squads aren’t able to target effectively, why we find issues after merge issues etc
Doing it nightly means you have multiple teams changes at once and require a team/teams to rotate review of the results and finding the team to fix issues usually not linked to the changes.
Nightly would find things faster than you would waiting for a decision to release master
Reduces time people are building of broken software than if you do automation on release.
Flaky environment/tests failing more obvious
Always need resource to review failures - if no one looks they don’t provide value
Teams fighting over if it was them or another team
Lower priority to review failures
Flaky tests will waste a lot more time review else results/failures start to get less trusted - aka that one always fails
Teams are made up of humans if they know a nightly build is being run they are less likely to run wider testing prior to merge
Personally I think we should invest the time in solving the issues as to why targeted testing is so hard, eg tasing in automation, and at invest in getting automation to run on merge against tests that are assigned to area’s of code. eg change the method around sign up - automatically on merge kicks off tests on sign up etc.
Whats the community’s thoughts?