Nightly builds for ui automation


(Russell) #1

Hey all.

I’m currently working in an environment which is trying to introduce nightly build to runs of our ui automation against. Currently we run unit tests on merging to master but not UI automation. The aim is to run automation each night (takes 3-4 hours, about the same time as our unit tests :frowning: ).

UI automated and manual testing is ran in sprint targeted to area’s impacted, the aim I’m I’m told is to find things that have been caused merge issues and or things that are missed by squad targeted/full automation run testing.

I’m of the view that nightly build and automation is an anti pattern - common path but a better path exists.

My view is fix the underlying issues - eg why squads aren’t able to target effectively, why we find issues after merge issues etc

Doing it nightly means you have multiple teams changes at once and require a team/teams to rotate review of the results and finding the team to fix issues usually not linked to the changes.
+ve
Nightly would find things faster than you would waiting for a decision to release master
Reduces time people are building of broken software than if you do automation on release.
Flaky environment/tests failing more obvious

-ve
Always need resource to review failures - if no one looks they don’t provide value
Teams fighting over if it was them or another team
Lower priority to review failures
Flaky tests will waste a lot more time review else results/failures start to get less trusted - aka that one always fails
Teams are made up of humans if they know a nightly build is being run they are less likely to run wider testing prior to merge

Personally I think we should invest the time in solving the issues as to why targeted testing is so hard, eg tasing in automation, and at invest in getting automation to run on merge against tests that are assigned to area’s of code. eg change the method around sign up - automatically on merge kicks off tests on sign up etc.

Whats the community’s thoughts?


(Chris) #2

Moving to nightly builds is usually a True North adventure for companies looking to reduce their cycle time. Automatic check suites are designed to pseudoverify a bunch of facts. When this happens is not necessarily important, but having the facts to hand more frequently is good for testing and may force improvement of your autocheck suite codebase. Fixing why your squads aren’t targeting or why you find post-merge issues would be helped by more visibility. More frequent UI check runs give you more visibility (unless they’re awful). So your solution is their solution. Problem solved.

Always need resource to review failures - if no one looks they don’t provide value

You do always need a resource to review failures, but that’s checks for you. What you’re saying here is that you value access to the information less than the time it takes for a review. That’ll depend on the quality of your UI checks. If you need to have that information at a certain time to inform people that matter of things they want to know then you have two choices: review a check suite or attempt the checks yourself.

Teams fighting over if it was them or another team

If your teams are fighting then you have a whole other problem that management needs to get on top of.

Lower priority to review failures

Why? If the build can’t complete then that’s a big blocker. It has to be someone’s job to review these things.

Flaky tests will waste a lot more time review else results/failures start to get less trusted - aka that one always fails

Good. This pain will force something that few companies do enough of: maintaining the development project your company launched (you may call it your ui automation). When you feel that pain you’ll start to make it more efficient, and do some much needed pruning.

Teams are made up of humans if they know a nightly build is being run they are less likely to run wider testing prior to merge

I think I might know what you’re saying. I may go into it below.

Now if you were to ask me for arguments against nightly builds that are not continuous deployment (i.e. autocheck builds that run overnight but do not actually deploy) I’d probably bring up the stop/start issue. Testers may feel that testing an old build is pointless, so they begin a new build each day. They therefore may put their sessions into one day, or not start any testing at the end of the day. Frequent builds also mean that regression bugs are ever-present. There’s much less sense of certainty in what we know if we keep changing things. We may also need more systems to control semi-finished development and that can introduce its own problems. If you are turning off code in the build for in-progress development then surely you’ll find the vast majority of merge issues when you turn all the code on. Run UI checks for that, instead, and put it in the lap of the developing squad.

As for your closing ideas I think that nightly builds could help the process improvement projects you’re talking about. I don’t see why you can’t do both. A lot depends on your context, so it’s hard to say exactly, but in general if there’s more useful and reliable data gotten easily then I want it. If not, then we have to ask questions about cost-efficiency and pragmatism.


(Russell) #3

Thanks for your answer Chris sorry it took me a while to get to back to you. I think we’re on the same page kinda.

My post was saying the above are common issues with nightly testing of a build. I didn’t really explain my view on a better pattern just hinted at it in the closing statement.

I liken nightly builds to doing only ui checks and no unit/integration checks - This is better than doing no tests/checks so is good but it is an anti pattern for automation vs doing checks at unit, integration and ui level - the ice cream cone.

My issue is this pattern of nightly checks on a builds not of checks at the right place at the right time is an anti pattern as its a common approach but a better solution exists as you mentioned at the end and I hinted at. Test in the pipeline prior to merge and make team doing work responsible and make it a blocker if there are failures.