When do you give up on a test script?

For context, this is very similar to my mood right now: https://twitter.com/BruceOnlyBruce/status/1572508414596771842?s=20&t=ti0Jq6tFc_sAHBl5QRk_kA

The big difference is that just one app is causing environment pain when it comes to testing it. All kinds of problems, ranging from being sure to be correctly configuring the app to connect to the correct test cloud environment (we have 3 or more really). All the way to the security hurdles the app has, have caused a 3 month project to take 6 months and still not finish. My automation is just flakey.
The goal of the script was to let developers test from home instead of just letting them wait for nightly results. Itā€™s much easier to then debug or rerun as the tests take ages to complete anyway, all sorts of reasons for that too. So we want devs to be able to test at home. The build and test lab is not in the office anymore, everyone uses a datacenter these days, so asking devs to come to the office was never going to help us here. A lot of the work delays have been that we need to do a lot of manual testing that the test rig just cannot execute right now because it requires automating things that are not possible, so we need some manual testing, but the flakiness of the auto tests has been a distraction which we are solving by re-writing some parts of the automation. The shift away from a nice office with everything in one place has been a big knock. Tech debt clearly plays a part. But each thing we fix , shows another thing that needs attention.

Iā€™m the kind of person who often uses pure enthusiasm to keep me chipping away at a thing, I donā€™t give up easy. But right now, just want to stab it with a wooden stake and ask to switch to a different team, or just rewrite the plan a bit. Why shouldnā€™t I?

2 Likes

Take off and nuke it from orbit?

Seriously, though, why are the automated scripts acting up?

2 Likes

I get like this often, in my experience, walk away from it for a few days, help out other team members even if its not your primary job.

Come back with a clean mind and more often than not (for me anyways) it seems so clear when i return, sometimes you dig so far down a hole with persistence that often you are blind to the obvious that you would of found without the frustration.

Ps. Hang in there!

5 Likes

The main blocker has been networking and VPN related. Iā€™m not keen to go into technical detail, but network troubles in our satellite office has also played a part by just being another hurdle I did not need @jon_thompson . Mostly caused by Covid remote working, one project overshadowing another and like @medigoldhealth points out, not stepping back and taking stock.

Had a war room, and we decided on some hard deals, we will cut back the scope and environments we want to support, which reduces the vendors we will execute automated tests against to just the main vendors. Trying to make sure tests ran against all vendors was probably a mistake with the small team we have. Itā€™s easy to aim big and then fail to deliver anything really as big as you thought you could deliver. So aiming a bit lower is humbling, but feels good. Basically the devs all came back and said that some tests against some vendors appeared to be stable, and keeping those as automated will loose us some coverage, but save a lot of lost sleep. And if we only run manual tests but focus on the remainder of difficult vendors we struggled to automate, thatā€™s probably a good chance to win back time too.

Iā€™m of a mind to do this kind of ā€œstep-backā€ exercise on a regular basis.

1 Like

I think I know what you mean about networking problems @conrad.connected Iā€™m developing automated tests remotely in the North West, connecting to the corporate network over a VPN, and then hopping through to a VMā€™s desktop in London running the actual web app Iā€™m testing. Pretty much every other statement is to make sure Iā€™m syncing with the UI before moving on, because if I donā€™t, the tests will randomly break because of random things not being fast enough.

Also, the devs rebuild several times per day. The app is built with Angular. Angular seems to randomly change static parts of the interface with each build just because it can. So, finding robust locators for each and every element that are stable regardless of Angularā€™s quirks is basically my life at the moment.

Still, if we knew what we were doing, weā€™d only be bored, right?

4 Likes

100%
I think thatā€™s the big thing that keeps me moving forward. Iā€™m suspect we all suffer the imposter syndrome, although I prefer to call it low self esteem. And Iā€™m then reminded of my fatherā€™s proverb:

If it was easy, they would get monkeys to do it.

Ah Angular, did a bit of that a year ago. It needs a lot of test-code organization and structuring skills, since lots of controls are just plain tricky to drive well. Reminding devs to add accessibility IDā€™s every time they make any changes is fun. I overall suspect that when the problem is complex, thatā€™s when we enjoy it most, but also when we need to give ourselves more room to improve our tooling, and clean things up more regularly.

1 Like

@conrad.connected - Each week i have that at least once. So you are not alone my friend. If it helps in any way know that, in my eyes you are a good tester and : You can do it.

3 Likes

When network (latency and/or bandwidth?) is the main issue:
How about uploading the automation application either to a PC next to the mobile devices or on the mobile decides itself? I one had the latter (totally individually developed by a co-worker because for Windows Mobile didnā€™t something like this exists)
And basically just getting reports, logs, etc. back once a run is finished?
Make the execution really running next to the devices or on them.

Another, more general, questions:
What are your trying to achieve by your (UI?) automation?
Are there maybe others ways?
E.g. you want to check that the app can be executed on the device. Why is it necessary to do this via UI automation? Why not using APIs on the mobile device?
Are you by UI automation also checking that the UI is correctly rendered? How about letting the automation ā€œjustā€ navigating through the app and making screenshots which your team (at least at the first time) checks by themselves to be correct?
You can do image comparison, but donā€™t get started with thresholds for deviations - imo make image comparison 1:1 or not at all. And be sure you comparisons screenshots donā€™t contain false-postives.

Imo there will be always parts which the humans need to do, they are better at improvisation and seeing the whole screen (not just tiny parts like automation code).
Think about how automation, and in general coding, could help your team to speed up the repetitive parts which are computers are good for.
And let the demanding things for humans. As handling a dynamically changing structure.

There is not such a thing as ā€œmanual testingā€ :wink:
I partly know what you are trying to say.
There is always a certain part left for humans. Guess what reading automation reports and logs isšŸ˜‰
Otherwise it wonā€™t be testing - looking for problems which matter to people.

:raised_hands:

2 Likes

@jon_thompson we use Angular a lot in our UI, and the way weā€™ve got round the problem for our UI automation of it dynamically rebuilding and renaming stuff is to put our own custom tags inside elements in the page HTML, and then use these as locators for the automation. As long as everyone in the dev teams sticks with some set conventions for naming these tags, it gets rid of a lot of the locator pain, like having to use xpathsā€¦

3 Likes

What do you consider not possible to automate?

1 Like

I once published a story of something I failed to automate. I had made 2 mistakes. I assumed that the proxy hardware rig or test-platform was going to mimic actual production systems, and I thought that managing to reset the system back to a know state by wiping the storage would be easy. As I started worked on each problem individually I lost sight of how much more work there still was. I had about 12 test cases, and one or 2 of them had eaten up most of the resourced time already. I donā€™t blog much, but I found where I wrote about it here Test automation failure

Being able to step back and realize, that for just a single test scenario or case, if you have spent too much time, you need to abandon ship smartly.

1 Like

I started getting my devs to do something pretty much like this on a Angular project which we eventually abandoned. I really miss that project, because as a technique it was kinda working. For me that project also felt impossible to test, but once we got going it got a lot easier. I love it when someone validates a strategy we used. Nice to get your very insightful inputs and encouragements @professorwoozle .

1 Like

In part a lot of this Sebastian, yes all technical issues with the IT systems, the environment, and a lot of history. I started this thread off mainly to get other people to share about how sometimes a project does struggle on technical hurdles, and I love how people have also responded with how they have had to practically solve those hard nuts to crack. Mainly Iā€™m impressed with how technical we all really are here in our solutions, but also how just talking about a solution that looked similar to one someone else used before too, grows the confidence that you made a good choice.

So sorry it took a while to get back, I went on holiday and then got back and was a bit ill. While I was away, my team bit the bullet and dropped the tests into the Jenkins CI job again, and got a few of them into a passing stable state. 6 months to get this far. Still a lot to do, but I probably need to go on holiday more often. Stepping back is good.

But at a technical level a lot of frustration comes out. Specific little thing like the way security always impacts test scripts is a common hurdle to any automation. On mobile devices itā€™s extreme to the max of hurdle heights for apps that touch the entire OS from security, screensharing/casting to networking, itā€™s not trivial anymore. Especially since the fruit-based phones are just a closed shop, testing OS interactions that the vendor chose not to allow you to automate is frustrating. Then the non-fruit phones where the vendor chose to change the security profile completely every 2 years, itā€™s so much fun when your test code breaks every few years. I donā€™t want to scare people away from test automation. But I do want to warn people that itā€™s not easy, all you have to do is visit the selenium and appium forums. A lot of people think automating is easy.

Manual Testing is not a ā€œthingā€ I agree Seb. But for me itā€™s been the fallback, and Iā€™m glad my team have stuck with me and we finally have something that is resembling an automation system running again.

3 Likes

I know this is an old post, but maybe this will be helpful:

Imposter syndrome: Iā€™m a dev and I sometimes feel this way too (Itā€™s not just testers). I think everyone feels this way at times.

Working through big problems, I feel like thereā€™s two big things that help me:

  • Rubber ducking: I try to explain my problem and intended solution to anyone who will listen (even if they donā€™t understand whatā€™s going on). Often by just saying it aloud it will become clear what I need to do
  • Small wins: I try to split the problem the smallest sub-problems possible and try to solve those first. I start with one that Iā€™m confident I can ā€œwinā€ to build a snowball effect. I sometimes think of it like this: how do you eat an elephant? one bite at a time.

Even when things seem large and looming, there is usually a way to work through it.

Sounds like you were able to reduce scope and find a way forward. Iā€™m glad to hear that.

2 Likes

Thanks for the reminder @natebosscher . Iā€™m going to share the emotional and the technical solution too. Iā€™m definitely a fan of rubber ducking and small wins, and thatā€™s how I got through this one too.
Technical: A few things meant that the network tunneling tool we used did not work reliably on macOS. This caused a lot of doubt for the tool choice, turns out it was because DNS resolution on mac is just simplistic and works poorly with VPNs.

  1. reboot your iPhone
  2. connect to VPN, and enable internet sharing to your phone again
  3. check that phone is now on VPN. if not reboot the mac and try again

Android testers merely have to install/copy latest github of gnirehtet (Also, the RUST binaries worked better than Java here.) Also they need to use the -d < DNS-server > parameter and point to an authoritative DNS on the VPN. Omitting that parameter and relying on defaults or local DNS (as noted DNS on mac was problematic, as was windows to be fair) was causing people huge frustration.

Emotional: After we got this working well, we just dived right into the next project and took no time to congratulate ourselves on the success. Iā€™m older and wiser now and have started to make a big thing about it more publicly when my team does a great job. Seeing another team win encourages everyone to start celebrating their wins.

1 Like