How has your context influenced you automation decisions?

🙋 Questions
,
1

Wouldn’t we all love to work on a perfectly flowing test automation project? With all the resources we would like? And access to all the tech we need?

What automation stories do you have to share about the context of your test automation efforts?
What were some great results, failures, or challenges?

3 Likes
2

Rather a lot, since my context has never involved software that could be unit tested or API driven. Which left UI automation as the only viable option.

The context also involved transactional flows - in one case involving point-of-sale software that included features like support of multiple different taxes and multiple currencies (and multi-currency display) and had been in continuous development for nearly 20 years when I started there.

In that system, the test automation had been running for close to 10 years, and had evolved from a set of record/playback scripts to multiple suites which, by the time I left, were well into the million lines of code territory and included a relational database in CSV files to drive the test automation.

It was horribly complex and scared the pants off anyone joining the automation efforts, but it worked and most UI changes could be handled by changing the locator in one file. Flow changes took more work because the step CSVs had to be adjusted (renumbering the things took most time). New functionality in areas that were already covered could be done by adding a single function, a call to the function in the appropriate place, updating the driver CSV files, and running the updated scripts to generate new baselines. It could be done in less than a day.

My current workplace, I never got automation working properly for several reasons:

  • I was the only tester, so I was doing everything else test-related as well as a fair number of other test-adjacent things.
  • I was working with classic ASP which meant my only option was UI-based.
  • I had no way to start fresh with the exact same conditions for each run - the software was multi-tenanted, and there was no way to create a partial restore that only held the clients I wanted that kept up to date with the changing data structures.
  • The software did not have simple, clean “units” of work. The base unit was one company running payroll - in which there could be employees added, employees modified, various types of pay added, and so on.
  • Before I had the time to stabilize my automation, we were sold to a different payroll company and started a project to migrate all our customers to the new owner’s system. As a result what I did was shut down because we weren’t adding features except for specific things needed to assist the migration efforts.

I won’t say it was a complete failure because I learned a lot while creating it and would do things differently if I had to build a test automation harness from nothing again, but ultimately it was a failure due to factors completely outside my control.

That said, I’d do it again.

4 Likes
3

The two most divergent stories are the project I am working on now and the worst code base I ever saw.

WORST CODE BASE EVER

In the worst code base I ever saw it was the company’s flagship product and although it wasn’t inherently very complicated the code was an absolute fucking mess. They would release between once every six months and once every month depending upon the customer and had a multitude of emergencies happening routinely, most of which were caused by the shit code. There was a big (multimillion dollar) contract that was getting very angry and was about to walk away.

There were 10 devs and 2 manual QAs. They’d done a bit of test automation but all attempts had failed due to a combination of flakiness, cost and lack of skill. Some devs were trying to write unit tests but these things were fucking useless because they were a mess of mocks and wouldn’t pass or fail reliably.

This is where my interest in test automation grew from. I abandoned feature / bug fix work and worked full bore for about 3 months to build a reliable, consistent, hermetic end to end testing framework. It paid off pretty handsomely in the end (for them, I got fired) as reliability crept back to our releases and I was able to refactor the dirty guts of the system and consolidate builds.

CONTRAST:

My current project is much smaller - it’s a microservice, but still quite complicated. It is built by me and 3 other really good devs. We use TDD for everything. The code quality is top notch. We don’t have any manual QA and release between 2-3x daily. I can’t actually remember the last time we had a proper bug in our code (most bugs are in surrounding microservices).

I’d like to build out some more sophisticated test automation but in my heart I know it’s not actually worth it on this project.

The one tricky aspect is that it involves LLMs which I have to deal with somehow. These have had “bugs” which were fixed by prompt tuning but we don’t have tests for it yet.

Nondeterminism is a theme that crops up again and again and again and always requires creativity to deal with. For the awful code project, the nondeterminism would crop up in terms of sleeps and select * queries without order bys which caused test flakiness. For my current project, I’m dealing with a nondeterministic LLM.

5 Likes