How to move from testing 'off the shelf' software to testing a 'service' (Trunk Based)

🙋 Questions
, ,
1

I’ve been in my new job for over a year now and have constantly felt like I’m ‘swimming against the current’ in how I’ve been trying to test and automate. It’s just been pointed out to me that my past jobs have been testing ‘off the shelf’ software that users install and run themselves (6 month release cycles). The current role involves testing changes for a live service/website where they are moving towards CI/CD and weekly releases.

In the new company there are only 4 environments (dev/qa/stage/live) that can run the whole product (it’s not containerized at all). The company focus is on ‘shift left’ and getting most of the testing done in CI, before code is committed/deployed.

I’m used to testing after code is committed, built and running on the full stack of dependencies. So far in the new company my focus has been on full end to end testing of the qa/stage environments (selenium) to help remove the manual regression testing that was being done before every release, but I keep getting pushback that things should be tested lower/earlier.

Has anyone else out there moved from testing ‘off the shelf software’ (like Word) to testing a service (like Facebook/Ebay - e.g. a website that does stuff)?

  • What differences did you find in how you tested the product?
  • How does testing environmental differences come into the picture? (particularly with Trunk Based Development?)
    (e.g. it works in QA, but not Stage because dependant product X is at a different version)
  • What sort of ‘full stack’ testing do other companies with Facebook-like services do? Do most people rely on lower level tests and not have full stack test automation?
  • Any other wisdom or stories you can share that might help me get a handle on how I need to think differently about things (or maybe I shouldn’t and the company I’m at is wrong!)

Caveats:

  • I may be mixing up ‘off the shelf’/‘service’ testing with the differences caused by Trunk Based Development, but they are both hitting me at the same time!

Many thanks for your thoughts in advance!

Martin.

2 Likes
2

Hi,
Nice to see you asking for help when getting stuck. It’s not intuitive sometimes how to adapt your mindset and can many times take months or years of trying things out.

Here’s a few things on how I’ve changed my approach when doing a similar change of product type:

  • be part of as many meetings, conversations, and emails as you can where product, business, requirements, etc. are discussed; this helps in keeping track of any quick changes that occur between releases; you might need to learn a lot more things quicker to integrate with others; show that you understand and are of value in these meetings asking reasonable good questions, talking testability, highlighting risks;
  • have better relationships with developers, with the code repository, products packaging, and deployments; this would again help you quickly see changes in the code, do walkthroughs, and inspections, already get an idea of what you might need to test; I have generally changed from waiting to be informed by a ticket move, to being proactive and looking for how I can support quickly and often;
  • learn and prepare while development happens, as much as possible in sync with developers in regards to change progress; once code is pushed you start checking it;
  • have the same development environment set up on the local computer or VM; learn the basics of managing the product; start the product locally and check the change before any automatic pipeline execution/peer review is happening;
  • it can take a team to build another product(UI automation); things are moving faster, testing needs to happen asap; if you do something by yourself you’ll fall behind and get pointed out as being a bottleneck; so agree with the team on the approach, and get the manager/stakeholder to approve time for the team; you have to manage the project yourself, but need support/time from the team;
  • create more regularly smaller code scripts; use various ready-to-use tools, addons, and extensions to support testing.
  • advocate for unit and integration tests from developers; support with API, subsystems, services database/data testing;
  • create system configurations with new test data, or product setup for developers; they might want to check their code and need the product in specific states;
  • add canary releases, blue-green, ab-testing, dark launch, or similar process; sometimes the team might want to test in production or quickly release something risky; this is a chance to evaluate with a lower user base or just internally in the company how the product change is going;
  • increase and add a higher priority to logging analysis and testing through hints from logs; it can happen that the team misses issues in production faster; logs can help with identifying some of these events; create scripts, queries, dashboards to analyze and investigate potential problems in various services;
  • might apply to both types of products, but consider testing or finding/learning from a lot of other things besides the deployed product: product ideas, a whiteboard schema, architecture/solution design, ui/ux mockups, requirements, written tickets, specifications, user interviews, analytics, data, documentation, guides;

I wouldn’t look for what the big companies do, they have the best development engineers, a huge amount of resources to support whatever approach they choose, and maybe too little testing.

3 Likes
3

You have lots of complex, mixed but good questions :slight_smile: but I believe, I understood the essence of your post and I’ll try to help you to find answers at least some of them :sweat_smile:

Firstly, you shouldn’t try to copy the development and testing strategies of big companies like Facebook or Google, etc. Your company likely doesn’t have the same resources, processes, risk tolerance, the ability to test on real users, or the complex infrastructure for CI/CD, dev and QA processes.

In general, there’s not a huge difference in testing approaches between testing off-the-shelf soft and live services/websites, whether it’s trunk-based development or feature branching.
Testing should start early and be present at all stages, from reviewing concepts and docs, requirements to deploying features on prod.
You don’t need many test envs - in many cases, you can test locally by running the app from the code.
Instead of using one branch like trunk and stable, you might have dozens of branches for each feature and bug to test them separately. Regression is tested in the same branch. Then, the tested code is merged into the master and shipped to production. Technically, you don’t need to test the merge, but you always need to rebase these feature branches on the master after each merge to the master.

Agile teams and modern companies often don’t have the resources for full ee2e test automation. They rely on unit tests written by developers, and QA may add some auto tests (even e2e) during the 1st phase of testing, but many checks are performed manually and never fully automated due to rapid development and change implementation.

You need to test quickly to find major issues; you don’t need full coverage or to perform full regression every time. It’s important to communicate closely with developers to understand what to test, how to test, and when to test to minimize your testing scope.
Frequent releases are important, so features are usually shipped as they are ready and tested (except for desktop or mobile apps) for web services.
Development might proceed in such a way that different changes on different microservices and changes may not be implemented simultaneously. So, you can test different services, APIs, and UI separately or start testing some functionalities before the entire feature is completed and then perform more like smoke/regression integration testing.

I have always worked in a relatively rapid and agile dev and QA process and basically, I only have had the transition from a trunk-based development to feature branching so it might be not really clear from my post how to transfer from off-the-shelf software testing to different processes. But anyway, I hope it’ll be helpful and help you ask more precise questions to continue our conversation on more narrow topics :slight_smile:

3 Likes
4

Many thanks shad0wpuppet. I think you’ve understood my rambling question (was it actually a question) pretty well!

I’ll dig into a few bits in more detail if you don’t mind. I will likely again be unable to properly articulate the actual questions I’m trying to ask, but even writing things out like this is helping me try and work that out myself!

My intention mentioning Facebook was more along the lines of saying what the application we are developing is like - it’s a permanent service (website) available to employees and customers of ours.
The testing used to do was software we released where a customer will download and install it themselves. Some of our customers were huge (10,000 users), so if we released a new version of the software, it absolutely had to work, or 10,000 of their employees wouldn’t be able to!

With ‘of the shelf’ that I’m used to, I can spin up as many applications as I like all pretty self contained - it was fairly cheap and easy to do. With this new ‘service’ website it feels muddier. The application talks to several other systems, there’s a communal ‘Login’ server, databases, and other services developed by different teams. (I believe when the devs are running locally they point at the deployed ‘dev’ env services where needed). Additionally, the development team are split - front end and back end. The first time front end and back end are generally put together is post commit/build in the Dev environment (they even have separate builds and deploys for front and back ends).

In my brain (and maybe this is a problem), the only way to be sure that the application actually does what it’s supposed to do, is to use the entire application. I logically get that you can test all the small components individually, and that should add up to ‘everything works’, but I suppose I don’t really have faith in that, whether it’s how complete our unit tests are, or that there are too many bits that just cannot be unit tested (like calling external services and discovering the firewall rules have changed and we can’t contact it).

“Agile teams and modern companies often don’t have the resources for full e2e test automation.”
Most of the QA’s at the company are domain experts rather than software testers (e.g. they have a financial background, rather than one in software development). One of the main reasons I was hired was my experience of test automation (to replace their manual regression testing), which I believe I’ve managed to do.

It’s also possible that in the time I’ve been here that the company has shifted (or tried to shift) ‘Left’ and a move to Trunk Based Development, where the focus (from the people pushing TBD) is on testing everything before commit, which doesn’t seem to mesh at all with trying to do e2e testing. I’m not sure how the domain expert QA’s would manage to do testing before code was committed. An interesting question for me to ask!

Q: How do other people handle testing websites/applications/services?

  • What split do you do of testing at the lower ‘component’ level, and the integration/system/e2e testing for the application?
  • How much manual regression testing goes on before releases? What is the main aim of manual regression if everything is already tested at component level?
  • How realistic do people feel it would be to test enough before commit to be able to release a product without (much) end to end testing?

Sorry, I’ve rambled again.

For a bit more background, I’m currently trying to decide how to deal with the frustration I’m feeling here - change how I test, change how the company thinks/works, or just leave and go somewhere where my mindset/testing focus fits better!

3 Likes
5

Welcome to the MOT @martinphillips .

I too started in testing a desktop app, then a mobile system, and that leap from app to “system” and to web and service is huge. What used to take 2 minutes suddenly takes an hour if you are lucky, and it’s often not in your control anymore. I’m not going to offer any duplicate and extra answers to your question, but rather extend a warm welcome and hope you will enjoy your stay. You sure have written a great intro question and look forward to more of the same.

1 Like
6

Got a top 3 hints for the main changes you had to make when making this leap?

7

Probably different people had extremely different experiences with desktop apps and web apps :sweat_smile:
I work with OctoBrowser, in a nutshell, it’s a desktop client (Mac, Win, Linux), a browser (Chromium-based), and a bunch of integrations with backend web services. I can’t say that the difference between testing websites with microservices and this aforementioned app is very huge, partly this is because of the tech stack (many desktop apps use the same stack for app UI as for websites, technically this is a web page rendered, e.g. on Edge WebView on Win OS). Yes, there are lots of additional tests in comparison with the web related to the specifics of OSs, hardware resource management, rendering, performance, memory leaks, etc. But again I can’t say that any tests are much longer or complicated. There is only 1 thing probably, sometimes you need app builds and it takes some time to get one but at the same time building a couple of web staging also might be time-consuming :sweat_smile: in other cases, you can run everything locally. But again, it’s a matter of dev process, apps/systems complexity, tech stack, infrastructure, etc, what work for 1 team and software product may not work for others, obviously :slight_smile:

What used to take 2 minutes suddenly takes an hour if you are lucky, and it’s often not in your control anymore.
So I’m curious to here some examples from @conrad.braam :slight_smile:

8

My experiences of web cleint and services apps is that you really do have to understand the deployment and how it scales up and down, and then know how to spot site deployment script issues, certificates expiring, services linking to the correct versions, etc. Then either deploying your own, or waiting for someone else to deploy into the shared sandbox. 5 minutes quickly becomes an hour.

1 Like
9

Many thanks everyone!

Some useful thoughts and ideas for me to take on board.

Martin.