This question sprung to mind because of a situation I had today. For this particular project, the QA team have recently joined. Prior to joining, the project office was doing all the testing the project. Now they are passing the testing responsibilities to us. So far, I believe they have only focused on testing Chrome up until now so it’s obvious to assume that Chrome is the browser that we must support.
Today, my QA lead said “If you have time, it would be good if QA can do as many browsers and devices as they can for the frontend to find regressions”. Yes, it naturally makes sense to have the website work for our audience. My logical mind understands that.
However, there is no documented requirement for which browsers we should support and which ones we have to test for. Because of that, the statement from the QA lead sent another part of me into fury. The devil inside me utters “Why should we do testing if it’s not a documented and defined requirement from the project office?”. “Ok say I have time, what browser should I go onto next after Chrome then?” “Where is the prioritised list of browsers?”. We don’t have those answers.
In my mind, this multiplies our testing effort and I don’t like this attitude of ‘It would be good if’. I’m more a ‘This must be done’ person so I prefer proper exit criteria every time we do a release. Besides, if we do find something (for example in Internet Explorer), how can that be called a regression if it wasn’t developed and tested for IE in the first place? I would rather spend time on test automation then undocumented requirements for cross browser/responsiveness testing.
Maybe I’m wrong, Agile says to value working software over documentation. I dunno, I’m just ranting, I wonder how organisations deal with this.
The team doesn’t define requirements. Somebody ordered a product, they will specify what browsers it needs to be supported on. You can’t make up requirements because otherwise I would say it should support Opera, Tor and whatever browser they never heard of… just to make a point.
So yea just ask the stakeholders and so you test accordingly.
We dealt with this same question for years at my company. I felt all the same things you do as I am also a “This must be done” person.
I did learn that the reason the decision wasn’t made because the business didn’t know how to make the decision. How do you determine it is acceptable for a set of users on Internet Explorer or Edge to be left behind? The business was frozen with that thought. It took everyone; dev, testing, and product designers working with the business to get a baseline of browser requirements. The business needed feedback and recommendations to even start to make a decision.
Then we added analytics to our site and those metrics are used to determine which browsers and devices need to be supported.
I’m glad I’m not the only one here, so I imagine I might get into the same situation as you very soon with everyone involved. How did you test before you guys got a baseline? That is the main thing we are struggling with right now, it’s just like ‘whatever browser you like, go for it’.
When we didn’t get a definitive answer we told the business we would test on the four major browsers (Chrome, Firefox, Safari, and whatever browser Microsoft had at the time) on desktop and mobile. We also said we would only verify core functionality is working and generally make sure the “look and feel” is correct. For us core functionality is searching, viewing items, and submitting contact forms.
Since we didn’t have requirements, this was all done in an exploratory way. If things were missed we used that as an opportunity to ask for requirements. It was very tough at times, but after doing that a few times the testing team was able to push the conversation along by giving feedback on what browsers we are doing really well in and what browsers we may want to work on.
To get through it, I thought of it like a fact-finding mission. We are constantly bringing feedback to the different teams in the form of bugs, so feedback on the different browsers is no different in my mind. I have learned we can help to get the requirements we need by first giving information to those who need to make the decision.
The final say should be business, as in supporting a browser means higher reach as well as a higher cost of development. The one body that should have the ownership of making that call is business.
I currently work with a highly device compatibility / browser support product. And this is not a “Support this browser.”. Business also wants to have new features some of them are best done by breaking some browser support. And that information comes from the architects. And finally the call on what you test on is for whoever decides your test strategy to define. Like we might officially support all Android 7 devices, but we sure don’t test on every one of them. Also when it comes to mobiles and browsers they are connect to the OS version more often than not. So you need to specify what OS versions you support too.
And for this it is good to understand Equivalence Class Partitioning. As in these two “browsers” / “mobiles” etc. are for the purpose of this test equivalent so I only need to test it in one of them.
