30 Days of Ecommerce Testing Day 27: Analytics oriented testing

In a mobile applications, i used to check analytics to know a bit more about our users.

Information such as:

  • Device model
  • OS
  • OS versions
  • screen density

Also, tools such hotjar, provided me cool stuff like heatmaps, recordings and etc. This allowed me to understand better our users.

1 Like

I think that is an interesting topic:
I think analytics can help us to know people in our platform. And it would be helpful for the creation of the Persona’s testing scenarios. Persona’s testing, as it was discussed in the 11th days, could be a good way to understand the necessities and the critical tests to implement in the software.

In the other way, we could use analytics in order to have a clear approach which pages the user tend to visit and what are the critical website to test and to create a regression test. For instance, if a user uses a lot the login authentication, it uses a lot the cart page and so on, it would be a good idea from the tester side to develop a regression test on those features.

Another think that it might help analytics is to understand which features might cause some important bugs: if you take look on the reports and suddenly you notice a drop then it is natural to think that something strange is happening. This could be a good occasion to review the features and detect / prevent possible bugs.


Analytics have become so important for every business to operate. It can answer so many questions- which products are bought the most, how user navigates, at what point most users turn away from the application etc.

1 Like

Day 27 https://wp.me/p9EXXo-6a :slight_smile:


Contributions from Twitter:

Good point about the persona with the privacy extensions Mike. Active content blockers such as NoScript make a lot of a difference when browsing the Internet but with all the malware served via ads or external scripts they are pretty much necessary :frowning:

Most of what I could find was around A/B testing, so I suppose that is the obvious answer. Google Optimize will let you use up to 4 landing pages and see which one yields the most results in the final product.

Isn’t this A/B/C/D testing? Well, I googled that, and, I’m kind of right:

When A/B testing is used in this way, a third or even fourth version of the page is often included in

the test, which is sometimes called an A/B/C/D test. This, of course, means that traffic to the site must be split into thirds or fourths, with a lesser percentage of visitors visiting each site

From: Optipedia Optimization Glossary

That lead me down another rabbit hole of univariate testing (same source).

Multivariate testing uses the same core mechanism as A/B testing, but compares a higher number of variables, and reveals more information about how these variables interact with one another. As in an A/B test, traffic to a page is split between different versions of the design. The purpose of a multivariate test, then, is to measure the effectiveness each design combination has on the ultimate goal.

Once a site has received enough traffic to run the test, the data from each variation is compared to find not only the most successful design, but also to potentially reveal which elements have the greatest positive or negative impact on a visitor’s interaction.

Now this is getting good! Any kind of testing that borrows terms like “multivariate” from statistics is cool with me.

BTW, on day 8 we talked about funnel analytics. To reiterate, this is where we track where in the process from “landing page” to “check out” we are losing customers. Back when I posted that (I was so young then, 20 days ago) I would have figured the main reasons for dropping along the process would be:

  1. Customer changes mind (I do this all the time)
  2. Customer drops connectivity
  3. Website Bug

As a tester we should be making sure #3 doesn’t happen. We might not be able to do much about #2 (it depends) and I wouldn’t have thought we could do much about #1. BUT now that we are older and wiser and know about A/B and multivariate testing, we MIGHT be able to make some tweaks to the site to keep that from happening. Cool!

-Dave K