I’d appreciate some views from those testers with real experience of A/B testing, in particular how they as a tester contributed and maybe a few pointers on what sources you leveraged from to build knowledge and skills in this area?
Here is some of my recent experience in this area.
The team I was working with had a healthy culture of explore, discover and experiment to help drive the product forward and A/B testing was a big part of this.
As a tester I was able to shift left getting involved in the Hypotheses discussions with the team, here we were testing a lot of ideas to design the A/B experiments.
Our OKR’s played a part in the hypotheses discussion, why do we think this experiment could have a positive impact on our business and do we think they could influence or OKR’s? for example.
There was still some hands on testing to do as often the B was new to the product, a different design, a potentially more efficient user route or perhaps comparing two features that have the same goal for the customer preferred option.
Testing had some challenges as often a variation is locked into a user profile so that once they get a variation they get it consistently, they team were great on this when I flagged this as a testability discussion and one of the developers built an add-on so I could quickly select any variation from a list regardless of device and user or other identifier that would normally force consistent variation.
In shifting right we had the control and analytics tools, these actually did some level of analysis automatically but almost always we had some level of discussion to interpret results.
Many times we often ruled out variation B, that was still a successful experiment as we learned something new even if it was not immediately a positive change to the product.
I had a side thought regarding whether A/B testing was actually ‘testing’, with my view that testing is all about asking awesome questions that guide the team and product forward, I’d have to say it is, here we are directly asking the end users questions with our variations and that’s testing 101 in my view.
So my exposure whilst limited has been very positive from a tester and testing perspective and something I am keen to do more of.
In particular that experiments culture really fits with my testing.
If you have experience of this, I’d really like your thoughts, they may help me contribute more with my tester hat on the next chance I get. Thanks