Lunch was tasty! There was excellent vegetarian food and I ate it all!
Next up is Daniel. He’s definitely a star in the mobile testing world, so I’m excited to learn from him in this talk.
His first question is “why is mobile testing so important?”. Well, in 1999 80 billion photos had been taken on old film. In 2015, 2 trillion photos have been shared. That’s only the shared ones, not even all the ones taken. That shows how much mobile devices are being used.
Mobile technology has really changed our life. That means that companies like Xing have had to develop strategies to keep up. Back in 2008, when they wanted to have their first app, they had no mobile knowledge. The first step was external development – but that turned out to be a mistake, because the external company had no clue either! Two years later, they decided to form their own team. There were 2 iOS and 2 Android developers, 2 testers, 1 product owner and one UX designer.
As fun as the team was, it wasn’t alone. There were 8 other agile teams, and the web software didn’t have an API – there was a lot of work to do. In 2014, they were having scalability problems. The pace of development from the web teams just couldn’t be matched by the mobile teams. Once the mobile usage exceeded web hits, the company came up with a new initiative – mobile unleashing. The idea was that every team is empowered to do mobile. They also started “full stack product thinking” with very cross functional teams, including mobile, data scientists, testers, agile coaches…
The team structure consists of platform teams as well as feature teams. The feature teams include backend and front end. Currently, there are 50 mobile developers and 20 testers that work on mobile testing.
This is where they are now. But there were challenges on the way.
The first one is to get away from “web thinking”. Working on a mobile device just isn’t the same as working on a computer. It’s a smaller screen, there are different scenarios, users want to (and can) use the product everywhere, not just when sitting in front of a machine.
The next problem was hiring. Hiring is a problem in general, and for a niche like mobile, it’s even harder.
Mobile test automation is the next challenge. It’s a lot of work to get automation up and running for the web, and for mobile platforms.
Communication was also a challenge. The speed of development means that a lot needs to be clarified. Face to face, slack, github – whatever channel. The communication is an overhead, but it is worth it.
The next potential problem is release coordination. Back in the day with one team, they could pretty much decide to release and just do it. Now, with so many teams and so many factors (is there a marketing campaign, are there dependencies?). Also, it’s not possible to just “roll back” a mobile app once it’s shipped. To counteract this, they introduce release trains: planned code freezes. There are fixed release dates, so the train is always on time (this is pleasingly similar to my knowledge from the Eclipse release train!). If you miss the train, the feature just doesn’t go live. The release trains are every two weeks, and they also allow for final checks before the product is released.
The release train involves 10 days of coding, then there is the code freeze on a Thursday. Beta testing happens for 7 days, and the teams do the manual tests that can’t or shouldn’t be automated. The rollout is 20% of customers for 2 days, where they do monitoring and gather feedback. Depending on what they find out, they might require a hotfix or stop the train. If all goes well, then 50% users get it for 5 days and so on. (My note: keeping all the potential releases in the wild configured and making sure you can update from each one must be a challenge!).
Back to test automation and how to scale that. They used espresso and keep it functional for Android and iOS when they had one team. But that became difficult with more teams – because it’s two frameworks, different languages and there was no code sharing. And this is next to the web tests (Ruby, Selenium). So, in about 2015 they migrated to Calabash (which is now dropped from Xamarin, so its future is uncertain). That worked well, although the migration was a long journey.
In terms of build pipeline, tools and devices, that was also something that needed to scale. The platform teams can’t be responsible for that for everyone. So they introduced a releases team. The build pipeline involves a local test run and then a pull request. The pull request kicks off static analysers and various tests (unit, integration and end 2 end). Following the pull request and the tests, there is a code review. Only when the build pipeline is green can the code be merged. The releases team also manages the internal app store. The Xing employees use the alpha/beta releases on their phones. The internal app store also contains apps they develop in Hack weeks.
Physical devices was also a challenge. With around 300 devices, they need to be tracked (both in terms of where they are and their status (OS version etc)). So they built an app renting system. They have an Android test farm that can be accessed via a browser. Obviously, it’s not quite as good as having a real device, but it helps a great deal. For iOS, they also use Sauce Labs for devices they don’t have.
To sum up, Daniel mentions that:
- Communication is key. All the improvements they’ve done come from talking about problems and challenges and feedback about things they’ve tried.
- Hiring is hard, and there’s nothing he can add to that.
- Mobile skills are important, and so people should pair test, learn more and work with real devices to get better at it.
- Invest time in your environment – that’s something that will always cause problems if you don’t give it enough care
- Reflect on your work and setup regularly. Don’t stick with the status quo – keep questioning whether you can improve