Help me setting up framework for test automation (Android, Appium, py)

🙋 Questions
, , , ,
1

So I’m pretty new in this automation world. As a sw tester I been testing apps on multiple platforms, but when it comes to automation, I decided to start with mobile devices first, starting with Android. I wrote several simple test cases in python and I used Appium and Androids Studio to run them. I wrote several separate test cases and ran them manually and they work fine. So far so good, but now I’m stuck on this step. My plan for future steps is to add the following:

  • add option to record/screenshot failed tests instead of failing test only
  • catch exception, add logging
  • organize tests into group or something, add some kind of structure, framework
  • add some kind of dashboard, graphs, charts

I’d appreciate any kind of advice or comment on how to proceed with my next steps. Especially about organizing tests and create some kind of framework where tests should be run easier and more transparently then manually running them one after another together with logging and collecting success rate results.

1 Like
2

The appium forums are a great start place for a lot of the how-to parts of it https://discuss.appium.io/. For the python specific implementation bits you need to use your test engine, are you using pytest? I use a fork of pytest, so may differ what we see.

  1. I am inspecting a internal variable in the test runner class that contains the verdict, in the tearDown() method, the startup and teardown fixture names may differ for you test engine, but basic ally I look at that verdict and if self.result == “FAIL”: I branch my teardown to do things like grabbing a few extra logs. I always grab a screenshot, screenshots and any smaller logs are good to have even for passing test cases.

  2. use a file logger class, you will have to work out what works for you, we have a commandline parameter that holds the output folder name. We join the test suite name to that path and then dump all logs into that path. Each folder holds logs for just one test case, so it gets easy to drill down if you do not have the web index page open.

  3. I’m not using vanilla pytest but basically each class you derive from the base class is a suite. You will have lots of base classes in the end, lots of them, once you have something working well, take time to refactor and tidy up. But you will still have lots of base classes.

  4. Dashboards are overrated and rot very quickly. Build an HTML report/index per test case and per suite, with a landing page. It becomes a hierarchy of pages that allows you drill down and then access to all the logfiles - in a way that helps you bugfix your test runs. Dashboards are for managers, and they change their minds, so invest as little as possible into them. Ask for loads of storage space in your CI or other system so you can keep logs for longer. Trust me you can never have too much space for test logs later on. Focus first on making triaging failing tests easier, dashboards will then be more meaningful.

2 Likes
3

separate point, you do not “need” Android Studio to use appium, but Studio has a better installation experience and an updater than the manual java/node and adb installer. Personally I suggest getting a mac as soon as possible and getting your tests to work on iOS as well or else you will find you have to rewrite much of your test framework code later just for iOS, to later tidy the spaghetti that will start growing is a pain. All just my opinion after doing this for about 15 years with native apps, your context will differ greatly from mine, so please do not take my suggestions as anything more than just signposts in a desert.

1 Like
4

Thanks for your response, sir. And happy 2024.

Now to clarify some things you addressed. Correct me if I’m wrong but I don’t think I need to explore appium part much further to make a progress on the points I mentioned earlier.

As for the engine, so far on this stage I’m not using any kind. Tbh I’m not even that familiar with the idea of test engine. For now I’m merely writing separate py scripts in PyCharm and run them manually using Appium and device simulators from Android studio. I covered a few test cases and so far it seems ok, but like I said, those are still just a group of py scripts in some folder than I have to run manually one by one. I guess should explore Pytest and try to use it next, though.

If you don’t mind, could you elaborate on that “fork of pytest” and how do you use it, what’s the basic idea behind it?

1.) I like your method, I’m thinking about something like this yes. Now I’m still trying to grab screenshot and logs for failed test and once I get this working will I plan to hang it to failed test, as you described.

2.) Noted, thank you for the explanation, I’ll try to do as you suggested

3.) Noted, I will explore the use of base classes. I just need to be careful and make sure to close virtual device after each test case, otherwise memory consumption will increase too much.

4.) Interesting idea, I haven’t considered using a simple HTML report/index per test case + landing page, but I think you’ve got a point. Why complicating if we can start simple, right? In any case, this step will probably come last and I plan to invest less time and effort into it.

Finally, thanks for your suggestion about getting a mac and getting my tests to work on iOS as well. I know it totally makes sense, but let’s just say there are some reasons why I focus on Android first and most, for now. The basic idea for now is to cover android first and then if necessary add iOS but surely not in a hurry and it’s not top priority. But yes, I’ll keep it in my mind.

5

Glad that heled you TesterTester
pytest is a unit testing tool very similar to tools like Junit and CXXTest, in that it includes “fixtures”. Fixtures are your friends, my favourites are the setup and teardown fixtures. These are “callbacks” that will run before and after every single test case and are mostly used to makes sure that every test starts with the system under test in the same clean state. The fixture that runs after a test is called the tear-down. And that’s the one that I use for all of my logging gathering.

Appium is huge, you will find that once you make the choice to use Appium, you will have to learn more about it’s inner workings. Like so many tool stacks it has it’s downsides. Always use the latest version whenever you hit a snag. The tool also takes away a lot of the platform security problems, so be aware that this will make some security scenarios harder to test.

Appium uses flags called capabilities, just like webdriver does. Note that most capability defaults assume that you are a novice. Whenever you want to test anything remotely advanced you will have to turn a few of those “safety rails” flags off. So knowing how the tool works internally does become necessary after a while, but for now, defaults will be fine. Also the docs for Appium are huge and not easy to navigate, but have now moved to github, so that should get easier.

1 Like