Hi all, I’ve added my answers below. There are some areas where I’ve tagged other people as I don’t have much experience in those areas but hopefully these are useful 
-
What should I suggest to someone interested in learning automation where and how to start? - Learn basic programming skills as a starting point and don’t jump straight to automation tools. Make use of free learning resources such as Test Automation University (they have introduction to programming courses too and most recently they released an introduction to javascript course!). Start automating on the UI level as this is the easiest in terms of getting started because you can see it visually. Once you’re comfortable, you can then look at expanding your learning to automated API testing.
-
How do you get development teams to take ownership of writing and maintaining automated tests, to avoid having a separate QA team do it? - Choose tools wisely. For me personally, I’ve seen success if the testing tools fits nicely into their development workflow which is one of the main reasons why we adopted Cypress and Applitools. Once the tools are integrated, for new features that they are working on, get it added as part of their task so the automated tests are included before deploying to production. You also need to be patient and be willing to help them out when needed since this is pretty new to them.
-
How proficient does automated QA need to be in coding ? A lot of qas in auto these days used to be developers - Hmm I would say for starters basic programming is really essential then if a task involves setting up a test automation framework for UI, API and other types of testing, then I would say intermediate level and an understanding of design patterns such as page object model or singleton is beneficial.
-
How to convince developers to maintain and add tests to the automation suite? - See answer in question 2. Also, show them the testing manifesto
https://www.luxoft-training.com/news/the-agile-testing-manifesto/ and influence them that testing is everyone’s responsibility. -
I am working in a fintech company and I want to show the value of the api testing on business logic. What things do you think are appropriate to assert aside from the status code and the response body in this context? - You can also test for schema validation where you assert the structure and the types rather than the correct data https://json-schema.org/learn/getting-started-step-by-step.html. Another thing to test is to measure the response times as this will help in performance testing.
-
Other than exploratory testing, where automation is not recommended? Also, do you think exploratory testing could be automated? - Accessibility testing can be automated but it still needs human intervention on the most parts. This is also the same with security testing. No, explorating testing could never be automated in my opinion… Even with AI, I just don’t see it happening for some time… How can you automate something that you don’t know?
-
Are there any performance tester courses you can recommend? I have some practical experience of automatoin, including recently doing perf testing using Gatling, which worked out well. However I’d like to possibly do a professional-level course to give me any additional perf testing theory and a certificate. Thanks. - I’ve seen @heather_reid answer this thank you!
-
How do you (or your team) decide what to automate and what NOT to automate? - If you are starting out with your regression pack, document all the features that are high priority, takes a long time to test and repetitively verified. These should be automated. On the previous team that I was part of, we listed all the scenarios that are high priority and document it on Confluence. Then when it’s added on our automated regression pack, we remove them from the Confluence document so we know which scenarios are left. With regards to what not to automate, if a feature or requirement is changing frequently, then don’t waste your time too much on automating it.
-
Should test automation code follow and conform to developer coding standards such as checkstyle? or should it be kept separate? - Yes I think it should conform to the same developer coding standards especially if you want developers to be involved. Test automation code should not be seen as inferior to development code. - It’s good practice to follow coding standards whether it’s test automation code or development code because it makes your framework easy to read and understand. Eslint and prettier are now pretty standard to most javascript projects for example.
-
Why is mobile automation (like Appium) so hard to setup? - I have to be honest here and I actually don’t have a lot of experience with mobile app automation! But I think the reason why it’s hard to set up is because mobile app automation is still less mature than web automation and also the community support and tools are stronger on web.
-
How do you measure success/fail of your automation efforts? - For Success: if you are not needed anymore to perform routine tasks because these have been covered by automated tests and automated deployment, if your team is releasing frequently and has higher confidence, if the cycle time for tickets going from dev to done is shortened. For Failure: if you are spending all your efforts in test maintenance and not doing other types of testing activities, if your team don’t trust the results of your automated tests because it’s flaky
-
Do you know of any good online sources that would help people get up to speed in automation. - I agree with the other attendees. Test automation university as a start and they release new courses frequently and it’s completely free.
-
What tools do you use to report automation tests? have you used Xray JIRA plugin before? - I have used Allure and Mochawesome plugin and they are both great! I haven’t used Xray JIRA plugin sorry…
-
is manual qa dying out ? - There’s a lot of interesting conversations about this already and I believe that manually verifying tests should be minimised and automated instead. Exploratory testing and other activities that need human verification is still very much required.
-
Have you used visual testing tools like applitools Eyes or Percy.io for example. Do they have value? - Yes, we are using Applitools in our current team and this has speed up and simplified our cross browser visual testing process overall. It really provides a massive value and has allowed me to focus on other testing activities.
-
API automation, what strategy do you have for a very very big data? How many permutations to cover? - Oohh… I feel like @g33klady is best suited to answer this question as I haven’t worked with an API with a very massive data.
-
Integration/E2E for System of System testing: most large organisations still have monolithic applications (that do not lend themselves to contract testing) any suggestions for tools to automate, or processes that may assist with semi automation of this? - So, with monolith applications in my opionion, automated testing is actually simpler as opposed to microservices because everything is in one code base… Both UI and backend code is in one codebase and then communicates to a database so if you need to run your test, you only have to run it on one application. I don’t know what the programming language and tech stack is for this monolithic application however, In terms of what tools to use, you can still use most of the tools such as Selenium, WebdriverIO or Cypress for E2E testing. The main problem I see with monolithic application is that because everything is in one application, most of the tests are on the E2E layer and we know that too much E2E tests should be avoided because they are costly to run and difficult to maintain…
-
How do you handle the need to login in an automated test? - If you can do it on the API level, do the login on the API level. Then depending on your application, you can generate a cookie or token from your API test and store it in your UI test. It’s still good to have a test to cover the login workflow so you can have one test that does the login and for the remaining of your scenarios that requires a login, do it on the API layer.
-
When testing a mobile app, do you handle some steps outside of the UI (eg: Api) for less execution time? - Again, I don’t have much experience on mobile app automation but I would treat the process similarly. If I can utilise API layer on some of my steps, I will go via this route.
-
We are struggling to find time to develop test automation framework. QA Manager - Overseeing the status of each story; qa process; SDET - understand requirements - design test plan and test scenarios - create test data Offshore testers- responsible to write test cases in detail steps based on SDET’s test plan and scenarios - execute test cases manually. Please share how your team’s process is like. Any advice how we can improve based on current responsibility - So our current team process is everyone is responsible for testing, not just the QA team (there’s only 2 of us
so we really need all the help). Our team is broken down into 4 smaller teams and on each team, there are different test strategies but the common theme is we are using tools that are developer friendly. It also helps that our head of engineering as well as our leadership team all share the same view that everyone is responsible in testing. It’s a change in mindset definitely. My personal advice would be to gather data about your current testing process and visualise it. Then share this with everyone (not just with QA team) so everyone can see where the inefficiencies are and what areas need to be improved. -
Please describe your latest test automation process. Which type of automation tests, how many (i.e. compared to other types), who wrote what, who decided scenarios, who organised/managed everything etc. - It’s different for every teams (I’m part of 4 smaller teams) but to explain one, on one of the projects, we are delivering a design system. The types of automated tests that we do are unit tests, design system component tests (functional, visual and accessibility), front end tests for our documentation site (functional, visual and accessibility too). Initially, I was helping out in setting up the frameworks and writing the tests but nowadays, this is now the full responsibility of the developers. Everyone of us decide what scenarios to automate it. This is normally decided during “tech analysis” sessions where we look again at the ticket requirements and make sure that from the technical and testing perspective, the developer has everything they need to finish the ticket.
-
Is it legit to practice automation by contributing to open source projects? - You don’t have to, it’s not a mandatory requirement. You can practice automation by using existing websites or if you really want to also improve your programming skills, you can practice automation by creating a simple app from scratch.
-
Do your tests use existing data or create new data? - It depends on the tests… Some tests need data fixtures so it’s more reliable and stable. General rule is that if your test will modify the state of the application, then it’s better to manage it with your own test data so you can clean it up as well after the test run.
-
What website can we go to learn and practice test automation?
- Suggest checking out Products and sites to practice testing on
-
Is Java King for Automation, or is Javascript better for Frontend Testing? - Java is still the most popular language for programming but I can see from the trends that Javascript is catching up. Both languages have great libraries and tools that can support your automation so learning either of them will be beneficial
-
Does the cost of all the tools justifies itself? (applitest, cypress, k6 etc) - Yes! Cypress (test runner) and k6 are open source but they have paid options for additional features. We currently don’t pay for Cypress dashboard and I can see the benefit already because our developers are writing the tests. Applitools I really had to justify the cost since it’s a commercial tool but once the team sees the efficiency and the benefit it provides to us in terms of cross browser visual testing, then it is really worth it. Before using Cypress and Applitools tools, I was spending so much time writing and maintaining our tests. Now, I haven’t even written a single functional or visual test because the developers write it
and I’m focusing more on other important activities. This alone is a benefit already! -
When automating UI, do you keep your assertions in the UI or do you assert in different levels (Database for instance)? - Normally my assertions are all in the UI because when I automate the UI, I focus at what our end users sees. Database assertion would be a different type of testing on the test pyramid.
-
What is Automation? - Using technology or process to speed up existing processes that takes a lot of human effort
-
What do you think about quality assistance (especially in automation field) as a means for upping overall sprint/team productivity and what about the downsides, i.e. how do we ensure we still do great testing? - I think quality assistance when implemented correctly has a lot of benefits. It improves overall efficiency in my opinion since more people test and the responsibility is not on QA engineers/team alone. Instead QA can focus more on areas where they can improve quality. It also means everyone on the team increases their knowledge about the products that they deliver. For it to be successful, it will need a lot of collaboration, knowledge sharing and influencing from the QA team/engineer.
-
if required to automated a COTS application with customized business logic within it? Do you have any suggestions on how to accomplish it? - Main challenge with automating a COTS product is you don’t have access to its source code, it’s all black box testing. So go with the approach of automating just the core business logic rather than trying to automate everything. The vendor should have some automated tests (I hope!
) anyway. Also, rather than automating the features that it provides, you can try to automate that the data between your application and the COTS product is shared correctly. -
Are you able to suggest any approaches to retiring and updating tests in an automation regression pack? - You should continuously audit your automated regression pack. It’s ok to delete tests that don’t provide value anymore. Similar with development code, you should continuously maintain your automated tests so you don’t leave any tech debt. When deciding which tests to delete, ask yourself the questions “when was the last time this feature was used?” or “is this test still valuable?”
-
What coding language should we prioritize to learn to help us get started? - There is no definite answer here, you should learn the language that you want to learn but if you want it to be a whole team responsibility, then use the same language that your development team is using so they can help you with any programming queries.
-
What can be the lessons learned for teams starting with Infrastructure as Code deployments & testing from a more traditional testing automation perspective? Thank you! - Hmmm bit confused with this one as I am unsure of what “traditional testing automation” perspective means since there can be a few interpretations. But if you mean setting up CI pipelines with code vs manually configuring it, then in terms of lessons learned, get involved as much as you can and provide inputs. If you are not sure on how it works, ask the person who wrote it because that’s how you can learn. You don’t have to be an expert and know a lot about IaC tools such as terraform but it can be beneficial to learn the basics of cloud infrastructure setup providers such as AWS so you have that base foundation of things.
-
I am curious about internationalization automated testing for mobile apps. how do you check how an app looks in different languages? - Again, I don’t have much experience on mobile app automation so I’m hoping someone else with the experience can answer this
- Hey, that would be a great question for the localization category if we don’t get to it this evening https://club.ministryoftesting.com/c/all-testing-talk/localisation-testing/118
- How do you test for email notifications during test automation? - I haven’t had much experience with this so can’t provide a technical advice but I heard really good things about mailosaur https://mailosaur.com/. @filip would be better suited to answer this