Identifying Key Requirements: What Factors Guide Your Tool Selection?

Discussions
, ,
21

Hi, I came up with these thoughts after using the software.

Requirements to Consider

  • Is there already a testing system in place?
    • Is there a way to make use of it, if it exists?
    • What is the current costs to maintain it?
      • Does using another tool save us money in the long run?
  • What kind of testing do we need to cover? Security, API, Regression, System, Acceptance?
  • In what ways, could this project grow?
    • Will there be more administrative features or employee features?
    • Or will it target a different audience?
  • What is the ease of use for the current and future testing software?
  • Do we need to consider other testing software, if the current testing software applies to a different use?
    - Example: Is accessibility testing needed? Is Cypress or other similar tool used?

Impact on Tool Selection

  • How fast will that project grow? What kind of releases are we going to have?
    • Will there be fast and small releases where the code changes often, or will there be long releases where the testers and developers have time to review the software before releasing it?
  • What team skills do we have?
    • How big is our team? Do many people do automation or programming or are they primarily manual testers?
    • Do we have the capacity to maintain testing software?
      • Do we have the capacity to write new tests?
  • Who is going to get the most benefit out of the well-tested software?
    • the managers?
    • the employees?
    • accountants?
  • Is the tool going to help provide reporting to the expected audience?
  • Does the future testing tool have an active support team and/or a community?
3 Likes
22

Afternoon. Just had a go at the ten minute, create some requirement points for tool selection before hopping on to this area :grinning:
My list was …

  • Is there any existing tools within the company?
  • Front end or back end testing required?
  • Is automation definitely required? Ie speed. If a one off maybe a manual process will do
  • Would API calls be sufficient for testing?
  • Does the UI need to be interacted with?
  • Can we establish team members with the experience in tools that have previously been used?
  • What is the expected test coverage?
  • Is there any need for assessing critical behaviours? Prioritising areas
  • What platforms need to be covered? Different devices and/or browsers
  • Is there any need for performance testing as part of the process?
  • How will reporting take place? Need for a generated report at the end of a run for general consumption?
  • Is there a requirement for plugging into a CI/CD pipeline or to be run ad hoc?
  • Any concerns over maintainability?
  • Have the Verification/Validation criteria been established?

Happy Thursday :grinning:

4 Likes
23

Hi all, so finally got around to getting started in this.

With the general nature of the question, my simple list is:

  • Cost. We would all love to work without budget constraints, but this is the real world. I would add that this can roll into my next point

  • Alternatives. This is more to do with internal alternatives, as in, can we do something ourselves ? There is a budget and we are proposing buying something that we might develop ourselves and use the monies to higher more people(or buy that pool table to fill that massive gap in the office!!).

  • Current skill set and learning curve. Is it a tool we can learn quickly and get value from straightaway ? Does it come with good training, or something that is used throughout the sphere and has plenty of how-to videos(e.g. Postman has great support on YouTube etc…)

  • Reusability. Is this only to fill a gap on a project or is it going to be used elsewhere, and in the future.

  • Compatibility with current tech stack ? Can we easily implement it into our current tech stack or do we have to involve IT, DevOps and others outside of Engineering.

  • Reporting and Logging. Will it produce reports and logs out of the box, or are we going to have to wrap it in something that does

  • Support & Maintenance. Has it got support when inside our office hours for time zone ? What are their SLAs ? How often do new releases come out for bug fixes and new functionality ?

  • Reputation - Sort of a follow on from the previous. I would say checking places like MOT to get feedback from peers. Also, do they offer discussions with current Clients, warts and all ?

2 Likes
24

Hello :grinning:

How easy is this tool to deal with:

  • Usage
  • Maintenance
  • Reporting
  • Updates
  • Logging
  • Training

And more:

  • Compatibility: trying to find out what our company has and whether it can be integrated into the tech stack
  • Security and the like: what are the privacy settings, complies with company policies
  • Documentation and support: presents a good catalog of documents, they are easy to assimilate, there is a support team, there is a help forum
2 Likes
25

These are some great ideas.

The only one I haven’t seen mentioned yet is parameterization, by which I mean the ability to write one test, run it with several different args, and get discrete reporting for each one. Apple’s new Swift Testing purports to handle this well: Go further with Swift Testing - WWDC24 - Videos - Apple Developer.

1 Like
26

I had some experience with selecting a tool to be used for automation.
Most of the time it ends up with the code language your team is using as a key decision point. From there you select a tool to be used for automation.
Consider also your team construction. If it is a front end dev team only, most of the time the code language is Javascript or Typescript. If it is a back end dev team only, most of the time the code language is Javas or C#. If you have a feature team with front end and back end dev, it could be wise to select a tool that uses the code the front end team is working with.
In all cases you have direct support of your dev when you encounter some coding issues within your tool. Your dev uses the same language and understand your code quickly. (S)he can pair up with you to do some troubleshooting.
In mobile testing there can be some challenges. All depends on the frameworks the mobile development takes place. Let your dev team guide you in making the tool decisions. When they decide with you which tool to use, you have the dev team involved and are more willing to support you

2 Likes
27

I know a lot of these have been covered by previous users, but here is my 2 pence worth…

Specific Use cases: Does the tool address specific challenges of the business/project? What am I going to use the tool for? Development? Testing? Data analysis, collaboration etc.

If it is a specific challenge that is met, great, but could this be expanded out for use with other/future projects? Allow for modifications and scaling.

Features: I consider whether the tool offers the functionality needed out-of-the-box or whether effort will need to be made to customise?

User Focus: It it easy to use? From a UI and UX perspective? if team members are getting frustrated the tool is not adding value. Read online reviews and forums to seek the answers to this out.

Integration: How well does the tool integrate with existing software ecosystem? Does it support relevant standards?

Performance: Can the tool handle increased demands?

Growth/Scalability: Will the tool scale as the business grows e.g more users? added features? Larger/complex data sets?

Security: Does the tool meet security standards? Data encryption? User authentication? Is software compliant with industry regulations? e.g GDPR?

Customer support: How reliable is the tool supplier in terms of updates, support and roadmaps? Are there live chats? 24/7 support? and active user communities and forums?

Reliability: is the tool reliable? Minimal downtime?

Tool Performance: is the tool responsive?

Free trials: does the tool have a free trial period for evaluation? I would want to test and pilot before making a commitment.

Does the tool adopt modern technologies and is it likely to evolve within industry?

3 Likes
28

In my opinion, the most important factor in deciding on a tool is the team itself because, ultimately, the team has to use the tool.

So how many of the team members are aware of the tools that we are planning to choose,
how many team members are willing to upskill,
how much time those team members will need to upskill
how many additional members will we need to maintain the testing on that tool?

E.g. if we decide Cypress and have 10 QA and out of QA how many are aware with Cypress, how many are aware with just Javascript, how many can upskill themselves, etc?

Apart from that other several factors need to be considered while selecting the tools like -
→ What is the company’s budget? if the company is willing to pay for the license then we can go ahead with AI-based tools like Katalon, test-rigor, etc… if the company is not willing to pay for the license then selenium, cypress, etc.

→ Compliance issues - what are the compliance issues with the data of the customers of the countries to which those customers belong

→ What is the time planned by the company to set up the automation framework like 3 months, 6 months, or 1 year

→ What is the company budget for hiring? does the company want to upskill the existing qa for the tool they are planning to choose or are they open to hiring testers specialized in that particular tool?

→ What is the programming language used for the frontend and backend development side of the application, because somewhere if the automation framework is in the same language in which the application is being developed it can be a plus point.

→ Whether the stakeholders are technical or non-technical? if the stakeholders are non-technical and still willing to know about the frameworks then a cucumber-type of the framework can be helpful.

Apart from all of the above factors one of the most crucial points for selecting the automation tool is understanding the project, every tool can not be used for automating all types of projects. Every tool has its flaws, its limitations so understanding every in and out of the project and the limitations of the tool should be analyzed to avoid any issues in the future.

for what type of applications we are looking for tools, like web applications, mobile applications, or API testing, because when it comes to web applications we have lots of tools, but for API & mobile we have fewer tools.

2 Likes
30

I put together an article with some of my experience and how I direct some mentees on how to choose a tool (it’s mainly in Portuguese but you can translate it via browser): Como escolher um framework de automação – fazendo uma POC (Proof of Concept) – Testing with Renata

3 Likes
31

I list key requirements by;

  • understanding how much of the application is stable
  • which features can be automated
  • my team is familiar with which languages and the tech stack matters
  • how fast the work needs to be done
  • the community support and documentation matters
  • is the tool scalable, what are its limitations
  • budget matters

Impact on Tool Selection ;

  • mostly the learning curve and time to execute I get is very short, so all these factors make it easier to select a tool that can be implemented and results can be achieved as management wants
1 Like
32

Hi @mwinteringham :wave:

KPI’s mapping wrt to requirements and tools.

:v:

1 Like
33

Hey!

  • Maintainability
  • Test levels we want to automate
  • What metrics we need to export from the automation
1 Like
34

Here’s my list - which I could definitely expand and enhance after reading all the previous posts! :smiley:

- What are the anticipated benefits of automating?
- What are we planning to automate?
- What are the requirements?
- What team(s) would develop and maintain tests?
- Cost of tools vs available budget
- Available support/community
- Available documentation
- Skills required vs skills available
- Time needed for implementation vs urgency
- Integration with other tools e.g. continuous integration tools, reporting tools
- Reporting capabilities
- API mocking capabilities
- Test data generation capabilities
- Reviews or case studies from other organisations
2 Likes