Choosing test automation tool: TestComplete, Cypress, Playwright. Comparison spreadsheet template

🙋 Questions
,
1

Hi,
We are currently assessing and comparing 3 tools for UI automation. I would appreciate it if you could share

  • your experience using any of the above tools. What you liked, what kind of issues you came across.
  • your thoughts and suggestions regarding the process of choosing test automation tool. What are important factors and things to consider.

Context:

  • relatively small QA team, formed 8-10 months ago
  • 5 “manual testers”
  • 1 automation tester, recently hired
  • most of team members do not have much experience with test automation,
  • 2 team members have intermediate level programming skills and CS background
  • product: web app, 3 instances (AU, CA, UK)
  • mobile app is planned to be released within next 1-2
    years
  • we have done proof-of-concept for all 3 tools
  • our web app has a lot of iframes coming from WP, multiple domains

Here is the google spreadsheet (this is not the final result) I created to compare tools which
hopefully will help us to make the optimal choice.
Did I miss something? What can be corrected, improved?

Thank you!

image
image2469Ă—962 213 KB

4 Likes
2

Looks good so far - good idea to do a prioritized scoring. But don’t put too much into the calculated final score. :wink: Here are a few things to consider additionally:

  • Are there any considerations wrt hosting the solution?
  • does it align with company tool strategy or similar roadmaps for dev tooling?

/Jesper

2 Likes
3

Wow, that’s an impressive spreadsheet :smiley: There are some empty fields, does it mean you need to finish them? (because from my knowledge of Cypress, many of them could have 5s. Also, I thought that iframes testing was actually one of the Cypress weaknesses. Maybe they improved it since then. ). Anyway, it seems that price might be the main decision factor for you. At least, when I was checking TestComplete (as we have been using Swagger by the same company), the price was the main reason why we didn’t go with them. Otherwise, if you use Swagger, Zephyr etc., then TestComplete could be a good combination, I believe.

I would personally try to write a few generic test scenarios in each of the tools (the same scenario in each of them). To see what your team prefers.

For example, I like that Cypress has really good tutorials and a wide community to help. It’s generally very user-friendly. When I started learning it 1 year ago, there was no one around me who would know it; I relied heavily on googling :slight_smile: On the other hand, then I saw a video with one automation engineer from a pretty large company. They used Cypress for a few years, then they’ve decided to switch to Playwright. For them, parallel test runs were very important. Also, their QA automation teams were basically QA devs, so they enjoyed the fact that Playwright was designed by devs for devs.

1 Like
4

Here is the link to my blog post
Test Automation Tools: Cypress, Playwright, TestComplete Comparison Example

5

Great blog post! The spreadsheet is also nice and thorough.

I think it’s really interesting that you essentially listed the tools you’re deciding between in order from “least open” to “most open” (with all of the associated pros and cons that come with each of those things).

My 2¢ would be - overall, for an org just getting started with automation, Cypress may be a good “middle of the road” solution balancing the pros and cons of completeness of solution, community, supportability, cost, etc.

1 Like