Should I choose Karate or Playwright for API automated testing?

[jean-l]

I’d like to choose a tool in order to build about 150 automated api tests.

For now I have prepared 10 of them with Postman but for bigger needs (running often, CI…) it could cost more money.

I have seen that Karate is relatively easy to use but maybe Playwright offers more flexibility.

Basically my need would be about :

• Manage about 150 tests easily (running, maintenance, folders…) → maybe both are fine
• The easiest to understand and maintain
• Be able to run the tests in different environments
• Pass tokens and variables from a test to another one

I have watched many tutorials and stackoverflow pages but sometimes there is not a clear answer for these questions

If you have any suggestion it could help me a lot ! thank you !

[utchbe (Butch Mayhew)]

I’ve had great success with Playwright for an API only test suite/framework, with 1382 checks across around 300ish endpoints.

I’ve also built out a lot of tests through this blog series covering basic and complex scenarios - The Definitive Guide to API Test Automation With Playwright: Introduction against the Restful Booker Platform APIs that are used here - https://automationintesting.online

The series of blog posts covers all your topics, but they are long and may take some time to get through

[jean-l]

Yes indeed I found your guide yesterday and I’d say thank you very much for this job that you did for the community

I haven’t found the part where you cover the topic related to multiple environments, like how can we decide to run on on dev or staging environment for example. On google people have different approches but it’s not very clear.

Thank you !

[kristof (Kristof Van Kriekingen)]

@jean-l

For now I have prepared 10 of them with Postman but for bigger needs (running often, CI…) it could cost more money.

You can run everything in Postman for free? I run thousands of tests in postman for free in a few minutes - they also run every night & deploy.
Do you use newman to run tests or the postman UI (collection runner)?

@jean-l

I have seen that Karate is relatively easy to use but maybe Playwright offers more flexibility.

I think Playwright is the current Hype + also from Microsoft, so I’m imagining if you use Azure DevOps, in the future there will be more out of the box support for it, maybe even standard integration. Huge community behind it, which is also important.

@jean-l

• Manage about 150 tests easily (running, maintenance, folders…) → maybe both are fine
• The easiest to understand and maintain
• Be able to run the tests in different environments
• Pass tokens and variables from a test to another one

I personally still went with postman for this, just to have everything in 1 tool (for exploratory testing sessions). Postman supports all of the above, and basically any API testing tool, so no need to worry about these specs.

I’m not trying to push Postman here, but what do you feel comfy with AND your team?
Are there non-technical users? Does everyone want to use a Full-Programming framework?

@jean-l

I haven’t found the part where you cover the topic related to multiple environments, like how can we decide to run on on dev or staging environment for example. On google people have different approches but it’s not very clear.

Everyone always will have different opinions on how to do X or Y, you’ve just got to find YOUR way to do it, to see what’s best for YOUR organization. I would compare the things you’ve found, and tailor it to your specs & ease of use.

Playwright - How to work with Multiple Environments!

[jean-l]

@kristof
For now I only ran collections from the Postman UI, but yes probably I should consider Newman

In addition, I’d say 75% would not feel very comfortable with coding so Playwright could be a problem, but when I see examples on internet I saw that the code part is very low. I mean when you have some examples with assertions like “verify status, count, check a field in the response” then it’s almost always the same thing. Probably the setup it’s the hardest part.

[kristof (Kristof Van Kriekingen)]

@jean-l

For now I only ran collections from the Postman UI, but yes probably I should consider Newman

Very big yes towards this. I believe the Postman UI collection runner is restricted to 25 runs monthly (free). And Newman is unlimited + WAY faster.

You’ll love it

@jean-l

I’d say 75% would not feel very comfortable with coding so Playwright could be a problem, but when I see examples on internet I saw that the code part is very low. I mean when you have some examples with assertions like “verify status, count, check a field in the response” then it’s almost always the same thing. Probably the setup it’s the hardest part.

I feel ya but the writing the assertions is 1 thing. In full coding frameworks, you’ll write methods/functions and work more OOP compared to a low code tool like Postman.

Postman is a good start to get the hang of it especially if coding isn’t your thing (yet).

[utchbe (Butch Mayhew)]

Ah I see I need to write about that! (Multi environment support). The key with being able to run your tests against multiple environments will be your test data strategy. If you have hard coded ids in your tests and test environment but those ids don’t exist in the other environments you will face challenges.