We’ve all had the dream at some point in our testing careers. We join a team, we see the test/automation frameworks and we want to scrap them all and start again.
Now imagine you get that opportunity, you join a team and you’re allowed to build an API framework from scratch. Obviously the technology you choose would vary based on the API.
If you had to build an API testing solution from the ground up, how would you do it and why?
I’m using an open source Concordion in Java (https://concordion.org/), as a unified test automation framework. It allows me to embed all my REST API tests using Spring framework. I also wrote an article on how to do REST API automation (https://www.methodsandtools.com/archive/restapitesting.php)
I think the question can be extended to cover test automation frameworks in general, doesn’t have to be just for API (testing).
It would probably have to be a compromise between personal preferences + team/company fit + management expertise with the chosen technology down the line based on hiring from market & existing team/company expertise.
Personally, having done such evaluation/research some time back, I would much prefer not to build a framework from scratch and use an existing general purpose framework for the job and customize it for one’s organization’s needs.
My personal preference would be to use RobotFramework where possible, for all its general purpose feature set & it’s ability to work cross programming/scripting language by way of remote interface for test libraries, allowing one to code test tooling for the framework in other languages besides python & Java. There are not that many frameworks out there that have cross language support.
Found a discussion on Slack about this, specifically related to a .net framework and the community had this to say:
used readyAPI - its great, they do rest and soap calls (I needed to support both). They connect to the DB for assertion, there is managed authentication which lets you handle complicated authentication (such as expired token or different rights to different users) . The learning curve is minimal and there is a lot of support from their team when you get to do more complicated things. You can start simple by developing tests using clicks and move on to more complex testing steps and assertions using groovy (or other languages that they support).
@mwinteringham added
the very excellent @g33klady built a C# version of my api framework model that you can check out here: GitHub - mwinteringham/api-framework: A group of templated API test frameworks based upon this series: http://www.mwtestconsultancy.co.uk/category/api-web-service-testing/
@gwendiagram offered
RestSharp is super cute http://restsharp.org/
I’d write it in code and it gets the developers more involved and feels like everyone is involved in testing and can see the results. I’d write it in C# to ensure the developers were still involved as well.
I feel this is applicable for any test framework development. These would be my consideration:
Language: Any language where my QA and dev members are comfortable.
Tool: Mostly open source and active project.
Put as much light wrapper on original automation tool, as too much wrapper may cause issue for someone new joining the team and not easy to maintain. A light wrapped method easy to maintain and users feel more connected. People joining new can immediately contribute. Over wrapped framework people choose not to use after the creator leaves the company ;).
It should be CI compatible and must run tests as soon as possible. The idea is to run tests early. Framework can mature with time with refactoring and adding new methods. Remember, our aim to have wider automation coverage not “The ultimate framework”.
Framework development and test addition need to have a fine balance. Where the 2nd items take bit more priority as it serves business priority.
@heather_reid - One important feature for me is creating mock services. This is useful when you are dependent on another service which is not yet available. Turns out that ReadyApi has that feature.
I would avoid building an api framework from scratch unless commercial products can’t meet my needs. Ideally, I’d like an open source product which can automate many things - api, desktop web, mobile, mobile web etc., handle reporting and more. So, in theory, we could potentially tweak the OSS framework to meet any custom needs. But, the downside is lack of dedicated customer support.
I’d bet that frameworks and their components are essentially repeated across many companies, i.e. reinventing the wheel in cases where a generic wheel would suffice.
It looks like the Karate framework by Peter Thomas (Intuit) is a unified framework which will reduce or eliminate the need for custom frameworks in many companies. Apparently, in Karate, you can swap out components like web auto (selenium), test framework (testng), reporting tool (extent reports), CI (Jenkins) etc. with the tools of your choice. I’ve wanted to try Karate, but just don’t have the time. Anyone here used it?
My 2 cents:
The problem with custom frameworks is that you need a highly skilled & dedicated team to maintain them. So, you are heavily dependent on that team & you have to smoothly handle departures of crucial members. If your company cannot offer as much growth, variety & pay compared to, say Amazon, then those engineers will leave you soon. You might not be able to replace them easily or with someone good.
OTOH, the challenge with commercial tools is that they try to please many customers. The advantage is that you might get a test tool which has features that you never even anticipated you’d need (thanks to the large customer base). But, the downside is that the tool might be developed to suit bigger customer’s needs which might not match yours. Bigger customers are often laggards, wasteful, bloated and bureaucratic i.e. a graveyard of ideas & dreams. Their requirements might take the testing tool backwards.
PS -
One suggestion to all companies who make their own automation frameworks on their own or with the help of consulting companies (often of questionable intent). Please try to make your frameworks open source, the earlier the better. This might encourage its developers to be more careful about their code. Those who consistently produce bad code will be called out by the public on github and will (hopefully) shy away from the automation team or maybe from even interviewing with the company. Does this idea sound good? Should it be a separate question? The downside is that good candidates who underestimate themselves or don’t want public attention might shy away from the interview/team.
I decided to go with a custom framework for several reasons: