API automated cloud tests

Hi,

I’ve got a question about a test automation idea a friend of mine and I have.

We want to do a platform that runs tests against production APIs regularly (every minute/hour/day) and send notifications if something breaks. The tests are just series of requests with variables between them (e.g. list all users, then get the details of the first returned user, expect the result contains an email field and status is 200).

The tests are written in YAML, but with additional features like shell commands and templating.

You can update the tests from the CI/CD pipeline when you release so that they are up to date with the code in production.

The tests can also be ran locally e.g. during development.

The question:

  • Does this sound like something you/your team needs? I.e. will it solve any problems you have?
    • If yes, what part do you like the most?
    • If not, how can we make it better?

Thanks for reading! :slight_smile:

How would is be different from Postman or HTTP libraries implemented in any major programming language?

It’s not by a large margin.

  1. It isn’t an http client, so it’s not meant to be used to just make requests.
  2. It’s declarative, so you don’t have to write javascript for your tests.
  3. Mostly, it’s a hosted service - so instead of running the tests yourself in your DCs (or public cloud), it will run them for you

How granular is it in making requests & validating responses? What can you send & validate? HTTP headers, status code, request/response payload, dealing with JSON vs XML vs plain text payloads. File upload/download? Redirection requests, HTTP (basic) authentication, cookies.

You can send anything you can write in plaintext, so XML, JSON, graphQL.

It can validate headers, status code, plaintext & json payload (with the option to validate just the schema, not any primitive values), we’re planning to add gRCP & graphQL.

It doesn’t support file upload/download but it won’t be hard to add it (not sure how that’ll fit in the cloud version though).

Redirects aren’t followed.

You can inline shell commands inside any field, so stuff like basic auth will be something like Authorization: {% echo $USER:$PASS | base64 %}. You can save stuff that a request has returned. So you can save the header from one request and then provide it in subsequent requests and do cookies like that.