[SOLVED] Help needed with this one API test that's been driving me up the wall

I describe the problem in this Racket Help needed with automating an API test to change password | Racket

But I’ll use text here as well.

Problem:
Decreasing confidence in the account API that we use from the mobile app. There is no backend dev in my team, which really sucks. The tests in the account API seemed kinda bare bones so I figured I’d expand the tests.

A specific API call that has been giving weird responses back is the call to change your password. This is the test I have trouble with to make it reliable.

What I did and thought of

  • the existing API tests are made with Postman. First, I made a very ugly set of 2 tests that were depending on one another. In Test 1 I change from password A to B and in test 2, which has to run straight after, I change it back from password B to A so the account is back in the original state. BUT: that wasn’t possible because you can’t change your password back to the last-chosen password (security reasons I guess) so then I’d have to expand this to a set of 3 tests who are dependent on one another and that just is a bad solution. A test should be able to run reliably and not be depending on 2 other tests.

  • The way to work around this is making me run into the limitation of Postman. You can’t persistently store data in Postman as far as I know. To fix the test and remove the second/third tests I would need to store a randomly generated password somewhere so the next time the test runs it can use that again. But in Postman I don’t know how to do that. You have the Environment Variables, but as far as I know these don’t persist over sessions. I did some Googling and didn’t find an answer. Is there an option here that I’m missing?

  • A total different avenue would be to create a new account every time. But, a new account has to be confirmed via an email or in the admin console of SAP…I have no access to confirm account via an API (only manual action in SAP Gigya CDC) —> someone from the SAP team is going to find out if they have API’s I can use, but I don’t have an answer as of now. There could be a way to work around this by using something like mailSlurp so you can access emails with an API, but it’s a paid tool…

So, now I’m thinking I could also create my own backend service to store this damned password as a variable so I can keep using postman OR, I’ll just make a NodeJS project with the API tests because Postmans client is sometimes making me want to pull my hair out. It’s so easy to end up with code duplication because it’s not fully an IDE but not “just a GUI” either. It can get messy quickly I just want some neat helper methods and I like my code clean.

I don’t want to use mocks because the problem is in the chain so a test with only mocks wouldn’t really add value for me.

Screenshot of “the thing I need” in Postman


I’d prefer to have one variable “password” that first gets filled at passwordCurrent, THEN somehow generate a new password, store that in the variable and use it in passwordNew. I just don’t know if this is possible in Postman. In pure code project this would be trivial.

Thoughts?

3 Likes

Hi Maaike,

Honestly, the test environment seems to be the problem but, as you’ve alluded to, this isn’t something you can change.

It looks like you’re testing the “change password” feature which requires the current password.

Is there also a “forgotten password” feature that you could use to set the account to a known password as part of the test setup?

2 Likes

The whole point of this test is to mimic a user journey so I’m going to test both forgotten password and change password.

The forgotten password flow is going to be a different nightmare because that again requires to click on a URL in an email. Don’t want to tackle that just now.

1 Like

Also, would definitely recommend using a code-based test framework.

If you’re comfortable with JVM I’d suggest trying REST Assured.

Fair enough. Although, if you have to be able to do this to test the “forgotten password” functionality anyway the code can simply be reused.

I prefer to use Javascript when it comes to talking with REST API’s that use JSON responses. Saves me some headaches by not having to (de)serialise JSON all the time.

For now I prefer to stick with Postman if I can because the other people in my project are using that, but if Postman keeps giving me issues I’ll switch to a NodeJS project I think.

1 Like

Are you sure that the variable you set in an evnironment like this is not persisted?

pm.environment.set("password", my_random_password);

Not using postman as a tool for automated testing, only for manual api testing, but in my experience, when I for example get a token for authentication and save it in the environment, then it stays there until I get e new one, even after restarting postman.

3 Likes

Dumb question probably, but are you using pre-request scripts?

This way you could avoid some of the dependencies between tests, by scripting some of the preconditions and firing them up before you send the request.

yup, can’t live without 'em.

2 Likes

Aaah I messaged you @maaike.brinkhof on racket but I’ll write it here also for the rest.
I don’t know why your environment variables are not persistent or what you are doing to them but…:

So eventually you’ll want to run these tests within a CI/CD pipeline + you don’t really want hard-coded passwords in any script or variables or even in version control.

For example Azure DevOps:

  • Create 2 pipeline variables Password1 (current) and Password2 (new password)
    **(set them as a Secret also!)
  • Start the pipeline and generate a new password (GUID or something) and set it to Password2
  • Run your first test with current password as Password1 and change your users new password as Password2

** your tests run **

  • If your test has passed, set Password2 as Password1 (which is the currentpassword of your user)

On your next run you’ll create again a new password for Password2, etcetc.
From a security standpoint this is also a nicer way of working since even you will never know what the password is and there is no hard-coded passwords in Git or anything.

This is our current setup :stuck_out_tongue:


Others options:

  • You could also make an SMTP request in JavaScript in order to test your create user flow also and you can run your test in isolation each time.
  • Are collection variables an option?
  • Can you reset password?
1 Like

Pretty sure you tried this and for some reson cannot do this. But for other people in this place:

I do a email login test for most 90% of tests. If you have office365 integration in the company, this might help

  1. Test must sign a “new” user up using a special testing mail-box and you want to get IT to create one mailbox for you. Call it perhaps mobile.test@companyname.com and hard-code the password to something pretty nasty.
  2. when creating user account in the actual app suffix the mailbox with random_string(len=12)
    mobile.test+ABCDEF12345GHT@companyname.com . Notice how we use the username+randomshit@company.com pattern here, it works with most email providers these days, not just O365.
  3. generate a random password now as well,
  4. save the random name and password in the suite-setup stage , to save doing this for every single test-case
  5. use whatever language your tests are written in, to access the office368 api to poll for the password e-mail sent matching that specific mangled name. 60 seconds polling it enough, mostly it takes 5-10 seconds. I don’t know enough Java to know why the global or environment variables did not work.

The 5-10 second hit to create a new user account is only incurred once in the suite, and once whenever a test case also needs to get the system to email you again as part of the test, because you keep the same username and password for the suite. I’m assuming that creating an account on backend is also instant here. But honestly, it takes about 10 seconds in our system, mileage may vary. It’s a bit evil to completely E2E test, it’s slower, but is my personal preference lately.

If you are having real smarts, you can

  • create the account via other means using api calls, but dont wait for it to complete
  • Start the app installer on a thread
  • wait for the email to come back on another thread

Once every few months have someone empty out all the accounts in the system called mobile.test* - LOL, forgetting to will only make your DBA angry at you.

1 Like

I solved it! Danny Dainton was kind enough to jump on a short call with me and from him I learned that with the online version of Postman you can persist variables. Combine that with using the test-tab as post-req tab and you can create the loop you need.

Let me show you.

On Collection level, create 2 variables like this, of course name them however you want.

Then, in the pre-req tab I had to login so I had to use the “changePwCurrent” variable. I use it to login (first green circle at line 14). I stop the script in the function starting on line 17 if login fails for whatever reason.
Then at line 29 I generate a new random password and store that in the other Collection Variable called “changePwNew”.

Then, I have to update the body of the actual call to use the 2 variables like this:

Screenshot 2021-06-23 at 20.32.36

Lastly, on the Test tab I have to update the variable “changePwCurrent” to get the contents of “changePwNew” as this is now the current password.

Now we are actually at the reason why I made this test. For some reason, the API returns a 500 but the pw does get changed. Weird, so this needs to be fixed. When the bug is fixed I’ll update this code a little bit. Only in case of success (a 200 response) do I want to update the variables. Else I want to throw an error and not update the variables.

Conclusion
This felt a bit like playing 3D chess, this whole thing would have been much easier in a full-code solution. But oh well, I learned more about Postman. Since my whole department is using it I’ll play along for now, but I’m not discarding the idea of moving to a full code solution.

5 Likes

I was late, but good that you found a workaround for this.

Regarding persisting variables in Postman, if you’re running it in the Collection Runner, you can enable a checkbox that lets you persist the environment variable between runs. Otherwise, there is another approach apart from the one that Danny mentioned (didn’t know about it btw), but it might be more tedious. Anyway, just in case anybody wanted to know this, you can use the Postman API to retrieve and persist environment variables between runs, useful if you want to run a flow of chained requests in a monitor or with Newman and need to persist information between them.

I use this approach in a Postman tool I made, just in case someone wanted to have a look:

https://www.postman.com/alopezari-team/workspace/learning-assistant/collection/1558965-71c39b5f-b9e5-45c4-9748-2edb8bb98f0a?ctx=documentation

2 Likes

You should write an if that checks if your request worked (by http code for example if == 200) and then set the collectionVariable.
Otherwise if the request fails you’ll still have set the next password while it’s still the previous password and it will fail the next login you do.

@kristof

I already wrote that in the response: When the bug is fixed I’ll update this code a little bit. Only in case of success (a 200 response) do I want to update the variables. Else I want to throw an error and not update the variables.

But right after posting it here yesterday I was too annoyed by it and fixed it. Once the “it returns a 500 error when it shouldn’t”-bug is solved I’ll remove the first statement after the if, of course.

My testset now consists out of +100 tests. And mostly, it’s just calling an API and verifying the contract. But just like you i’m also running into the limitations of Postman.
When testing an OIDC flow, I learned about the sending requests in the Pre-req script. It helps, but at that point I also started thinking about a full code solution.

I was wondering, do you run your test with newman on a server as well?
And are the variables also stored in that way?

Variables are not persistent with newman and if you are running it from a server or CI/CD pipeline then you want to do it this way: Pipeline Variables for Postman

What you can do if you run them locally through newman is the following:

newman run coll.json -e env.json -g globals.json --export-environment env.json --export-globals globals.json

You basically resave the variables by exporting it again. Source: HOw to update the environment variable or global variable with collection results when fired with newman command line · Issue #831 · postmanlabs/newman · GitHub

2 Likes

The key thing to realise is that with the ONLINE version of Postman the variables do get stored! Which is really handy if you want to run some CI/CD stuff.

I created that part yesterday and wrote a blogpost about it
https://www.maaikebrinkhof.nl/2021/06/how-to-postman-tests-html-report-to-microsoft-teams-with-gitlab-ci-cd/

2 Likes

This is going straight away to my reading list!