Running API Tests on TFS 2015 Build Process

(Alastair) #1

Hi guys,

Sorry for the really specific question - but I’m wondering if someone on here can help me out.

I’m interested in running Integration Tests using TFS 2015 automated builds. What I’d like to do is to set up an automated test which runs against a deployed API.

In my head I see it like:

Build starts - unit tests run - APIs deployed - integration tests run against deployed API.

I know we can mock/stub etc for unit tests, but what I want to do is check that behaviour is correct against a deployed API, but I want it to happen automatically alongside the build process.

Does this question make sense? Am I going about it the right way?

Thanks for any feedback.

(Kate) #2

Short version: yes, you are going about it the right way.

Some caveats: the TFS2015 build process can only build one application at a time. If your tests are built with the API, this shouldn’t be an issue. If not, you may need to do this as one or more builds, with the last one setting off an automated deployment of the API (using the automated deployment process), then running the tests.

A few other things I’ve uncovered as I worked through this process a while back:

  • The build process assumes that your directory structure has the solution directory then each project in the solution has its own directory. If your solution(s) don’t follow this pattern you may have to do interesting things with your paths.
  • The build process does not handle linked projects in a solution. When you’re building with Visual Studio and add an existing project to a solution, you get a link to wherever your project is relative to your solution. TFS2015 build does NOT recognize this and won’t be able to find your linked project.
  • You need to make sure your source control directory structure is properly referenced or you will have problems.
  • You must have Visual Studio and your test framework installed on the build machine. If this is the machine that will be running the tests, that’s all you need. If another machine is running the tests, you will need to have the test framework installed on that machine as well.
  • Every machine the build service/deployment service talks to must be in the same subnet and all tasks must be run by the same user. The user must be a local administrator on the machine(s).
  • Make sure you export your test results to a location that will not be cleaned (deleted) by subsequent test runs. You may need to adjust permissions to do this.
  • When you are getting source, make sure to get it from the correct directory. The build service will pull everything in and below the source directory you designate.

I went through this setting up distributed load tests to automatically build and run using TFS2015/2017 build services. Everything is working well now, but it took me a long time to work out how to make the system do what I needed.

(Alastair) #3

Thanks - this is really useful and I’ll need to look into your comments.

Another question - is it possible to deploy the APIs to the relevant environment, have the integration tests run, then have them roll back if the integration tests fails?

I’m really new to this stuff - so again these questions may be simplistic/not make sense.

(Kate) #4

It’s possible, but again, you need to have all the permissions set up.

You’d need to use the result of your test cases as a trigger to determine whether or not to remove the new build and re-deploy the prior build. You might need to experiment with this one and possibly have a series of chained build and deployment tasks.

(Cody) #5

Hey! Interesting use-case. @katepaulk gave you some really good advice so I won’t expand too much on that :slight_smile:

I was curious, what do your integration tests look like? Are they scripts that you’re checking in? Or are there some features built-in to TFS, VS, or similar, where you can create your tests and just run them during the build?

I’m not too familiar with TFS deployments, so sorry if it’s an obvious answer!

(Alastair) #6

They’re mainly API test scripts written in C# in Visual Studio.

Just now they’re separated from developer’s code, but we’re wanting to move our integration test projects into the same project as the code itself.

Basically, in TFS you can set up a build process that looks a bit like:

Build Code - Run Unit Tests - Deploy Code - Run Integration Tests

You can set up ‘rules’ etc too so if any of the tests fail then it will roll back (failing the build).
It identifies unit tests (in our case) by looking for any file in the solution with the word ‘test’ in it. If we’re wanting to run our integration tests after deployment we would need to ‘tag’ them as something different.

This is brand new to me - but it’s pretty cool.