Is it that bad practice to use Azure DevOps pipeline as test runner?

[mtest]

We use Azure DevOps pipeline to set up test system on self hosted agents.

Basically, we have a computer, set up azure agent on it, create a pipeline to run some scripts on that computer to set up a test system. We don’t generally clean the system (unless we have to).

I was told that an agent should be cleaned up at the end of the pipeline.

So I am wondering if this is a bad practice.

In a nutshell, we are using Azure DevOps as a glorified cron job.

We decided to use Azure DevOps since the agent is very convenient for setting up permission for executing scripts, network permission and etc.

Thanks.

[sebastian_solidwork (Sebastian Stautz)]

It should be cleaned up if leftovers can cause trouble in future runs. What can you imagine in your context?

@mtest

(unless we have to).

What is this situation? One where tests break for whatever reason?
How fine are you with having it unexpectedly at times you are under pressure?

Also it’s not a binary question, if or not, but also to what depth.
I once had it to the extreme that we made a snapshot of virtual machine in a clean state and always reset the VM to that state at the beginning of the tests.
(The product was changing also the Windows registry which was not easily to cleanup by a script)

This is another lesson:
Do the cleanup better upfront if doable. Because when your pipeline breaks for whatever reason the cleanup is not executed and the next run gets tainted.

Think of potential problems in your situation and what might me appropriate actions.

[mtest]

Thanks. That gives me something to think about.

[kristof (Kristof Van Kriekingen)]

Welcome to the community @mtest !

@sebastian_solidwork

This is another lesson:
Do the cleanup better upfront if doable. Because when your pipeline breaks for whatever reason the cleanup is not executed and the next run gets tainted.

Not always, you should make it idempotent so every time you run your pipeline and let’s say something breaks the “job of the pipeline” stops, there should still be a separate job to do the cleanup.

@mtest

Basically, we have a computer, set up azure agent on it, create a pipeline to run some scripts on that computer to set up a test system. We don’t generally clean the system (unless we have to).

So having a local agent is all fine, but having it on a regular computer might give performance issues in the future. (Or something happens to the computer)

@mtest

I was told that an agent should be cleaned up at the end of the pipeline.

I think that’s a bit context dependent, sometimes it’s nice to get something out of the cache to speed up your process.

[sebastian_solidwork (Sebastian Stautz)]

@kristof

@sebastian_solidwork

This is another lesson:
Do the cleanup better upfront if doable. Because when your pipeline breaks for whatever reason the cleanup is not executed and the next run gets tainted.

Not always, you should make it idempotent so every time you run your pipeline and let’s say something breaks the “job of the pipeline” stops, there should still be a separate job to do the cleanup.

Here it depends on the context, the details.

Even that dedicated job can fail in its execution for different reasons.
And in some situation I do not want an immediate cleanup. E.g. we have automated checks on multiple servers run once in the night and keep them like this to maybe do some tests during the day. It’s also a way of preparing our test servers with data.

And sometimes the cleanup might depend of what was installed on the server. Then a cleanup is better afterwards.