Have you ever come across this problem as a test manager?
Your organisation has procured an application which they plan to integrate into their technology stack and put live. Obviously after testing, of course.
Fortunately, a test environment is included in the commercials. After the initial implementation, the test environment remains to support the testing of enhancements and integration of peripheral applications.
Suddenly the is a problem which can only be attributed to the test environment. You report this to the vendor. If you’re lucky the ticket goes to the bottom of a ridiculously long backlog. You are told that Service Level Agreements (SLAs) are in place for the production environment incidents, but there is no procedures for test environments.
If I’m able to insert myself early enough in the commercial discussions, what could test environments SLAs look like? Has anyone encountered and resolved problems like this before? How can I ensure that long-term arrangements are being made to support test environments?
That’s a challenge - especially in the outsourcing (Saas?) situation. I have experienced similar issues with test environments: scale, uptime, integrations, and failure response time. One story from back 10 years was service windows (for the deployment of new code) on the test environment: daily at noon and between 5 and 9 in the evening. As the deployments usually hindered testing.
Towards management, I would argue that SLA for the test environment is based on the risk of not releasing the set date and chances of losing manhours prior to release. Besides that, it needs to be up and scaled perhaps only during “testing” business hours. Perhaps you can afford to lose one test day - so MTTR (mean time to repair*) could be based on that? Perhaps with specific guidelines on the peripherals and integrations?
Time to market plays a huge part, as well does the company’s ability to change from cumbersome on-offs to a more consistent and recurring delivery - including the vendors.
SLA is the product owners problem, not yours. It becomes a team impediment -all you can do is request the resource in your test plan. And then work hard to understand how the vendor’s test environment differs from live. Often the test environment needs to be on a newer version anyway, so normally “downtime” is not going to be a problem. If their test environments are flakey or often are down, then they will be unable to onboard new customers themselves. Always assume there will be some downtime, and plan for it. Note that test environments will perform differently to live, and will also not “scale”. You need to include these limitations in the strategy you come up with.
All the team can do is assume that at some point you might even switch to another vendor, if there is no plan to switch vendors, then your vendor integration will just add risk over time, and the way you communicate with the vendor will probably become worse over time too.
Thanks Conrad. You make some valid points.
However, as testers are going to be the people most affected (delay in progress of execution), I believe that someone with a tester’s mindset needs to make sure test environment availability is baked into the contract. For the duration of the engagement with the vendor and not simply during distinct test phases.
As an engineer I just stay away from anything that is commercial, it’s a politics problem. Keep a black book for every time when the service is unavailable, and get the contract manager to take it up with the vendor. Contracts are only as strong as the “time-to-repair”.
I completely think it’s wrong to say that “testers are impacted”, the “project is impacted”. Testing are a link in the chain, and part of the team. When one person on a team suffers, the entire team suffers. That is the definition of “team”. Never blame late releases on test blockages, that’s a planning failure.
I can see where being the test manager makes this your problem, but without the vendor account manager being in the picture, you cannot hope to address downtimes directly with the vendor. You personally have no leverage, get the person who has leverage into the room, and make in it their interest to push for better support.
My “testers are impacted” comment is probably a personal and emotional one. If my contribution to the team effort is impacted, even if it’s through no fault of mine, I take it personally and I would seek out long-term improvement opportunities.
Regardless, if I can do anything to ensure that continuous test environment availability, escalation procedures and SLAs are discussion topics during contract negotiations, I feel that’s value added. I believe that was you final point. Get the relevant information to the in the “person who has leverage” and is in the room.
I hope to never be in a management position. Im’ good at finding things, but bad at consensus. I respect every manager and try to help them as much I can, because it’s a tough job. especially when an engineer rises through the ranks and the company turns them into a manager, without giving them actual management support. Dev teams will also want access to the test environment whenever integrations need to happen or upgrades need doing, so you should not be entirely alone.
Best of luck Goke
/Jesper