Testing As An External Team - Need Opinions

I once worked on a project like that; not only did we work with remote devs, but the project specs were drawn up by consultants and the person who had commissioned the consultants had left the company.

We worked to a sort of agile process, whereby deliverable product was thrown over the wall once a month. We had weekly telephone conferences with one of the devs to decide bug triage issues and not much else. There was no discussion of “what is this bit supposed to do?” In the end, we made it work by various devious ways, though there was a lot of foot-dragging by the devs. But in the end, they delivered the last sprint, it all fitted together, we did integration testing and demo’d it to senior management. The CEO declared it to be “the best-tested software we’ve ever had.”

Then it got rolled out to beta test - except that the marketing people were going around, bigging the app up, and the beta test very quickly became a rollout across our whole customer base.

Then it all fell apart as we found:

  • Customers wanted to use the app in ways we had never anticipated
  • No-one had specified which legacy systems the app was supposed to interface with to ensure that data hand-off to the legacy systems would work
  • No-one had looked at the changes required to the legacy systems to accommodate new data types and features that the new app was offering clients for the first time.

One of the legacy systems that the new app wouldn’t hand off data to was the invoicing system. At one point we had about £3 million worth of invoices tied up in limbo between the app outbox and the invoicing system. You can guess how that went down with senior management.

At the very least, I would recommend thinking very carefully about this idea, and I endorse Kate’s ideas in the previous post. Have management said what possible advantages they can see from conducting all their testing from inside what effectively sounds like a locked room?

3 Likes