Hi,
I embarking on a journey of trying to reduce the number of defects we have to send back to our suppliers.
Now I know this is a double edged sword, not only to we have to work closer with the Supplier to monitor their testing and agree what reports they should produce. However, some of the onus is on us, as the customer, to feed back better information on the defects we find and provide good evidence, to help the suppliers do their job in rectifying any issues.
Has anyone already trod this path ? What is deemed to be an âacceptableâ % of defects to be delivered from a supplier. I hear cries of âNONEâ!!!. How have others worked with, monitored supplier testing. To make sure the quality and coverage is there?
As you can tell or maybe have guessed ď I am going through defect meltdown in phase 1 of a transformation and am eager to improve things for phase 2.
Any anecdotes, suggestions, lesson learnt and implemented. Would be most welcome
It could be of course that the answer is blowing in the wind
Firstly a few words on metrics. A number, or ratio, or percentage of problems isnât a good marker of quality. It could be that 1000 typos does not outweigh one login issue (Iâm talking about software, but the principle applies to anything). Also an issue for one person might be a non-issue for another - it depends who matters. Letâs say we pick someone who never wanted the software and said it was a bad idea from the very start - this person will pick apart anything they think is or might be a problem and report it as more important than it is. I realise that this doesnât give you a nice, neat KPI, but it does mean you donât have a fake KPI. Rarely is there another kind!
You also donât have a number or percentage of defects, you have a number of defects found, which is dependent on where and how hard and how efficiently people are looking⌠and if they would know a problem when they saw one. You might be finding problems because your supplier has no idea that they are problems. Look at your reports and see if the problems you are finding are to do with being fit for purpose.
Next, the onus is on your supplier to do what you ask of them, but, yes, youâre right, you need to work closely with them to ensure they are building whatâs right for you. One way to see the problem (Iâd expect a software house to do this, but theyâre not all created equal) is to identify what kinds of problems youâre finding in the software. Are they mostly business critical? Are they mostly usability? Are they crashes? Are they platform related? This will give you a better view of the situation. If youâre swimming in bug reports on typographical errors maybe they donât really matter (maybe they do, context is everything). If the systemâs regularly crashing then you may need to have words with a supplier about why theyâre supplying something that crashes all the time. If youâre paying for something to solve a problem, and it doesnât solve the problem, then the thing youâre paying for doesnât work. If it solves the problem but itâs a bit clunky and dull⌠maybe thatâs okay. You and your supplier need to communicate your expectations and what can be achieved.
Happy to answer more questions or add more if you have more details about the problem :).
Hi Chris,
Wow thanks for the detailed reply. As soon as I wrote %, I was going to backspace as knew it was not a measure of quality. But let it ride.
Guess the crux of the4 new problems I find myself facing, is that my new company, the customer, take the view of âaccept the code in then see how good it isâ. I think that we need to get a better feel of what the Suppliers are testing, how they are testing etc. Also, what do people in a similar position do re monitoring the supplier while they are testing.
I may be wrong, but I keep getting this niggle at the back of my head that we are paying the supplier for delivery, but at the same time testing their product for them.
I was no around when the statement of work was written, so there is no mention of âsupplier testingâ and how we as a customer get a view of this.
The defects are by and large code related, there are a small percentage of configuration issues. However, we get caught up with the usual âitâs the requirementsâ discussions between us. It is my first time dealing with external suppliers and while am trying to forge closer links in the testing area. Am wary of not letting the tail wag the dog. Hoping others who have cracked this problem, while still maintaining a relationship can advise on how they did it.
I may be wrong, but I keep getting this niggle at the back of my head that we are paying the supplier for delivery, but at the same time testing their product for them.
Welcome to the bleeding edge of the New World Order. For years mankind has written about fascist dystopias and inescapable hells born of accidents, destiny or a seizing of power. Nobody predicted that weâd build one ourselves, our faces etched with a twisted, plastic grin, smiling at the construction of our own downfall lest we upset someone with our frowns. But I hear itâs very popular, so what do I know?
I havenât solved this problem myself, and I may be missing the full picture, but if someone sells me something that doesnât work Iâd take it back for a repair/replacement. Enough of those and Iâd feel it time for a refund. By that point Iâd be angry. Just saying. Iâm sure thereâs a diplomatic way to ask âwhy are we paying you for something that doesnât work?â. Maybe itâs part of the discussion âby what point do you predict this will be usable?â, or âare there normally this many problems this far into the project?â. Maybe itâs part of the discussion âweâre worried about the number of problems in the software, and weâd like to examine your test strategy/processes/procedures.â. If youâre paying then youâre in the position of power, and these questions sound totally reasonable and understandable to me. Software is an investment. Asking risk-based questions of your supplier is protecting that investment, and clearing up communication problems.
Of course I might be wrong - maybe this is your first run-in with a heavily agile project that is frequently releasing minimum-viable software to you and the software is basically in alpha. Maybe itâs a way to get great feedback to build the perfect software. Or a way to say theyâre doing that while giving you bad code. Either way, yes, youâre testing the software for them, which is sometimes not a Bad Thing - the question is are they doing their fair share? It might be that they are not communicating their expectations to you very well - maybe the softwareâs not supposed to work right now, itâs just to give you a feel of what it might look like in case you say âthis is not at all what weâre looking forâ, as opposed to âthis table doesnât sort properly when I click on the little arrowâ. If theyâre worried they wouldnât tell you. Youâre worried and you may not have told them. But youâre paying, so Iâd go knock on the door. Bring someone tall.
The balancing act that I foresee is them coming back with the âhmmmm, documentationâŚâŚ. It is agile you knowâ, counterbalanced with us needing some metric as to what they are, have tested. Sight of what they are testing and the evidence to back that up. With deep regret, I find myself in the land of poorly written contracts, weak requirements and a never ending cycle of code deliveries. Well that was phase 1, Phase 2 kicks off in Feb, hence my push to get things into a better shape (as well as picking the brains of others who have been there done that).
The code deliveries are meant to be (were meant to be), code releases as opposed to a monthly code delivery, but that is a discussion for another day.
Do realise, this is a double edged sword, in that the onus is also on us as the Customer to deliver requirements that are fit for purpose, so that the Supplier can supply the software for for purpose. To that ends I have instigated show and tells, before they move into System testing, so at least we can try and catch and misunderstanding around what the requirement was.
Well what you want is communication. Documentation is one way, a frank conversation is another - preferably with audio and preferably face-to-face. Maybe you donât need evidence, maybe you just need someone to clear up problems. Iâd find out if I needed to worry, then, if I did, I could start citing evidence. When you ask for evidence youâre saying âI donât trust you, prove youâre not as bad as I suspectâ - but you can build that trust with conversations and honesty. Unless they act suspiciously, and then asking for promises and evidence to continue receiving your money is, I think, pretty reasonable.
If you want evidence and they donât want to give documentation then ask them what they can do to make you more comfortable. You want reassurance and youâre the client, and you can go elsewhere armed with experience. Thatâs how I see it, anyway.
Going to throw in a âsecondedâ here on the communication. If you are receiving software with bad defects (errors in money, errors due to mishandled data, crashes, etc), then they are not spending resources on testing their stuff. In which case you want a weekly conference call (or video, or in person) meeting to say, hey, here are the outstanding problems, where are we at?
Likewise, if they feel itâs on you, that you arenât providing good requirements or that your bug reports donât have the info they need, then they should assign a business analyst or test manager to you to check in with you weekly or every other week to say, hey, you told us this, we need to clarify.
Developing a relationship with SOME representative there will help you get across what you need and earn you a champion in their camp. (Though you could get someone bad, unluckily. It happens.)
So glad I asked the question now. Such useful input. Think the weekly meetings is a great idea as is their reviewing our requirements. From looking at docs from the first phase of the project there seems to be a lot of âwe thought you meant thisâŚâ. You are right, the time for those discussions is before the testing starts!
Backing up whatâs already been written here. Working with a rather uncommunicative remote team, we first of all set up a weekly conference call to triage bugs and address other issues; then, when the product had been deployed but still had problems (not all down to the remote dev team, it has to be admitted), we ended up negotiating that their product manager would travel to our office and work remotely once a week, meaning he was on hand to troubleshoot/advise. He wasnât necessarily working on our product 100% of the time he was in our office (though in practice he often was, at least in the early stages of that phase), but just having him on-site improved communication and delivery.
When relying on a remote/separate team to deliver a product it can be very hard to separate bugs from miscommunication - however if you work on the latter then the former will inevitably become less common over time!
Remember also that being one step removed from the user will often result in important context being missed - what makes sense to the supplier may be entirely useless to the client, and the clientâs description of a feature can bear little relation to how it is interpreted.
Metrics can be useful and communication is essential. But Iâd suggest thinking about what is needed for your receiving organization to make progress with what the supplier delivers. For example one criterion for a successful integration can be is it good enough for system test, or for some part of system test? The answer supports the decision to assign resources to the next phase. In your case itâs your organizationâs work. This should be a common understanding with your supplier.