Lies, Damned Lies and Statistics

Recently, I sent out the most basic of surveys to my company, with two questions - what department you are in, and a score out of 10 of the quality of what we produce. There were a number of reasons I did this. Firstly, it was to get a general feeling of where we’re at. Secondly, by repeating the exercise, so that I can get a feeling of the direction we believe we are heading (especially as I am making changes to process and want to see if they work or not). Thirdly, to make people ‘aware’ of quality. Obviously people are, and I’m proud of the work my company turns out, but I want to promote a conscious dialogue. I want people to be “aware aware”. Unfortunately the response size was a little on the low side to be confident in it, but it was an experiment and I’ll see how we can improve on that.

Sorry for the big paragraph, but essentially, I was wondering if other people had undertaken similar exercises, and if so, how useful they were and whether they had any advice. Thank you.

Hello @chris_dabnor!

I applaud your effort! I believe these surveys are useful in getting the pulse of an organization. Our organization surveys employees once or twice per year. The surveys assess workplace culture and results are shared with employees. The survey results provide inspiration to make changes or improvements in the working environment. In my opinion, they have had a positive impact.

Quality, however, is not part of the survey. It would be challenging to measure quality at the enterprise scale since IT is part of many other groups. We do assess quality within IT in two other ways.
First, quality is assessed by the project team during retrospectives. At first, I thought the assessment was little more than an exercise. What I learned was to give it time. As I worked with the project team longer, I came to understand what quality meant to team members and calibrated the response accordingly. As a Test Lead on one project, I published what quality meant to me and I believe it provided context when I responded to the assessment.
Second, there are a few objective measurements in place at an enterprise level. While there were some initial challenges collecting these measurements, I believe many groups now publish them for themselves and provide them for aggregation at the enterprise level. I also believe there has been a positive impact on raising awareness and actions inspired to improve quality.

My sense is that large surveys require some time to establish as a part of the ecosystem. I think you can get there with some perseverance and perhaps some C-level interest. I wonder if, while working on the survey, you might get a grass roots sense of quality by surveying project teams as mentioned above.



Hmm. Might be interesting to get that grass roots view. Maybe have an aggregated score from the POs at the end of their projects.

Like you, I’ve written an article on what quality means to me, and sent it out to the company. I aim to write one for external consumption, ie removing all references to us and what we do and replacing it with something more anonymous and generic.

Incidentally, what sort of level of response do you get? In my Big List Of Tasks (or BLOT), I do need to look at a way of increasing engagement.

Hello @chris_dabnor!

I agree with Joe’s endorsement of retrospectives. They have some important advantages over surveys if you can get participation from the right people:

  • You have informed opinions about a specific project/release.

  • You can follow up on the answer to a question or a comment, and get confirmation or elaboration by other participants.

  • Participants may offer useful information you would not know to put in the survey. For example, there may be a process improvement that relates to how another department operates.

In short, it’s a conversation among participants, including suppliers and customers, to identify what went well, what can be improved, and possibly how.


1 Like

The culture survey usually sees a better than 50% response; the retrospectives are closer to 90 and 100%.


Thank you. I think, according to my rough attempts at figuring it out, I have a confidence interval of 4.45%, which isn’t too shabby. However my confidence in my calculation is not much different…

We have a quarterly team health check using a traffic light system and run by the Scrum Masters where we are asked questions like how is our speed of delivery, do you feel like a pawn. There’s about 12 questions and each has a brief description of what red and green looks like. The scores are put up for all teams and if they have moved up or down for each one. Quality comes in in some questions like, are you proud of the software your team is creating.

1 Like