We had a bug recently in one of our pipelines such that errors were being swallowed rather than stopping the pipeline. So it might be worth having some way of testing your pipeline[s], to check that they really will stop if there’s a problem in step X (if that’s what you want).
Also, what level of detail is useful, to whom? Do you need e.g. a list of all the unit regression tests that have passed or failed, or just a summary of how many there are of each kind? If you summarise, how can you get the extra detail if you need it, e.g. to investigate build failures? We summarise in the main output, and write the detail elsewhere, and put something in the summary about where to find the details.
Who needs to get notified about what, when and with what info?
I guess some high level advice is: the pipeline is a software artefact in its own right. So it’s probably worth doing at least some of the normal software development stuff (as much as is helpful for you):
- What are the requirements for the pipeline, who decides on what they are and their priority? Does this priority reflect the effort and risk involved?
- Are there dependencies between the different bits of work that will influence the order things get implemented?
- Is stuff under source control, is it reviewed, is it tested (see my point above)?
- Do you need a separate development environment, or is it OK to tinker directly with production? If you have a separate development environment, how will you address risks of (not) making the same changes to production?
- I’m sure you can think of other things I’ve missed.
If this sounds like your needing product management for the pipeline, with some kind of backlog and owner, then yes, I think you do. If you don’t do this explicitly, then it will be implicit in people’s heads and probably inconsistent as a result.
So, for instance, you might have requirements that appear to pull in opposite directions, such as: must have fast feedback, and must run loads of tests. One way of resolving this is to throw lots of hardware at the problem. Another way is to have more than one pipeline - a fast one that’s run often e.g. as soon as a commit happens, and a slower one that runs if the first one succeeds or overnight etc. I’m not saying which solution is better for you, except that in general having the pipeline’s requirements explicit and agreed upon will help manage expectations and minimise grumpiness.