Our final Ask Me Anything of 2018 with the very talented @bangser.a on the topic of CI/CD and Delivery Pipelines took place tonight. As usual, we had a lot of great questions 
If we didnāt get to your question, why not ask it here? Maybe youāre catching up on the Dojo and have thought of some more questions or follow up ones to those answered by Abby tonight. All questions welcome!
Unfortunately my own internet connection was not the strongest tonight so I wasnāt able to be as much of a link ninja 
A relevant Club post to the discussions had tonight
Abby also referenced Gatling, you can find more about that here
0 bug tolerance
The state of DevOps
And the unanswered questions:
Hi all! I am around right now for a bit, will try and get some of the unanswered questions in but feel free to shoot some more over 
We talked a bit about this, but definitely techniques and culture. There are a few things which can make tools limiting (e.g. hem not being able to be run from a command line during the CI process) but there are WAY more things which can be culture limiting. Getting the team used to sharing not yet finished code is first and foremost. It can be really scary to show not perfect work. If you have a cut throat culture around mistakes this can be a very big hurdle. But it is a necessary one to get started! Even if you have the right culture, the techniques around ALL changes in source control (yes I mean DB migrations, yes I mean config changes, yes I mean the actually pipeline configurations) and being able to quarden off the unfinished bits using techniques like feature toggling and/or branches by abstraction can be hard and take practice. You donāt want to be trying this stuff when you are already going straight to production.
Start solving problems for people. People view some of these cultural and technique changes as scary and sometimes even posing a risk to their jobs. Combating this is arguments about how much it will help the company isnāt going to work. BUT if you can help someome get their work done in time to get home to their family or get recognition you have a shot. So basically try and stay away from fancy terms and ābestā practices and keep everyone as close to home as you can. That will let these communities grow organically.
Absolutely not, CD (delivery or deployment) does not speak to release. It is just as you describe, but for many people release and deployment are 1 and the same!
I am going to take āinfrastructureā liberally and include development practices. I would want to see that ALL changes that impact an environment are in source control and these changes explicitly versioned/labeled. I would also want to see either trunk based development or at least clear continuous integration. From there you are looking just at how you gain confidence in your changes, how much do you trust your automation? How well can you evaluate success of a deployment/release/live running of your system? How quickly can you fix a problem once you identify it? All of this needs to be evaluated against your appetite for risk.
You project/product architecture absolutely impacts your pipeline architecture (and it can go the other way too!). Your ability to trust changes to one component independent of all other components is a key indicator of how well you can follow CD. So, if your application is all a single repository and a single service, you can probably get a simple pipeline up and running (and trusted) very quickly. However if you have 10ās or 100ās of services that have frequently changing interfaces and are tightly coupled your pipeline will need to deal with that complexity as well and it may be more challenging to gain confidence without setting up some more structures for how these services will interact.
I believe that the very first step is identifying what is absolutely necessary to build confidence to release. This can be by looking at your current process and making that visible. But if you can get people to step away from current process and look at ideal you may end up in a better spot. Soā¦do you require a security sign off? Really? For ALL changes? Ok, maybe only for certain changes. Great, now you can make visible that changes to a certain repository or type of code will be stopped by a manual gate but other changes will pass right through.
So try and write out the name of the stages you have and what risk you think that they are mitigating (or if you donāt have a defined process yet, identify what risks you want to mitigate then name the necessary stages). Put this up in a visible place and maybe write on a different colour slip of paper some example of changes that could be made. See if people feel those changes could be trusted after passing through the pipeline. This could identify missing steps for your pipeline and engage people in different parts of the business.
I am not sure about the stop/start, but basically CI/CD (just like any sort of automated regression testing) is most valuable when something is under change. So, even if you often drop projects for months at a time, if you are highly likely to need to pick that project back up and make changes to it it may be worth while. This becomes even more worthwhile if you find that picking the project up again will mean needing to revisit the context and not being confident in what the behaviour of the system should be.
Always back to value. If these tests fail, would you choose not to deploy? Then they have to be part of the pipeline. Could some of them fail and you would still deploy but the quickly follow on with a fix? Maybe you can split these out. But if all need to run and they take a long time (honestly an hour is much better than many!), look at ways to kick them off in parallel either with themselves or with other things. Sure, usually you may run API testing before UI testing, but if UI takes a long time you may choose to run them in parallel so as to get the feedback faster.
Both of these techniques allow a code in the āmasterā branch (or any other shared branch you choose to deem the one that gets deployed to production) to be deployed even if some changes arenāt ready to be released. I usually advocate for feature flags (even though they can be trickier to get used to) because they allow more flexibility in refactoring as well as more usefulness in āroll backsā. So with CI, if you are running either of these you need to add in to your pipeline a way to validated the not yet released features. This may mean exercising your application in a different toggle state than production or running your tests against branches that arenāt master. But I think this is the biggest reason that these āWIPā techniques can impact how you do CI/CD.
If a pipeline is your only way to get a change to production (which as I spoke about is an ideal way to work) then any failures would mean you are UNABLE to update production. This means if a critical big comes in for you would need to fix your pipeline THEN you may be able to fix the critical bug. Therefore, it is prefered that teams ātool downā any time a pipeline is failing. To make this feasible you would need to be alerted immediately of a failing state through visible team dashboards as well as individual alerts either on your desktop, email or chat mechanism. However, if the team is not going to stop and fix things, or if the team is not even empowered to fix a broken stage, these alerts can be quickly fatiging and should not be done. You may want to read up on SRE alerting practices (for example in the Google SRE book or many articles around the web) to understand more about alert fatigued and how to create effective alerts.
I think this depends a lot on what you are trying to monitor. Are you in an on prem/pets kind of an environment? This can allow for certain tools versus if you are in a more container or even serverless environment. The next set of questions are around what feedback you want. Are you looking for time-series data like CPU usage over time? Or are you looking for more action created data like a release occured, or a log was written? Most people need both but the tooling is very different. Finally you need to look at how you may use the outputs. Are you looking for something to track your knowns? Or are you looking for something to explore your unknowns? Each of these point to very differnet tools so I am not sure it is fair to name ones here without more context.
Unfortunately this can be a tough problem to solve. What you need to evaluate is what you need from running thats service. Is there a set number of responses that you can mock out? Is there a contract you can trust? This is a great example where your application architecture can make a big impact on how complex your pipeline architecture needs to be. If you are able to siphen off your connection to external services to a very small and isolated bit of code you can stub much easier.
It may be controversial, but I actually donāt think all defects should require a new test in the regression suite. Some defects can be fixed from actually removing functionality or changing things in such a way that needs to be monitored for or a different risk mitigation put into place. So back to your questionā¦I think that the only things in the pipeline should be things that are absolutely required to trust a deployment. In the experience I spoke about, we did not need 24hours of tests to trust a deployment, but noone felt qualified or empowered enough to remove any of them without risking the finger being pointed at them 
So I stand very firmly on the ground of only putting deployment necessary tests in the pipeline which may or may not be things which have proven defective in the past.
The biggest risk here is that the mock drifts from realty. To mitigate this risk I look to run the mock against both the thing it is immitating AND the thing it is being used for. You can look at a tool like Pact to better understand how this can be executed.
Whew! Final one! So yes, I do think testing will evolve (potentially dramatically) but that is because of an expansion in our arsenal and not because of a lack of value. The idea of being able to double or triple the range of places where we can mitigate for problems is empowering. We can be thoughful about if it is worth immitating production load when we have the ability to just trial the new service under that load with low risk! I would highly recommend checking out what the likes of SREs and Chaos Engineering specialists are working on and come say hi in the testinginproduction channel in the slack. There are lots of interesting things happening! A great intro would also be to check out the talk from Test Bash Manchester by Marcel and Benny.
Hey!
Hope youāre doing great Abigail & Heather 
Am I still OK to ask two more questions? I missed my chance on the AMA session. I hope you wonāt mind 
Many thanks, and I appreciate if youād have a chance to look at my questions.
Jarek