Our first break is done, just finished changing my baby’s diaper at a TestBash conference theatre (a life achievement that I was not previously aware of until I unlocked it) and I’m back just in time as Elizabeth Turner (my teammate!!!) is taking the stage.
This talk is about the experiences she had on the native team when it comes to hot fixes. “There is no problem in production, we have a new feature but it did not make the branch cut” That is not hot fix.
Every time you release, there is a risk to the user. No matter how “minor” the change. Let me repeat that for the folks in the back… EVERY RELEASE INTRODUCES RISK
So what is a hot-fix? What are some of the factors that leads a team to feel that it is safe to ship a hot-fix?
Definition: Small piece of code to correct a major s/w bug or fault, and released as quickly as possible
Okay, that is pretty ambiguous. What makes an issue major? When is it less risk to release super quick than not release at all?
And the “this is fine” dog makes his second appearance this year 
This is in fact a pretty good analogy for anyone faced with making prioritization decisions around risk when being pressured to release a feature under a hot-fix label. It is better to have a list of reasons or entrance criteria
- Is it actually a bug?
- If you missed branch cut, was it tested thoroughly?
- Did you avoid taking a proper disciplined approach?
QE… we often have a desire for perfectionism. Truth is that while everything “has a priority”… that is not the same as treating everything as a priority. We do have to consider this
We also have to deal with impatience of others. Everyone has deadlines and pressure. For a team facing a deadline, they ARE representing to you their highest priority. Even if this does not match our highest priority (avoiding bugs in prod)
Oh my, this old song: “It’s a really small change”. Everyone in QA has heard this and we all know how the smallest changes can have a massive impact. They can offer us a chance to focus our attention however
Pace… when things move super fast, that is when steps and precautions get skipped and mislabelled as strategy or mitigation. The reality is that we need to be able to properly explain the purpose of those tests in order to be able to properly represent the risks in the team discussion. Nobody questions the need to properly reproduce a bug before a fix is implemented… even if that time is expensive. How do we harness that energy and introduce it into properly deploying and testing before releasing when there is intense pressure on hot-fixes (not actually hot-fixes)? Deploy once is always better. Always.
What are we saying then?
For discipline, you need definition.
Checks and balances should be well understood and well respected as necessary components of releasing software
Definitions:
Minor Fix:
- How many areas of the software does it touch? (not what was changed)
- Who are the stakeholders?
- Can the issue be mitigated safely?
Major Issue? - What is the User/Revenue/Team impact?
- How is the user impacted by the bug vs how will they be impacted by the fix?
- Any security or legal issues? If your user’s personal info is at risk, can you afford to rush a fix?
As quickly as possible - how fast is this? - What is the platform being released?
- What is the typical release cycle?
"A SHARED DEFINITION ALLOWS TEAMS TO HAVE SHARED PRIORITIES" (This is a great keeper)
Wrap up:
DEFINE
- what qualifies as a hot-fix for your team?
PRIORITIZE - if everything is a priority, nothing is. It is just a pipe
ASK - understand all areas, thresholds of risk, impact before signing off
If you understand these, you can understand how to qualify a hot-fix
Final on-stage selfie for her first talk ever… can’t believe it. So natural. Great talk and crucial topic.