30 Days of DevOps Day 25: Feature Flags

Opening the final week for 30 Days of DevOps with a great team discussion starter:

Does your team use feature toggles/flags? Why and how are they implemented? If not, discuss what using feature toggles could do for your product quality.

I remembered reading an article from @jamesespie that mentioned feature flags and while I’m out of the game a little while, it helped me get to grips with feature flags and how/why they might be used so I thought it might be useful to share

I also found a blog post on the Testim site which has a nice summary of feature flags

Some excellent replies to @jamesespie on LinkedIn about this when he asked:

For me, my current question is… use a pre-built tool like LaunchDarkly? Or roll your own?

@mot2 says

I kind of think that feature flags are (in web based situations) one tiny sliver of functionality around routing requests received by the server. That is, based on a single boolean flag, requests are either rejected, or allowed. I am always a real big advocate for expanding the routing to be much more dynamic and flexible than that. Imagine a routing framework where any number of variables can be used heuristically to route calls to any desired back end implementation. For example, AB testing: route user id’s ending in even digits to implementation A, route the odds to implementation B. Deploying a super risky breaking change: only let users with id’s ending in ‘123’ use the new implementation, ie. only 0.1% of traffic, when it fails, set it so that no traffic can hit the new implementation. Maybe you want to try out a feature for just one given client, or one limited geo-location, or just one user, or just between certain hours of the day. Once you have a well implemented, well managed, dynamic routing strategy implemented, you can use it to really help drive feature delivery, risk management, testing, and even engagement with users.

So to answer your question James Espie, I think you’d be best to roll your own solution to ensure that your particular context is innately baked into such a framework, but most importantly, the solution needs to simple and fast, and should not pollute the feature implementations. Ideally it could be part of the reverse proxy? As long as you make sure the information you need to perform the dynamic routing is available in the request (or highly performant cache)

@testerawesome shared:

Piers Chamberlain talked about the feature flag in a recent virtual meetup event. He explained it well and here is the recording https://vimeo.com/416882792

Ankur Sheel says:

Like all questions it depends. Rolling out your own gives you more control and has no recurring cost associated with it. But, it requires a lot of effort to get it to a decent place… Also it might not be user friendly if its going to be used mainly for a/b testing, canary launches.
The inverse is true for pre-built.
Also this calculator might be helpful

Baremetrics: Build vs Buy Calculator

We mainly use it to hide features which are not functionally complete/ ready to release but still have the code in prod so we are rolling out our own. Also the main consumers are devs so it make sense to put it in git and version control it. But its tough to put dedicated time into building it with all the other stuff that needs to be done so keep that in mind if you go down the roll out your own path.

From Twitter we’ve also got:

Oisín with a home grown implementation
https://twitter.com/oisin/status/1264943383799365632

@marie.drake with a thread about why they use them and the possible downsides
https://twitter.com/mcruzdrake/status/1264931550245855238

@mot2 with a possible alternative suggestion
https://twitter.com/VincentDirks/status/1265076331189989377

@jsandell with an important takeaway
https://twitter.com/sandell_johan/status/1264943679019593729

@schumitza with examples of how they are used in their organisation and reiterating that important note to remember to clean them up
https://twitter.com/schumitza/status/1264944117458575362

@guna.petrova with some examples of how their company used them in the past

And @merrua had examples of when to remove them and how to manage their removal

https://twitter.com/merrua/status/1264961680183889922

From Slack, one user said:

If we need to analyze something, we use experiments (a in-house solution) where we have metadata like when it started, etc. (We do our analysis in our DW.)

We try and remove experiments/feature flags ASAP though tbh that’s mostly aspirational. But usually when things are quiet or we have a tech debt sprint. We’ll just go through and purge any old ones we can. We are a mobile app company too. So we try and bump min versions pretty often.

another user followed this up with

We have a massive amount of old experiments/feature flags because we bump minimum required versions very rarely. https://eng.uber.com/piranha/ this looked potentially interesting though

Nice reply here from Pete Goodman too:

1 Like