Power Hour - Curious, Stuck or Need Guidance on DevOps or Observability?

Ministry of Testing MoT Power Hour
#1

On the 1st of May Abby will spend an exciting hour on The Club tapping away at her keyboard answering your questions related to enabling DevOps delivery, testing on cloud infra teams and how you can get started with Observability. Ask your question to get it answered.

Hi :wave: My name is Abby Bangser.

Aside from testing I love walking my dog in the UK countryside. I’m here to answer your questions about:

  • Enabling DevOps delivery
  • Testing on cloud infra team
  • Starting with Observability

Get all your questions in by 1st May before 7pm and I’ll do my best to answer them during my power hour!

7 Likes
#2

What is observability? And why should software testers care about it?

4 Likes
#3

Is there a difference between observability and automated tests? If there is, what is it?

#4

Do you have one key piece of advice for testers who are moving towards the DevOps space in work? It seems that there is a bit of a gap between people already there and people looking to be there.

3 Likes
#5

How do we know we are doing good in observability (success criteria)?

2 Likes
#6

What are the different methods for observability? What would be the first steps for improving observability?

2 Likes
#7

I came across the following article ‘Observability vs. Monitoring’. It includes the following quote: If you are observable, I can monitor you.

What is the difference between observability and monitoring? Are there benefits of observability other than making a system easier to monitor?

2 Likes
#8

How do you start thinking about an observability strategy? What are the overarching things you should be considering?

4 Likes
#9

How can you measure observability?

3 Likes
#10

DevOps is all about automating workflows, bringing issues closer to the point of development and stimulate global learnings. Where and how in your opinion should manual testing occur and how can the results from this manual testing actions become global knowledge for the whole business?

2 Likes
#11

What sets of tools can we use to get started with Observability?
Does tooling need to change as we become more agile as a team and experienced in DevOps?

2 Likes
#12

Thinking about DevOps and starting with Observability I have the feeling that developers and testers (and probably product owners as well) get new responsibilities and have to learn and various levels, which might intimidate them and cause objections.
How can I help the people to embrace a DevOps culture? Which positive aspects of this change would you highlight for the individual, the team and the organization?

2 Likes
#13

How have you handled performance/load testing in a DevOps context?

2 Likes
#14

How would you define what DevOps is, key terms to know, and what it hopes to achieve?

1 Like
#15

How would you say DevOps Testing differs from the normal testing? Is it time-based and more to do with the backend or would you work out some UI testing also into the mix?

3 Likes
#16

This power hour is coming at a very timely moment for me, thank you so much for organising this session!

Here are the questions I am trying to find answers for while I am in the process of transitioning from Agile(ish) to DevOps world and creating a continuous testing strategy that enables us to deliver quality at speed:

  1. What are the key principles for continuous testing in DevOps so that quality can be achieved at speed?
  2. What changes do these key principles bring to how Developers and Testers need to work? For example from Dev perpective, unit tests seem to be very important but it seems clear that it is not enough to tick off the box saying we have unit tests written if their coverage is not tied into covering the main risks. And for Testers working in factory test case cultures, a change to exploratory Testing methodologies is necessary and better skills in using and create tools to help speed up testing. Is this going in the right direction? And there other important changes to bear in mind?
  3. For me, these examples require fundamental changes to how Dev/Test need to work, but how can we influence that change of corporate culture ?
  4. How do you manage the transition to continuous testing and support the people when Dev and Testing skills are not yet there, while still delivering quality software?
  5. How can Dev/Test best work together effectively in a mutually supportive way in the continuous testing DevOps World? For example, in assessing risk, identifying what checks to automate, fast collaboration and feedback, good communication…
  6. In DevOps, automation is key for speed of feedback and throughput, but it’s only a small subset of testing. How can exploratory Testing be integrated into the DevOps pipeline without becoming a perceived bottleneck due to the non-automated nature of it?

I’m conscious this is a lot of questions and one hour is short! But if you or anyone here can also point to some good resources that can help explaining and answering these questions that would also be a great help.

4 Likes
#17

Heyya everyone! :wave: Hope you all have had a great day!

I am gonna get started now and see where we get to. I want to try and make sure to do a bit of chatting on devops and a bit on observability but definitely will keep an eye on new stuff coming in so let’s see how fast I can type… :sweat_smile:

2 Likes
#18

Seems only fair to start with the first and so far most liked question!

Observability describes how easy it is to see what your system is doing, but from the outside. When things go wrong in distributed systems, you can’t always jump right into what is wrong, you often need to first figure out where the problem is occuring. Being able to do this from tools that can see your whole system allows you a lot faster diagnosis than needing to connect to specific servers and potentially having to recreate the same kind of query in each case. A good rule of thumb is that if you need to ssh into a server, observability could be improved. But even more than that, if in order to answer a question you need to add new data fields to your logs, this too means observability could be improved.

So why should testers care about it? Observability is in essence the testability of post release testing by giving us the tools to be more creative and productive with our exploration. And why should we care about post release testing? Well first of all, we release to many environments so this could mean better support in dev and staging environments. But, of course, it also means empowering our ability to understand production. It has been said that we all test in production, just some of us listen to the outputs. This really hits home how there are always interesting new behaviours uncovered by real users under real load and we all should be listening for them. Wanna chat more about testing in production? Join the (aptly named) slack channel #testinginproduction at ministryoftesting.slack.com!

#19

Worth getting a good definition out there for DevOps as well, so…

DevOps is often defined as developers and operators of software coming together to stop throwing changes over the wall and instead work together. While this is often a byproduct, I have found much more success when thinking and talking about it in the sense of devloping and operating software. The true outcome of DevOps is when the people who are writing new features are also answering pages for production incidents. This means that people will be writing their software in a way that it can be run safely and debugged effectively.

Some key terms you should get familiar with revolve mainly about how you delivery working code to your end users. This includes Continuous Integration, Continuous Delivery, Continuous Deployment and dark deployment techniques like canary releases, feature toggling, and rolling deploys. But some other key terms come from how you develop your code and these include strangler approach, expand and contract development and infrastructure as code (arguably the influence of development on operations) and observability (the influence of operations on development).

#20

FUD (fear, uncertainty and doubt) are all very real and challenging feelings that we all get sometimes. In my experience these can only be overcome when very individual and personal challenges are supported.

So…what can you do to help someone who is feeling overwhelmed (even yourself!) Find out what is a major (personal) challenge and figure out how DevOps can help them. For example…we had a security team member who was always being surprised by changes operations engineers were making by hand to the environment like port openings. We were able to showcase how infrastructure as code could (in the long run) keep a proper audit of what changes are made and as a secondary measure (and more short term friendly) we showed how synthetic monitoring would identify and alert of them of these changes.

But our poor security specialists always get thought of as behind. So another example for a Product Owner who was worried about if their newly re-written system could be turned on or not, we introduced traffic mirroring for weeks to watch as the old and new system behaved given the same production traffic. This built up confidence that the new one not only was working correctly (after some bug fixes of course!) but also that it could handle the load of production.

It is great that you have already started to identfy some of the different stakeholder groups as well (so anyone who hasn’t…give this a try!). Just look to apply that same problem identification issue at all levels. I have been ranting and raving about improving our metrics for deployments to help our team streamline more. But I also know that the CTO has little to no interest in that while she is very concerned about outages and MTTR. Both are valid, just different view points :smiley:

1 Like