What were/are the concepts of testing and quality among Operations teams/Systems Administrators before DevOps?

#1

Before DevOps, how did operations team test infrastructure? I am interested in knowing about teams which did not use Infrastructure as code.

How do such teams test now, i.e., if they are using physical hardware? Suppose they are provisioning a server or networking device, what does testing mean?

#2

Same as today. By performing actions on the system and observing behaviors in the response. More specifically what do want to know. When we were working with setting up new systems on a regular basis we typically had a check list with checks to perform. Something like. Should respond to ping. Some command to list what services are running. Possibly health check end points. Logs are being written. In my experience DevOps changed two main things. We spend less time writing extensive operations documents in development that are then handed over to operations. And we have fewer operations specific tools as they are instead built into the system. Infrastructure as code limits number of errors due to some operator having changed something on the production system that cause the system to act differently in development and then causing production errors.

For infrastructure testing is typically a testing of the configuration. Testing of the design in terms of security and performance and resilience. Something on the line does all services have access to everything they need and only that? Is the design optimized for the expected traffic model? And does the system handle failures and can recover when it is repaired?

2 Likes
#3

Thanks for the response.

As I think through, I don’t think there is an explicit strategy for testing (as general known in this forum) in infrastructure. I think the focus is on creating a working system and running checks. This is probably the same as most other activities, which don’t have a ‘test team’.

#4

I think there are several available strategies to use when working with infrastructure. If they are explicit or not I leave to each organisation. With infrastructure as code a common strategy is to use what would be called contract testing among services. Basically you test the infrastructure by running the applications it hosts and test that those have all they need. The weakness with that strategy is that you might not test for capacity, reliance and security. On top of those capabilities your infrastructure commonly also solve logging and monitoring which as very testable as they are. I find thinking of your infrastructure as your platform that have certain contracts (we used service level agreements) as well as some common functions. It may be easier to identify what test strategy you want to employ. Finally infrastructure can have a substantial cost which means that you normally want to take as many shortcuts as you can which may lead to errors.
This reminded me of a weird bug we had back in the days where the virtualization was setup inappropriately causing the response times to shoot through the roof when using https since they all used the same crypto chip and that became a huge bottleneck.

2 Likes
#5

If you look back further, particularly in large companies, operations and development were in different organizational silos. Handing off a release was a big deal because it was less frequent and had more content and configuration changes. There was some duplication of work (and test systems) in development and operations because the bigger organizations had to work more independently of each other. This was not agile; plodding might be a better label. In my experience, the better managers recognized the limitations of the organizational structure and the processes it required. They worked hard to foster communication and collaboration because they were needed.

#6

@hattori I understand the drawbacks of siloed working. I was more interested in the testing of infrastructure. What does it mean to test a server? Was there any explicit approach for testing? Even with infrastructure as code, is there really any (explicit) approach to testing?

#7

nilanjan,

Here’s some more on your testing questions. I hope it helps.

George

It seems to me there are at least three ways to look at this. They’re related, and they overlap. They also align with typical roles in development/operations.

  • How do you test infrastructure: servers, networks, operating systems (traditional or mobile), databases, etc.? Typically this question is for the organization developing the infrastructure system.
  • How do you test/check that the integrated and configured infrastructure operates correctly in support of an application or service? Typically this question is for the organization operating the infrastructure system.
  • How do you test that an application or service works correctly with its infrastructure? Typically this question is for the organization developing the application and/or operating the service (end-to-end).

I’ve worked on projects where people had all three of these test roles, but I’m not expert in any of them.

  • In the first case, developing infrastructure, the testing varies widely on what is being developed. Testing a server is different from testing a network of course. But before you deploy or sell either of them, you need to test each with the other to meet customer needs. In addition to functional testing, there can be load tests, response time measurements, tests at performance limits, and more. I.e. the infrastructure needs to be tested before customers integrate it further.
  • Ola gave good suggestions for verifying infrastructure. And operations organizations can use spare or back-up systems (if available) for testing beyond development, reproducing service problems, etc. In my experience operations designed their own configurations and purchased or built their own tools.
  • The third case is the one I think most testers would encounter: testing applications running on previously tested infrastructure. The infrastructure enables the application for testing, and often the new build is the source of a fault. But problems also can arise from interactions with infrastructure.

Here are answers I’d suggest to some of your specific questions:

  • Before DevOps, how did operations team test infrastructure? In consultation with development, operations would acquire, integrate and configure the infrastructure. Application installation would be supported by development as needed.
  • I was more interested in the testing of infrastructure. What does it mean to test a server? It varies widely. It can depend on what services it supports, on the networks it uses, on the client operating systems, etc.
  • Was there any explicit approach for testing? Even with infrastructure as code, is there really any (explicit) approach to testing? We would start with requirements, which needed appropriate buy-in from the organizations on the project. We also wrote test plans. If the infrastructure was not developed inhouse, it would be acquired from a supplier. The supplier would be responsible for their testing and support for the component.