I’m working on a new project in the airplane industry, that consists of organizing testing in the “Move To Cloud” and I wonder if there are volunteers who want to share their Return of Experiences ?
I wonder if anyone’s of you worked on a testing strategy that address topics on how to move to cloud and what’s different from on-premises ?
I didn’t work in such context before, so I’m searching for information what are the challenges when moving to cloud ? what impacts can we have on the testing strategy ? and how to follow them :
The areas I’ll be interested in the most are :
tools/ test automation : exploring how to verify they can still work in the cloud? is it preferable to converge to the same tooling set between various teams ? any difference on choosing which ones should be the standards ?
test environments : how many environments should be considered ? for how long should we keep history of all the history, logs continuously ? or at which frequency can we clear it ?
e2e testing : what are the differences ? how to reduce maintenance cost?
test data management : what are possible risks ? how to ensure data anonymization process is reliable?
any ideas, videos, blogs to read would be very appreciated
In my experience, it depends on the software being tested. When I was involved with a move to cloud project for a web application, the key parts of the project were:
Making sure all parts of the system worked exactly as they did when locally hosted
Stressing the system to check for any performance issues since the move included changes to the routing of connected web applications and services
Testing that load-balancing arrangements did not cause problems with user sessions, since we were moving from single host for each web application to multiple load-balanced servers
Testing that the only access to test environments was through the organization network
Determining through testing/exploration where we needed to add exceptions to the cloud platform firewalls. We started with the firewalls in their standard web application configuration, and added exceptions as we found they were needed.
Since this involved web-based software, test automation was simply a matter of pointing the automation to the correct URL, so that didn’t bring up any issues.
For history, we started keeping everything to get a handle on how much and how quickly we would be accumulating data, then started to determine how much and what to keep - it depended a lot on each web app and how much use they received.
We made no other changes for end to end testing or test data management - instead, we worked to ensure that communication between our corporate environment and the cloud-hosted software occurred over dedicated VPN tunnels, and added IP limiting to ensure that only machines in the corporate building could connect with the cloud test environments - or perform data transfers for deployments to the production environment.
A lot depends on the nature of your software - there will be different requirements for desktop apps hosted in the cloud than there are for web apps, for instance. The security requirements will also matter a lot, since they will impact access for your team to test. Similarly, the criticality of the software will govern a lot of your choices: if the software is for flight path management there’ll be a much higher level of security and safety required than for displaying trip information to customers.
You will need to learn as much as you can about accessing and using the tools provided by your cloud environment, but beyond that it will depend on the software and the risk tolerance of the organization.
For me it wasn’t that different than testing stuff that’s deployed on-prem servers, it was almost a seamless transition.
The only thing that I had to get used to was that sometimes for cloud-hosted environments you need to start them up, as often they get scaled down automatically to reduce cloud costs, as often it’s billed by minutes used.
From previous experience there are a few answers here which are dependent on your use case, resourcing, goals, objectives etc. However, a few bullet points for consideration;
Security implications / penetration testing
Disaster recovery / failover
Costs. Aside from test data there are many other configurations when dealing with and hosting data in the cloud which will impact your decision making for cost, data and environment needs, for example; Hot vs cold storage, ingress/egress, storage account regions, replication (ie LRS/GRS). All this, plus more, impact into performance and cost considerations
Performance is an area of testing to consider a meaningful approach, moving from on-prem to cloud can impact this at times significantly (good and bad!) including scalability.
Authentication - user authentication and testing of integrations between applications or 3rd party services (it was mentioned above but I have seen tests fail in this area due to firewall/VPN/IP errors or missed requirements/steps)
Not a comprehensive list but some areas to think through. Good luck, sounds an exciting initiative!
Thank you everyone for your helpful ideas !!! I hope in the coming future I’ll have something valuable to share back with the community as my return of experience
Great questions and considerations. Here’s my 2 cents to the challenges and considerations you should have or other things that should be considered in that migration process that will impact testing:
CI/CD - this may be affected depending of your deployment process, so consider if there’s a learning curve around use of the infrastructure. Like @mirza mentioned you may have had a different strategy when on-prem vs now being literally in the cloud .
IaC - Ifrastucture as Code might be a new area to consider as it impacts your project. Now being “in the cloud” has it’s benefits but does mean there are a larger number of components to manage so a foundational understanding of the cloud infrastructure would be key. Yes we get the benefits of increase in deployment speeds, cost reduction but it does mean it impacts testing if not done correctly.
Observability - a key area that aides in the testing would be logs, traces and metrics. How logs were accessed previously for instance directly on a server is now with a cloud hosting provider, which means querying those logs would be different possibly; Azure for instance use KQL (Kusto Query Language), events being stored or even customer data retention might be limited to 90 days. This shift in architecture might mean having to support both existing logging and new logging metrics during the transition so something to think about.
If i think of anything else ill be sure to reply but others have done well to add wider considerations!
.