Recently we moved our CRM to a cloud-based Salesforce implementation. (It wasn’t easy.) Now we are dealing with this new world where upgrades to the software are coming, we don’t control them, but we want to sufficiently test in our non-production environment before the prod drop comes. Their release notes are massive and I’m struggling to advise the group on how much testing is enough. “Test everything” is not an option.
My question is: do you deal with cloud software with hard new-version install dates? Do you test key features + key workflows, or have you gone to “make the automated tool test everything”? What do you feel is the risk level?
Kinda losing my mind here, your thoughts appreciated.
My last company used Dynamics CRM very heavily for business rules processing. We had a team whos whole job was managing that application and impacts of change. When a change was coming that would affect my product we would get it into the backlog, create necessary stories and deeply investigate the change to understand its impact. Then testing would be based on that understanding. That testing might consist of identifying cases (automated and manual) to run, or identified Exploratory activities. This is also how we handled incoming changes from other services within the company that we depended on. For example: another team was responsible for the email comms processing that my product used. When they had a change incoming we would go over the change and identify how it would impact my product and organize test and change stories based on that. Of course it was never as smooth as I make it sound. Often the change communication was late “Oh yeah. forgot to tell you we are making this change to the XYZ. Its going into prod this week.” (primate screeching noises ensue) But the key was SME’s understanding the scope of impact, identifying available resources (cases, tools, activities) to leverage and planning/executing from there.
One thing we did do that helped a lot was across the company we got into a habit of including other team SME’s on Pull Requests. I was included on a lot of those. Usually it was a matter of looking at the code and noting how risky it was “Hey let me run some tests on this branch before merge” or “Low risk change, I think we are ok. anyone disagree?”
I dont know how much that will help. when Salesforce is jamming an update you might not get much warning. But hopefully some of my thoughts might give you a direction to go in
There’s not much to find on testing SaaS ERP systems, so thanks for bringing this up. You have a whole range of things to consider: new stuff from the vendor, updated stuff from the vendor, your customizations, and how that is affected by the updates from the vendor. I can see I wrote about this first in 2012…
What matters to your stakeholders is key here. Some companies scoot this over, others (regulated ones) want every little option checked. I have often been successful in not explicitly testing the “Out of the box” (OOTB). Though if we find errors they will be reported. And then we focussed on our own updates and critical paths - which might indirectly cover things in the release notes.
Making an agreed test strategy is the path to success, as it highlights the decisionmaking and priorities. You cant test everything - you wouldn’t be done before the next version would come out
