Have you got any cache based testing stories to tell?

We have another great article from @restertest. This time itā€™s a guide to working with browser caches with some suggestions of tools to help with managing cache:

Browser Cache Basics For Software Testers

The article is fantastic for those who are not totally familiar with caching, but it got me thinking: What caching stories do people have to share? As a tester I had endless issues with CSS caching issues when the site HTML would change but the CSS wouldnā€™t load. It drove me insane!

Do you have similar stories to share?

2 Likes

I donā€™t have particular stories (or details) to share/tell, but I recall one could also hit caching issues during testing of a website or web application with respect to the server/cloud side, like server side caching, CDN caching. That might also relate to the CSS caching if itā€™s an external CSS file imported into the page (via stylesheet linking), possibly similar for javascript code/file caching when the code is also externally imported via script tag on the page. Sometimes the cache isnā€™t purged/updated properly or takes time to propagate (for CDN).

And how you might need to try to force refresh the cache update via HTTP headers that you send in the request (or API call).

Also on server/cloud side with DNS caching and load balancer not properly switching over to new/proper servers behind the load balancer and thus routing to the wrong servers to handle the request.

2 Likes

I agree that the term ā€˜have you cleared your cache?ā€™ is up there with ā€˜it works on my machine?ā€™ from the development lexicon :slight_smile:

My approach to caching has changed over the years, but general rule of thumb is:

Open new browser - Start ā€˜Testā€™
If there are issues experienced during test execution - Attempt to replicate on existing browser. If can successfully replicate then i would then test on incognito browser (or different browser completely).
This is only so I can provide more context in any subsequent ticket raised to development.

1 Like

As someone who has done very little web app testing, most of my experiences of cache errors are me as a end user or watching another team in the company get it wrong. For some reason cache errors seem to be 90% of web site defect panic level bugs, and it completely perplexes me why that is. I thought CDN was supposed to sort this kind of problem out. Does everyone have broken CDN, or does everyone have CSS issues because files are really being managed on different sub sites that donā€™t join up? Iā€™ve attached a recent one. I had hoped to get a reward for helping them uncover the issue. The support tech was blown away by my report detail, so I had to explain, Iā€™m a professional tester in the end. So thatā€™s my most recent horror story, but there are more.
carparking
r

1 Like

Have you got any cache based testing stories to tell?

ā€œIn computing, a cache (/kƦŹƒ/ (listen) KASH)[1] is a hardware or software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere.ā€

Debugging user issues that involved a lot of caching combinations was painful at times for me:

  • a/b test feature flag cache;
  • user auth session cache;
  • content/data cache;
  • reservation ā€˜cacheā€™;
  • browser auto-fill/autocomplete ā€˜cacheā€™;
  • client data form save cache;
  • page navigation URL parameter save cache;
  • forwarding of data to an external provider and using it when the user returns to the app;
  • external systems cache dependency + real-time stock check;
  • server firewall resources caching and user id mapping to server;

Weā€™ve had caching implemented everywhere: APIs, external systems, internal servers, browsers (local, session, URL, cache, state storage), databases, CMS, etcā€¦

2 Likes

I disagree that caching is a good choice for test automation. To reproduce issues it is always needed to have a ā€˜green fieldā€™, a browser with a new profile, free of any things which could change the result. The rest of the article was very helpful.

I donā€™t have a particular story but from my experience, I can just share some advice from the QA perspective. Part of my testing of web apps is to evaluate the caching approach used by the app - server, DB caching, and CDN, to improve performance. I check if the app uses cache effectively, checking it doesnā€™t introduce bugs or disrupt UX. If caching is absent, I always recommend its implementation because in many cases itā€™s crucial. Best practices are using CDN for static resources to reduce load times and implementing a server-side cache for frequently used API methods to improve responsiveness. The goal is to find a balance between performance and up-to-date content delivery. And remember about memory caching (e.g. Redis), it stores data in RAM for quick access.
Tes with and without cache to understand the impact of cache on application performance.
Use tools to monitor cache hit rates and the effectiveness of caching.
Test different cache configurations to find the optimal settings for performance and resource usage, if possible and applicable.

The web application Iā€™ve worked on for the last 10 years has caching issues. Boy, does it have caching issues. Adding to this, itā€™s written in classic ASP and hosted on IIS, which means the IIS server-side caching comes into play.

Oh, yes, and we have (had) a customer base that takes ā€œchange averseā€ to nightmarish levels. And is not terribly tech-savvy.

Long story short, whenever weā€™ve done a UI refresh, weā€™ve had to send all our customers instructions on clearing their cache because when this thing was first built it included a lot of CSS and javascript which - of course - caches. For a few of the critical pages, thereā€™s a timestamp thatā€™s used to force fetching a recent copy, and a few others use a version parameter that will fetch a more recent copy if the current asp page version doesnā€™t match the version of the javascript page. Assuming that IIS wasnā€™t being ā€œniceā€ and caching all the CSS and javascript for us.

It got to be routine with testing that Iā€™d try the new feature, and if I didnā€™t see it or it didnā€™t work right, Iā€™d log off, clear cache, and try again. If that failed, Iā€™d do something else for at least 1/2 hour in the hope that IIS caching was a problem.

It was only if those three tactics failed to show the new feature that Iā€™d ask the developer to please check that everything made it to the server.

Iā€™m so glad more recent versions of IIS arenā€™t as eager to hang on to their cache.