How do you know what to test?

Iā€™ve put this question out to Twitter too, as the responses can be ever so useful.

When given something to test, how do you know what to test?

What guides you along the way?

What resources do you refer to to help?

Who do you go to for support?

Iā€™m asking this as I know itā€™s such a common question and it would be really useful to have an epic discussion on it.

3 Likes

Every time I test something, my starting point for knowing what to test (first) is closely related to another question: what donā€™t I know about this? This second question has many parts:

  • What donā€™t I know yet?
  • Do I know who knows? Or where to find the answer?
  • Can I use the application already to find out what I donā€™t know?
  • ā€¦

Once I have tested some, and gathered some answers to the above questions, there are additional related questions to ask:

  • What assumptions have I made at the moment?
  • Are there any newly discovered gaps in knowledge?
  • Is it already useful to have a second opinion on some of these gaps, or on some of these assumptions?
  • Can I give feedback already about something: how certain am I about some specific piece of functionality?

These sets of questions usually take me a long way already.

Resources I use:

  • OneNote (for notetaking)
  • Documentation (user/technical/ā€¦)
  • TestSphere cards (for defocusing, and sometimes to organize my mind/strategy)

Support:

  • Everyone in my team that I think can provide insight, or a second opinion, or guidance in prioritisation, orā€¦
  • Sometimes co-workers that are outside of the project (usually as a sounding board)
5 Likes

On a related note, I was thinking of this when I was just casually pondering that question one day: https://autumnator.wordpress.com/2015/02/23/what-the-fk-to-test/

Good starting point.

Iā€™d add:
What is that ā€˜somethingā€™ given to test. Can I see it? Can I ā€˜accessā€™ it? Does it have a name/description? Is it a thing, a person, a thought process or something else?
Whatā€™s the purpose for me having that ā€˜somethingā€™ to test? Is it to play with it, to give feedback, to be part of a lawsuit, for compliance, for my information, for auditing, for releasing a product, for firing someone of incompetence, for a product purchaseā€¦
What part of that ā€˜somethingā€™ is of interest to the ā€˜someoneā€™ asking me? Is it user facing, technology, development architecture, business workflow, optimization of testing, costs, efficiency, performanceā€¦

I try to identify things that are the same, or similar, and things that are different. Building off of what Sarah replied, looking at the existing application or a competitor application.

Are there different roles with different access? Do different users have different views? For example, an online menu could have a gluten-free version, which only shows the items marked ā€œgf.ā€ Or applying a senior citizen discount on the price.

Are there thresholds to consider? For example, first time user has to accept a EULA. Failing login attempt three times locks the access. Quotes over $100,000 require manager approval.

Are there any date implications? Does something expire in a year? If so, what happens then? Is the cost or refund prorated for a partial year? Is there a grace period? What is the date format? (International dates are stored DD/MM/YY, versus U.S. of MM/DD/YY.)

Covering CRUD - Create, Read, Update (Edit) and Delete. When uploading files, what if nothing changes? Does the app take the time to process all of the imported data if it doesnā€™t need to?

Are there any time-related processes? For example, report aggregates run overnight? Bills generated at the end of the month? Tax documents generated at the end of the year?

Boundaries. For numbers, consider null, 0, 1, 1+, fractions, negative numbers, whatever the limit is, and limit + 1.

Any special characters (like El Ƒino) in the text?

State changes. Toggling something On and Off. Do states persist or reset? For example, how does an expired account become active again? What value is used if something is toggled back On? Does it revert to default or was a custom value saved and is now displayed?

Looking at values in the database. Are numbers stored as strings? If so, how is sorting handled? (Strings would sort like ā€œPart 1ā€, ā€œPart 10ā€, ā€œPart 2ā€ā€¦) Are numbers stored at floating point? How much precision is retained?

So, someone has asked you to test something. What do they want to know?

They might want to know if the SUT still works after some code changes - might suggest testing some functionality.

They might want to know how the SUT performs - might suggest some load and performance testing.

Or the might want to know how usable/logical the SUT is - might suggest some usability testing.

Understanding what the stakeholder, that asked you to test in the first place, wants to know/learn from your tests, may help you determine what to test.

This is a very common question. So, I will share a standard practice that is followed by most of the experienced testers in software testing services companies. When a tester gets an application build for testing, he gets a User story associated with it. The same user story was referred by Developer to write the code for implementation.

Below mentioned points are commonly used by tester in order to fulfill the testing requirement:

  1. Tester starts working on User story point by point.
  2. Tester has to use his domain knowledge to deliver the quality testing delivery.
  3. It is suggested to test each and every web element available in application with different scenarios. For example, If there is amount field then use positive number, negative number, zero, float number, double number, Char, special char and etc.
  4. Tester has to test all positive as well as negative test scenarios.
  5. Tester has to verify the validations on web page.
  6. Tester has to make sure the new feature do not have any impact on existing features. He should verify the existing features related to the new change as well.
  7. Web Application should be tested in different browsers like: Google Chrome, Firefox and Internet explorer.
  8. It is also suggested to try cross scripting in text fields as well. For example, alert(ā€œHelloā€);

Hope this information is helpful for you.

This is an interesting topic to me because Iā€™ve seen variations of it asked on quite a few occasions, and in different contexts. I sometimes wonder whether people are just looking for new ideas and inspiration to keep expanding on what they already do, or if they really just donā€™t know what to test. To me, it often feels instinctual, obvious, implicit, and I need to remember that this isnā€™t always the case for everyone. For the record, that doesnā€™t make anyone better or worse.

Soā€¦ Thereā€™s a lot of ways I know what to test, or methods I use to find out. Here are just some of the ones I can think of right now.

When given something to test, how do you know what to test?

  • Experience as a tester
  • Experience as a user
  • Curiosity for what features, buttons, gestures, etc. might do
  • Knowledge of other bugs or common pitfalls, either in this software, this platform, similar products, my own usage habits, etc.

What guides you along the way?

  • Experience
  • Instinct
  • Curiosity
  • Documentation (official, user stories, bugs, acceptance criteria, project contract, etc.)
  • Claims
  • Assumptions
  • Other people (dev, PO, BA, designer, marketer, user, other testers, etc.)

What resources do you refer to to help?

  • Product documentation
  • User stories / bugs
  • Company / product mission statement, goals, elevator pitch
  • Marketing, support pages
  • Mnemonics and heuristics
  • Cheat sheets
  • Google!! (SQL injection, scripts, status codes, etc.)
  • My own blogs (sounds conceited, but sometimes I suddenly remember that Iā€™ve dealt with something before and dropped nuggets of wisdom that I can refer back to)
  • Strategies, presentations, resources Iā€™ve prepared or used for previous projects

Who do you go to for support?

  • Dev, PO, BA, designer, marketer, other testers, etc.
  • Testing community (people I know, Twitter, Slack)
  • Colleagues not on the project

So many thingzz!! I hope no one gets discouraged by experience helping so much. This isnā€™t to say that if you donā€™t know what to do now, youā€™re gonna miss a bunch of stuff - itā€™s more about taking lessons, tips, tricks, reminders, from everything you do and carrying that on into the future. Just because it was a previous project or three years ago, it doesnā€™t mean itā€™s not still relevant. Everything you do should help to build up your experience if you reflect, process and extract in the right way.

Reflect, process, extractā€¦ I like that! Maybe Iā€™ll write a blog on that to remind my future self :wink:

5 Likes

Hey all, I just published a new blog inspired by this thread. I hope itā€™s helpful!

2 Likes

Two things that will help me to know what to test:

  1. When I reveal the gap between what I know about the product vs. what product claims to be.
  2. Are there any other risks?

I follow a Risk Based Testing approach to know what to test. Testing based on risks helps to ensure you focus more on high impact/critical areas and give less focus on lesser important modules.
For example- Say you are testing 2 stories - Story A - Payment module for bank. If the payment module does not work properly it is high impact and there are severe consequences for that. And say there is Story B - deals with changing the font size of the website from 14 to 16. If the font size is not changed correctly, yes it is an issue but not a huge impact to the customer and business. The customer does not care about it.

If you are given 2 days to test the above 2 stories, it does not make sense to test Story A for 2 days and Story B for 2 days. Instead, you would dedicate about 3.5 days on Story A as it higher risk and probably 0.5 days or less for Story B. This is where Risk Based Testing helps i.e to ensure testers are giving the right amount of focus and effort in testing higher risk modules.

I use an excel sheet where I note down in different columns 1) The module I want to test 2) The risks associated with the module 3) Ask my developer to give a score of 1-5 based on complexity and technical aspects of the module 4) Then ask my business person give a score of 1-5 based on the impact to the customer for that module 5) Then ask a domain expert to give a score of 1-5 based on historical problems in that module. Based on this I calculate a Risk Score (Historical problem x (max of developer and business score),

So, in my above example the payment module would be a Risk score of 25 and the font size checking for the website would be Risk score of probably 2.

Based on the risk scores we clearly know what to test first and can prioritize our testing.