Any thoughts/opinions on whether it’s better for tests to have known values or if they should recalculate things, specifically for integration or end-to-end tests?
For example, let’s say you have an API that calculates sales tax. Using pseudocode for examples, would you prefer:
expectedTaxValue = "6.16" // 72 dollars at 8.55% rounds up to to 6.16
I lean towards calculations, as many of my tests lean more towards property testing, where the inputs are somewhat varaible (i.e maybe I use the same test to test the min, max, and a half-dozen random values in and outside the acceptable range). This ends up with my implementing some logic sometimes that’s in the requirements. Balancing act of writing a flexible test with diverse coverage vs. a test with known values. I think the strongest argument for the fixed values is that if you’re treating the test as documentation, it does a good job of making the requirements obvious (w/o someone having to read/understand the code).
My preference would be fixed values (I’m avoiding the use of the word “better”).
For the example you provided, the purpose of the test, in my opinion, is to evaluate the behavior of the calculation-under-test. As you suggested, the test explores large and small values, data type extremes, and a value or two more. I prefer a fixed value because this sampling of the behaviors would tell me if the calculation-under-test has changed. That is, I expect a specific response from the API and if it is different, I may have something to investigate.
Where I substitute a calculation for a fixed value, I may introduce more maintenance into my test automation: if the API changes so must the test. If this idea scales, so does the maintenance.
Ah, I guess the third option would be explicit as parameters, i.e. to test all the cases, you could feed in three params to the test, e.g. varying combinations of amount, sales_tax, and expected value (effectively making it data driven testing).
One of the strongest kinds of oracles are the reference one. I.e. you make the production implementation and a reference implementation of the same business logic. Something we are currently doing which then basically feeds the both systems the same request and then see that we get the same response from them. Using calculated values have a similar property you are in fact making a reference implementation of the logic. But the value of the reference implementation is that it can handle testing more kinds of scenarios. So if your automation are asking for the same thing all the time you do not utilize the value. Hence in that case static values might be better in the regards that it will reduce the human error factor.
So as mentions data driven testing where you can provide different data for the same test is in general better since they cover more cases as little extra cost is good. Also if your domain (the grouping of values) are business heavy like the tax has to be between 5% and 15% or the calculations are like if price is less than $100 -> normal tax, for any price between $100 and $1000 reduce the tax by 2 points for the part above $100 or something, then using generators is also a good pattern to have variance in input. Like priceBelow100(), validTax(), tooHighTax(), negativeTax() which produces a random value but with an expected trait might also be a good idea.
I personally also prefer the calculated value because the code becomes self documenting instead on relying on comments. With the caveat again that you are more likely to introduce an error in the test code that way.
