Hermetic testing to me is a method for ensuring that tests run consistently and offer a consistent value. If we apply the idea of Hermetic Testing to only the data element of a tests requirements, we can understand what this means.
Imagine there are three ways to handle data in our testing:
- Our tests can create their own data and then test that data
- Our tests could expect data to already exist, and choose something valid at random
- Our tests can expect specific data to already exist, and use the exact same data everytime they run
The third bullet point is how I imagine Hermetic Testing. Test-001 expects [item] to already exist, it navigates directly to the UID of [item] and performs the test.
This is a more scientific method of testing when compared to 1. and 2., because they inherently have risk built into them.
- In approach 1, if the data creation step fails, the test fails without even reaching the point it’s meant to test.
- In approach 2, if the assumed data isn’t present or changes unexpectedly, the test again fails for reasons unrelated to its actual assertions.
- In approach 3 (Hermetic Testing), the framework pre-seeds the required data in a known, stable state. As a result, tests should always run the same and in theory only fail when there’s a real bug.
This, to me, is the purpose of Hermetic Testing: to build a regression pack that executes exactly the same way, every single time, so that when a test fails, we can trust that something really broke.