Hermetic Testing

Discussions
, ,
1

When you hear Hermetic Testing what does that make you think, what is Hermetic Testing?
Has anyone done it before?

(Ps there is no Glossary entry yet on MoT)

1 Like
2

I see this on the MoTverse about this:

Ask Me Anything about Hermetic Testing | Ministry of Testing

2 Likes
3

Frogs :frog: :frog: :frog:

I had to google this one :rofl:

So it’s basically testing if a lit is sealed properly.
Imho this is more of the “QA in food industries”

Nope :smiley:

2 Likes
4

We have integrated lots of third-party API in our product for different types of checks, and the majority of them are black box. We are not sure how their logic works, so usually, on the test environment, we mock those API to avoid their dependencies and continue with the testing as they don’t support sandbox for test environment. Couple of them support sandbox but that also only for staging environment. So even though this is not the best approach from my perspective but we don’t have much option in such cases.

1 Like
5

Without google and anything (cause I need to google this one) I will try to derive the definition right here, right now :smiley: and I know I will be terribly wrong:

Verb: HERMETIC: Her-metic
Her - Pronoun for a female
Metic - from automatic

Definition: Something that has to be done by a person recognised as (her) - automatically - lol :laughing:

What Mr. ChatGPT says about it:
“Hermetic testing is a software testing approach where the test is completely isolated from external dependencies such as the network, databases, or other services. All inputs and outputs are controlled, ensuring the test environment is consistent and reproducible.”

Interesting, will read more about it! Thanks for sharing! :ringed_planet:

1 Like
6

what chatgpt is saying, isn’t that what we used to call mocking or stubbing?

2 Likes
7

There’s quite a good definition in the overview from this 2023 TestBash talk. I’ve posted in the Slack Glossary channel.

8

Indeed. It’s a great way to avoid finding any bugs, and your tests are fast, simple and stable. What’s not to like? Your users are now going to be the first people to find those bugs, and you don’t need to pay them. This reduces your costs and delays the reporting of bugs till post launch when you’ve got more time to fix them. Is there no end to the benefits of this type of testing?

1 Like
9

Yup - but it’s not in the Glossary yet ;). I know what it is just curious what everyone else thinks.

10

@ajwilson

Hermetic Testing typically refers to an approach that keeps external agents from interfering in the test environment: no network calls, no real database connections, no file system dependencies, etc. Their aim is to keep tests fully self-contained and repeatable and deterministic.

Appreciate you sharing this really valuable insight!

11

We predominantly utilize hermetic tests, and honestly try not to over think it, basic unit tests are hermetic tests. It’s just a sealed or contained environment to test in. Which can be achieved by using fakes, stubs, or mocks.

We take a thing and we test it without it’s dependencies. So a unit test. Now it depends on what you’re wanting to test and validate for, since you can hermetically test method(s), API and UI layers. We try to use fakes over mocks when doing it but you have to work in the bounds of what you can.

Unit tests → create mocks for anything that unit (usually a method) is dependent on. Generally using some type of mocking library.. like Moq.
API (integration tests) → We usually do this to test behaviors and not function. So we’ll call our gateway into the service, however, we’ll mock out any external dependencies such as other services. But we will utilize tooling to generate a real database, or a messaging service, authentication etc
 This way we can mimic production like checks and service dependencies to make our test high fidelity. This allows us to test our Client, Database and anything else really. WITHOUT needing the other services to be online. The test is self contained and has everything it needs to do.
UI Layer → you can utilize something like playwright to run through the UI but pass in the network traffic you want to mimic. So we can sever the UI from the backends and just test it on it’s own. So it’s the real UI but fake data.

There is pros and cons to it but when utilized correctly, you’ll be able to catch a lot of problems sooner in the develop process. But the biggest thing is the “What”. What are you testing? What are you trying to achieve.

2 Likes
12

Hermetic testing to me is a method for ensuring that tests run consistently and offer a consistent value. If we apply the idea of Hermetic Testing to only the data element of a tests requirements, we can understand what this means.

Imagine there are three ways to handle data in our testing:

  1. Our tests can create their own data and then test that data
  2. Our tests could expect data to already exist, and choose something valid at random
  3. Our tests can expect specific data to already exist, and use the exact same data everytime they run

The third bullet point is how I imagine Hermetic Testing. Test-001 expects [item] to already exist, it navigates directly to the UID of [item] and performs the test.

This is a more scientific method of testing when compared to 1. and 2., because they inherently have risk built into them.

  1. In approach 1, if the data creation step fails, the test fails without even reaching the point it’s meant to test.
  2. In approach 2, if the assumed data isn’t present or changes unexpectedly, the test again fails for reasons unrelated to its actual assertions.
  3. In approach 3 (Hermetic Testing), the framework pre-seeds the required data in a known, stable state. As a result, tests should always run the same and in theory only fail when there’s a real bug.

This, to me, is the purpose of Hermetic Testing: to build a regression pack that executes exactly the same way, every single time, so that when a test fails, we can trust that something really broke.