So we currently have both Unit-Tests and E2E Tests (front-end and API tests using Playwright).
We want to start adding Tests we can run during PR, that doesnāt require a deployment like E2E tests require.
To explain the context, we basically following what is being described in here:
Component Testing is testing your Micro-service in complete isolation. Other micro-services, databases and such - are completely mocked.
Integration Testing purpose is to test your Micro-service integration points. So you exclude most of the internal micro-service logic, and focus on only verify how the end points communicate (with another real Micro-service or database)
While this makes sense, it raises a LOT of questions. Hereās couple of the top of my head:
We run Lambdas, not containers. When testing components, are you supposed to run Lambda locally, or is it OK to just invoke the function that the endpoint invokes?
How do you mock an external AWS-call? You can use Unit-Tests tools and stub the actual function that does the final call (which kinda makes it fell like Unit-Tests of multiple function instead of just one) or you can use HTTP interceptors to catch the request on the network level and fake it back? (but those normally just work with HTTP Requests I think)
Whatās the benefit of in-memory databases over a local Docker container for example? isnāt it easier to use a Test-container? Are those big databases in use these days supported in memory mode?
Integration Testing sounds very complex. While you can stub your micro-service functionality, if you are actually invoking a real database or another micro-service, and you want to run those during PR - you have to spin up both, and their dependencies? as you can easily mock external service (as you donāt have access to the code)? This sounds like I canāt really test it during PRs unless Iām ready to spin mini/full environment?
Etc etc. Components testing feels like an easier goal to follow, as it excludes other services and the need to spin them up, so I guess we will start there. But are there any good examples/articles out there in the wild? Bonus points if they are in the AWS-Cloud/NodeJs/Typescript world
Also, I would love to hear your experiences! Thanks!
Each team really wants to be testing their own stack parts. We all know that mocks cost effort to maintain, and that in-memory database choices win out when it comes to platform design. I always go with whatever is most reliable and fastest, even if itās not my dream test environment.
This week Iām busy trying to semi-E2E a stress/load/scale test of a feature that already shipped. Only to find that not only are the interfaces and flow between the services a bit confusing, but so is the spread of knowledge across teams. I only want to plug into and test one service, but it relies on others, and obviously I donāt want to spin up an entire configuration in database for each test. So Iām going to play with it a while before I choose my route.
I have probably been doing E2E testing and GUI testing for too long, but for me my motto is to try something, learn from it and then only make your commitment to a route. Changing your testing approach gradually over time and continuously means you get to see fresh views of the product, itās not that evil a time sink. Definitely, less evil than choosing one way and thus only ever finding one class of defects that is suited to my chosen test approach only.
Hey! Thatās actually a big initiative Iāve started at my company, doing integration testing. So this is near and dear to me. As my company is trying to align closer to the Test Pyramid. So minimizing our E2E tests and focus on Unit and Integrations
Iām not familiar with Lambdas, as my company does work in containers. So thanks for saying something new for me to look into!
When it comes to mocking external calls, we adopted WireMock. We have it structed where we have a boolean(true/false) trigger to say (true)āmake a real callā and (false) use the mocks. We have it structured in a way so that when we do a real call (true). We have WireMock listen to all the traffic and automagically build out external API mocks. Although, to be honest, you can just build out the Given/Then calls for mocking. So that way when a Given endpoint is called with a Given payload. It will Then return a pre-defined response instead of doing a real call. This is nice so we can control the API calls and responses. However the downfall is, if the external endpoints change signatures/responses. WireMock doesnāt know about this.
2a, To solve for that aspect if an external endpoint changes contract or signatures. We are now starting to proof out PactFlow.io for contracts. So that way we can verify that the endpoint isnāt changing in a way that will break our connections. So we have WireMock which is about isolation, and PactFlow for external contract checks.
Speed. In-memory are much smaller overhead, generally can be easier to maintain and much much faster. I utilize TestContainers so we can have a mimic of our production microservices as close as possible so we can verify the SQL procedures and checks as our Production environment would work. We definitely notice some longer build times due to this, and weāre investigating ways of minimizing the impact.
It can definitely be a tough one to navigate and Iām still running into issues and constantly question the smarter people of āis what weāre doing worth it compared to xyzā Our integration tests run as part of our CICD pipeline. So that way we can just on doing exploratory and Acceptance Criteria checking in PR environments. Since in theory our function and code has all been tested before that point.
Hey, great question and Iāll try and offer some guidance based on my experienceā¦
As mentioned in the previous answer by @sharmon contract tests are a great PR / pre merge test with a high amount of value that can be run for quick feedback - more so in a microservices architecture. You can quickly verify the external contracts / API specs between consumers and providers without having to spin them up or deploy them. Pact is a popular framework for doing contract testing and has implementations in all popular languages (including Typescript). Disadvantages of this approach is that it can be complex to setup initially, youāll need a server to host all the contracts and it works best with internal microservices your teamās/company have control ver - if you rely on a lot third party services it may not work so well. It needs all teams developing microservices to be keeping their contracts up to date for it to work well.
For the component test level, with AWS specifically, I remember using a framework / tool called LocalStack which allowed you mock out any AWS service you needed. E.g. S3, Lambdas Dynamo etcā¦ which helped in terms of testing a microservice in isolation, locally without having to deploy it out to AWS. But this was some years ago and Iām not sure where LocalStack is now and whether itās the best framework to use for this.
When it comes to mocking external calls, we adopted WireMock.
@sharmon I keep seeing WireMock mentioned fondly, but didnāt yet sink time to learn how it works. From what your saying it sounds like itās sort sort of a gateway between services, that records network calls, and you can turn off the real network calls when you want and replaced with those records? So itās sort of mocks but you donāt specifically state the mocks but collect them by running real scenarios?
Contract testing is indeed something we look at, we played around Pact as a POC already
Speed. In-memory are much smaller overhead, generally can be easier to maintain and much much faster.
When we say āin-memoryā - is it a version of that database supplied by the company that wrote the database? or āin-memoryā basically means saving a object in memory and manipulate you code to somehow consume it?
For the component test level, with AWS specifically, I remember using a framework / tool called LocalStack which allowed you mock out any AWS service you needed
We have had great success at my company with it, although weāre still in the infancy of this. Itās worked great for us. So you can structure it either way, building mocks or the āgatewayā system you described. We created our own wrapper library around WireMock to do this gateway. So I can dictate whether or not I want a real call out or not, and each Real Call is listened to and a new mock is generated if any changes happen. If I want a ultra real call that I can view on our webpages I would comment out the TestContainer so it hits the real database.
For in memory I have used it as the latter explanation you had. An object saved in memory and manipulate it. So for my tests Iāll bring in an āIDatabaseRepositoryā and when I build the interface in my test class. Instead of it using real calls Iāll just have it return whatever Iām expecting. Itās more of an in-memory fake than a real database in my case. You can see more of what I mean here with the āSliceā method, as this is what I based my implementation off of https://youtu.be/dV3SSY7I9VM?t=3251
Thank you @sharmon . I briefly looked on the video on the way home, and it seems by āSliceā his basically introducing dependency inversion? where you pass the object to the class and then you can pass a mocked options of that data?
Are you using and ārealā data-bases? like, Test-Container that saves data on the RAM rather than Disk? Thatās also considered In-memory a lot right?
@okiba Possibly, Iām still pretty new to all of this and Iāve just been diving in and learning and applying stuff Iāve learned. Although to me, itās basically removing the database calls from the method and making them their own private methods, That way for the test you can just pass the interface into test suite and create a mock/fake/stub of it and have it return what you need for the test to continue.
For the testContainer part, With the size of the databases and how this documentation lists it . I would say they save the Disk. Ryuk (the testcontainer manager) deletes the container and cleans up the volumes after each test suite runs.
Yep. What you describe is called ādependency inversionā (or injection, depends on the side your looking at). You take the dependency, and you pass it to the class, instead of initialize it in the class (therefore able to pass mocks).
Thanks for sharing the documentation about in-memory! sounds like indeed TestContainer is the prefered option, at least by TestContainers
