My question is in regards to a post deployment situation. We have plenty of unit tests and even integration tests where stubs and mocks are being used. I want to understand the habitation of code that is meant to only be run after code leaves lets say a development environment and is deployed with other system components. I guess we could really call this System Integration? For instance, take an example of a catalog API that provides a user a way to store a picture of the catalog item. The storing of the picture is actually done by a completely seperate API. For code, that performs testing like that would you store the test code with the original artifact (catalog API) or would you create a new test artifact to handle the System Integration aspects?
In this system, it sounds like there are tests that are meant to execute in a different environment. The tests are deployed into an environment that interacts with other parts of the system - System Integration as you said. You are asking where such tests would be maintained.
Tests that are meant to execute in an integrated environment have a specific purpose and belong, in my opinion, in a separate artifact.
It is an interesting testing approach to have specific tests for integration. Since they are specific, a separate artifact provides some flexibility on maintenance, deployment frequency, and an ability to remove the tests without impacting the system or products.
I wonder what would be specific enough to warrant deploying tests into an environment rather than executing tests from outside the environment. Could you elaborate?
For me there are two guiding ideas here. The first is this. The more the tests will need to change as the product is changed the closer it should be to the developer. Normally high level system integration does not require that frequent changes so it makes sense to favor it as a separate entity. Especially if the group of people that will work in that code base are different than the group of people that work in the production code.
Which leads me to the second idea. Conway’s Law. Your code structure will copy your communication structure. This goes both ways. You can design your code structure to change your communication structure, so if you want it to be separate keep it separate, if you want the communication structure not be separate then don’t keep it separate.
Sorry if i am misleading, it’s not necesarily that the tests are deployed to a different environment. The coded artifacts are deployed through a myriad of environments and the tests are actually executed as a part of the CD pipeline. My question truly relates to the separation or non separation of purposeful test suites. For example, unit, functional and integration tests that utilize mocking techniques i can clearly see that those would reside with the coded application under scrutiny. I begin to question when the purpose is truly to test the integration and deployment of a coded application as it passes through multiple environments. You know, when we begin to hit other APIs that we are dependent upon or other system components.
I hope that helps to clarify what i am really asking about?
That’s awesome that the tests are part of CD pipeline! I’m trying to get there myself.
I still believe these tests are separate. They have a different purpose than unit/manual tests that evaluate product behavior. These tests, at an integration level, evaluate connectivity, configurations, and security. When I execute these kinds of tests, I don’t focus on product behaviors as much.
Thank you for the advice! I am going to create a different test application for my system integration suite. If i can be of any assistance with the pipeline, please let me know?
We’re a Java shop and we have everything in the same repo. The tests that are lower level and use mocks are in src/test/, while we have more of the e2e/live system tests in a top-level qa directory.
The e2e tests are in a top level directory which is not included in any of the build artifacts, so they never get deployed anywhere. In our CI/CD system, we clone the repo, and have steps that run the unit/integration tests as well as the e2e tests, but again, our build artifacts don’t contain any of this, and the deploy step only deploys the code that’s required for the application to run.
We tried separating out the e2e tests from the application code, but that actually ended up being more headaches, trying to keep the two in sync, the CI/CD system didn’t like having to clone from multiple repos, etc.
Would you mind providing a sample directory structure or tell me if i am correct here?
src
…test
…qa
…place where all your unit/integration tests live
With your suggestion, i could see how one could gain all the benefits of cohabitation of test and source code. However, i wonder how you run tests independently? We use an annotation approach. We created custom annotations like @Unit and @Integration and then use a profile to execute each independently. We could establish other custom annotations to perform system integrations tests ie. @SystemIntegration or @EndToEnd.
src
…main - the application code
…test - the unit and integration tests
qa - e2e tests
The items in src/main are compiled to generate distribution artifacts, while the items in src/test are for test. Are Java (and Groovy tests) include unit tests as well as integration tests (which are pretty much e2e tests, but they run against containerized dependencies instead of real SQL, rabbit, 3rd-party APIs, etc).
The reason qa is a sibling to src is that those are end-to-end tests - they don’t have any reliance on the underlying application code. We do web APIs, so all they need to to is speak HTTP and parse response objects, though sometimes they get a little more intricate and also check the persistence layer(s) directly.
For the unit and integration tests, we do the same thing with annotations, and use different Gradle tasks to trigger different types of builds (i.e. build the application, run the unit tests, run the integration tests, etc), with everything in one Gradle build file.