For a long time, in our team we’ve not been able to apply any check or validation on terraform changes.
If you change something infra-related, you want the application to behave without any changes, this is why we are exploring tools like. Terratest and checkov
Firstly, congrats on the fancy assignment! Not many people actually do test the infrastructure if any at all.
There are usually 3 types of tests can perform:
The infrastructure provisioning tests will depend on the cloud provider that you use (AWS/Azure/GCP/etc.). Almost all providers (and definitely the large ones) have APIs where you can query the infrastructure. For example, on AWS you can list EC2 instances like this. Your command in the last example would look something like this: aws ec2 describe-instances --filters "Name=tag:Name,Values=MyInstance"
There are amazing frameworks that simplify tests significantly like Terratest: https://terratest.gruntwork.io/
All the best! Sounds like a challenge and fun!
Like any tool, Terratest can sometimes be a bit overkill for the testing you may want to carry out. Infra testing is expensive both in terms of literal cost and time, so its important to fully understand what you specifically want to test for or risks you want to address.
Most of the time Terraform code is responsible for configuration, so thats what youre looking to test - its important not to waste too much effort in testing vendor functions e.g. that the AWS provider does indeed work and make an EC2 instance - these are unlikely to be your risks. Really you need to be making sure your tests are targeting the code your team is actually responsible for.
Checkov and built in SAST tools like Gitlab’s SAST are nice cheap tools for checking things like firewall rules and general good practice. However they cannot test any logic in Terraform code.
Ive recently been playing with some different tools to try and “unit test” Terraform logic at the Plan stage to avoid having to physically build infra to test it. But not found any brilliant ways to do this yet, currently grappling with writing OPA policies. Feels a bit brittle relying on Terraform json paths atm.
If you happen to be using Ansible anywhere at all to handle configuration, Molecule is an excellent tool for testing Ansible roles and supports spinning up Docker containers for it to keep it cheap (albeit still somewhat expensive time wise).
All that said, Terratest is still useful, just need to be cautious that youre using it for the right level of testing, its important to keep asking can this testing be done in a cheaper, more direct way and asking what are you trying to test. If your Terraform doesnt have much logic in it, does it even need testing? Or is the effort better spent elsewhere in montoring/improving the load balancer health checks/resilience of the configuration? E.g. regular automated testing of backup/restore process?
I’ve approached the infrastructure testing aspect a bit differently. I avoid infrastructure tools coupling to specific technologies. After all, you probably want to target specific behaviors. Whether your stack uses cloudformation or terraform is irrelevant from the user’s perspective - it’s behaviour that matters!
I would take a group of end to end tests that hit a set of services running locally in containers and tweak them to test a deployed stack.
An example would be a browser test that ended up sending an email.
With a local end to end test it would spin up a set of containers - one of which sent an email to another container running a mock SMTP server. It would trail the logs to check that it was sent.
With a “deploy” end to end test it would spin up an environment using terraform and then run the same scenario (the browser steps would be almost identical). Instead of checking the mock SMTP server it would send an email to a service like mailinator where you can programmatically validate a real email was sent.
There would generally be some overlap between the “deploytest” end to end tests and “localtest” end to end tests but there would also be a bunch of stories which really only make sense to run in one environment or the other.