Hi People,
I have a CI/CD maintenance problem.
We have a Jenkins(which is separate from Dev Jenkins environment which is for build and deploy) which is having over 100 jobs for various automation tests. And all these tests run on the Production environment. Some of the tests are flaky, hence give false results.
How do you guys maintain your Jenkins environment? How would you setup alerts for these 100 jobs?
Also if you have to run any tests from the Jenkins on a test environment, do you use production jobs OR do you have a separate Jenkins job for test environment and even a separate Jenkins environment for running tests on test environments?
Sounds like you’re asking how do you manage a large and growing number of jobs on a Jenkins environment.
re: managing a lot of jobs
With Jenkins 2.0+, there’s no reason not to be using Multi-branch Pipelines and Scripted/Declarative Pipelines. They are an easy way to define your Jenkins job as pseudo-groovy code in a file called Jenkinsfile that gets committed to your projects repository. No more having to tweak the build scripts in the Jenkins UI! You could just commit and push 
You could turn your multiple Jenkins jobs/pipelines into a set of Scripted Pipeline files to reduce the number of jobs you have to manage and move more of it into a git repository where you can commit, make changes, and push.
Also in Jenkins 2.0+, you can create a Github/Bitbucket org (if you use them) - which means that Jenkins will scan your org in Github/Bitbucket and pull in all repositories that have a Jenkinsfile. Definitely useful to make that process of discovering new projects easier.
re: running jenkins in different environments
Yes, one common implementation I’ve seen is that if you have a dev and prod Jenkins instance. Devs have access to the dev instance, and the prod instance is locked down to a few people who can push buttons that can run things in prod.
A good solution to that problem about dev/prod access depends a lot on what you’re testing.
- If you’re running a test suite against a REST API that is already public, then it doesn’t matter whether you run it in a prod/dev Jenkins instance because either one can already access it.
- If you need to be inside a network in order to see specific private IPs like a database, then yes - it would matter whether you ran in a dev/prod Jenkins.
The project I’m in using separate job for each environment.
We’ve got pipelines in Jenkins for a large number of APIs on a number of test environments. We make use of the functionality in Jenkins to feed a status flag icon through to another website (Embeddable Build Status), and have dashboards set up in Confluence. You can then very quickly see what’s passing and failing at any time. Over time you can identify the tests that are flaky and see whether anything can be done to improve their stability, or as we’ve done with some of our tests that were too unreliable, disable them pending a fix.