If you ever been bitten by a plugin upgrade that broke some functionality you must have given some thought about this problem: What should be the Jenkins plugins upgrade policy? How do you test changes before deploying them?
Has anyone gone as far as having a test instance running dummy jobs to test the new versions or do you just pray that upgrading the versions won't break anything?
According to the policies of the company where I work we have dev, preprod and prod environments (on some service dev can be missing). And the path of new version preprod->tests->validation->prod.
In our case jobs in preprod are heavy and complex enough to be sure we do not need to pray when implement in prod :)
Note: We use svn to maintain and deliver configuration. We do not make changes inplace.
We needed a 100% HA Jenkins environment. we often upgrade plugins / Jenkins itself.
This causes a big headache if the build breaks after the upgrade.
The safest way to sort this is really getting a Demo Jenkins setup. Maybe on the same machine using multiple Tomcat application, you can achieve this cheaper.
What we did is create a separate (Demo)VM and replicated the prod set-up on the Demo VM. Before changing / upgrading anything we would take a snapshot of both VMs. Then we would test the upgrades on Demo VM. If it works good, change it on Prod.
I guess you can look up the community (like SE/SO) if anyone has faced any problems with the plugin you are planning for.
I'd always manually trigger a re-run or two on at least one recent green (or almost green) label on every relevant project/branch that uses the respective plugin and check that I get the same results. Just to be on the safe side.
Any result discrepancy would need to be investigated to determine it they're caused by the plugin update or not. Maybe a few more re-runs with both the old and new plugins?
