8

Here is a partial quote from an answer to the question about "How to avoid continuous integration-caused instabilities in test environments?":

This environment usually freezes during the tests.

My question: what are sample implementations of a frozen environment? I.e. what can you do to technically enforce that nobody (except if allowed by an authorized user like a release manager) will be able to change anything in such frozen environment.

Clarifications:

  • I'm not talking about what (I think) is called "frozen periods" during (eg) year-end processing in banks. That is about not being allowed to apply any (repeat any) changes to production environments, to reduce the risk of new changes/fixes being introduced that may impact the year-end processing.

  • Assume that users who are allowed to approve/apply changes anyway (such as the release manager in my example), will only do so in exceptional cases. Such as where during testing a high severity issue is encountered, for which deferring a fix to a next release is not an option (since it would production at risk if the release would be activated without such fix).

  • This could just be about suspending any automated update during the time of the test. The point is: avoid someone else upgrading an Application A to version Y while another team is still testing application B in version X which rely on application A. This could mean having a guard to avoid a test team to require an update on a dependency under test.

030
  • 13,383
  • 17
  • 76
  • 178
Pierre.Vriens
  • 7,225
  • 14
  • 39
  • 84

3 Answers3

4

TeamCity has a Shared Resources build feature which allows you to define a resource which multiple Build Definitions depend upon it. Build Definitions can either require a Read Lock or a Write Lock, you can also define if these locks are exclusive or allow a degree of parallelism.

If we make the following assumptions about a shared environment named PreProd:

  • A Shared Resource exists named "PreProd".
  • All build definitions, such as deployments, that make any change to that environment take an exclusive Write Lock on "PreProd".
  • All build definitions, such as non-restrictive tests, that perform read-only operations on the environment take a non-exclusive Read Lock on "PreProd".
  • TeamCity is the only process that can do anything on PreProd, albeit perhaps via another tool.

Therefore the following are true:

  • When a deployment is happening then nothing else can use PreProd, they will be queued.
  • When a test is running, any deployment would be queued until the tests complete.

You can use a similar mechanism with Jenkins using the Exclusion plugin. In fact, you could build this functionality into any process using locking or semaphore - for example Apache ZooKeeper or HashiCorp Consul.

Richard Slater
  • 11,747
  • 7
  • 43
  • 82
3

My answer in the context of the question is a very expensive test environment (like mainframes or very large telecom equipment, for example) which are expected to be shared by multiple users for multiple tests, even simultaneously.

In many cases such equipment has a software management system responsible for, among other things, controlling software (un)installation/upgrades/downgrades/etc. Which should have some mechanism for blocking such operations (which would be, if I understood the question correctly, equivalent to freezing the environment) based on some more or less programmable policies.

In such case a specific policy could be developed exactly to support the desired freezing schedule necessary for the test environment. Ideally automated, accepting freezing/thawing triggers from the external sources which could be the test execution wrappers or CI systems. And maybe with human-triggered overrides, if deemed necessary.

Of course, such feature of the software management system would be helpful when testing other components of the equipment, but maybe not for tests of the software management system itself.

Pierre.Vriens
  • 7,225
  • 14
  • 39
  • 84
Dan Cornilescu
  • 6,780
  • 2
  • 21
  • 45
0

This sounds like an anti pattern to me. I believe everybody or nobody should have access to all the environments.

If users are subverting the process then I would take a serious look at the process to try and ensure it's not getting in people's way.

Implementing an automated mechanism that enforces a specific state is also useful for encouraging people to do things the right way. This can be via Config Management or destroying any immutable instance if someone SSHs to it

Robo
  • 755
  • 3
  • 6