How to keep track of broken releases?

Question

I spent considerable time googling around, but found nothing that would answer my question, so here it is.

If you do a canary release, and you learn the the release is somehow broken: how to you denote that a certain tag is broken and should not be deployed anywhere else, but removed from your productions ASAP?

I had several ideas:

1. Using GitLab, I could add something like **broken** to tag release note. Deploy jobs would call GitLab API to ensure that the tag I want to deploy is not annotated as broken.

2. I could have a text file in repo, where broken tags would be listed. In this case deploy job checks if the tag you want to deploy is not listed in the file.

3. Do no save information about release problems: just deploy a new stable, patched version to production and never look back.

How do/would you handle this? Any tips or hyperlinks are welcome.

Answer 1

I am not sure about other approaches, but the one that I use is a modification of #3.

There's usually a classification of releases (for example think of Ubuntu LTS and regular release).

1. For the regular release, I would just release a new version and add a new tag.
2. For LTS/Stable releases I would delete a tag pointing to a faulty release or change a commit this tag points to

Answer 2

I'm building a product to solve this problem among others - https://relizahub.com (note it's pretty early stage at the moment).

Idea goes as following - for all your releases you stream their details to Reliza Hub. You would then have a client that would pick up releases you need based on combination of tags, approvals, etc. So it is somewhat similar to your #1 approach. For broken releases you either do not approve them or mark them as 'REJECTED' and this would stop them from propagating in your pipeline.

Answer 3

I think that it might help to distinguish between a "release" and a "release candidate". If you really have shipped a version of software to other people and it is 'totally broken' then something went really very badly wrong. To avoid this opensource projects have milestone release and snapshot release (aka "nighly releases" or "freshly releases"). These try to flush out critical failures through pre-release public testing. Once the risk of shipping something totally broken has passed then the software is tagged as the final release. After that, if there are bugs, and there always are bugs, then it is a matter of issuing advisories and warning folks about bad old versions within release notes.

For example, spring.io releases are versioned something like "5.2.5.RELEASE". We might expect that they have a "5.2" branch and that they had many release candidate builds on that branch. In fact they have actually have two special repos for milestone releases and snapshot releases as described here. These approximate to beta releases and alpha releases.

If you are not releasing software to other people to run, but are running the software as a service yourself, then your option 3 may work well:

(3) Do no save information about release problems: just deploy a new stable, patched version to production and never look back.

This approach can work well if you are a building a web-app or mobile backend based product. The internal software at large firms like global banks will typically use this approach. In fact, I have worked on many systems at both banks and startups over two decades and we only ever used this approach in practice.

I should mention that your Git server might have a "latest release feature". Both GitHub and GitHub Enterprise have this. Even if you release via a build pipeline to some artefact repository it can still be helpful to make a "release" within the Git server so that an API call to something like GET /repos/:owner/:repo/releases/latest API will return details of the latest version including the git tag that should match the release version. Your deployment piplien can then pull the corresponding prebuilt code from the artefact server to deploy.