The "real and effective" GIT CI/CD strategy

Question

I've started in this new company a few weeks ago, this is the CTO CI strategy:

Current: Developer team has the repo prod/master and they merge everything into master (no branching strategy). Once the code is ready in prod/master they'll ask Infrastructure team to start the deployment process which uses Jenkins.

The Infrastructure team executes a job in Jenkins that performs this actions:

1. Clone the whole prod/master into build/master (so they don't mess with the developers)
2. Execute scripts to build the binary(ies)
3. Generate a .txt file with the version of the build
4. Commit and push this changes into build/master (reason: prepare the deployment)
5. Apply environment specific settings and push, configurations, binaries and code to distro/master

We end up with three repos at the end of the day for each application, that means, if we have 10 applications we would have 30 repositories

Reasons of the CTO for this:

1. prod/master: For developers and their code (no branching, only master)
2. build/master: For Infra team to generate versions (to prepare the deployment)
3. distro/master: Binaries + code + specific environment configurations (to perform rollbacks, traceability and havebackup)

Cons:

• Really complex process
• Unnecesary large data ammounts in repositories and slower processing when performing deployments
• Only works for FileSystem deployments (Databases are not considered in this sceneario and that kind of changes are manually performed)
• No instant feedback for developers
• Complexity when crossed patches/fixes and deployments
• Developers are involved in the production deployment (quite often, in order to test and apply changes on hot)
• Most of the deployments are performed directly into production

Pros:

• There's backup and posibility to rollback
• Easy traceability (for rollbacks, not for development)
• Specific configurations per environment are stored in the repos with the code and binaries

And this is my approach:

1. Developers create a JIRA ticket, which will be used as tag for the build and to create the branch
2. Developers will deploy and test in a Q.A/PRE-PROD environment
3. Once the code works, it will be integrated to master
4. Once integrated with master, the binary goes to a "binary repo like artifactory or other"

Pros:

1. Traceability: The code deployed is easy to find through the tag (JIRA-XXX) for an specific build.
2. Rollback: Taking the binary from the repo (Artifactory)
3. One Repository per project, it means 10 projects are 10 repos, not 30.
4. Instant feedback to developers, if the deployment is not sucessful they can change their code
5. This design contemplates db scripts as hooks
6. The configurations per environment will be handled with Ansible + GIT, generating templates with placeholders and a backup of each configuration.

Cons:

• Re-educate developers to work in branches
• Force developers to integrate code only when it really works
• Change the CTO mindset only will happen through examples (working on it)
• We must create new infra (new environments to create deployments and not going to production directly)
• Lots of hours automating through hooks, rest apis
• Need to implement new technologies

I'd like to know the opinion of people with expertise on this git strategies and the balance between development and operations.

Regards.

H.

Answer 1

By having three repo you kind of have a branching strategy already but a complex one.

I think you're not facing a technical issue but a cultural issue. Even if your solution is good (I think it is but I did not go into details) you won't be able to have it accepted as is.

In those kind of situations you need first to understand why it has come to this. What problems where the team faced that they worked this kind of solutions. It may have been:

• No understanding/fear of git. If it was the first SCM tool implemented by the CTO or ops team, the may have not understood how git works and have think multiple repo safer to ork with.
• Lack of trust in developer (or ops, or other): Maybe someday, someone broke something and now everyone pay the price.
• Politics : Someone if protecting himself or his team from any kind of change, because if it works... it woks and any change that would break something could come to bite them.

In fact writing this I always go back to a lack of knowledge of some tools be it Git, JIRA, or any other too in the CI stack.

Steps to proceed:

• Book a meeting with someone in charge/knowledgeable. If you CTO is accessible, it may be best. Don't discuss solution or issue. Just try to understand why things are how they are. (try the five whys)
• Find the deepest issue (lack of trust, fear of change, fear of something broken...) and find a single change that address this issue. It may be hard, but it may be harder to push for an overhaul that fixes every issues.
• Alternatively you can propose practices that don't impact the current workflow but can render it obsolete in the future. For example having developer working on branches to avoid merge on master (if it is an issue) or add JIRA ID in commit title to track work easier. Having things in place and proven to work is a good argument to push them to "higher" level (infra, prod, ...)

Answer 2

At a previous client we switched from no branches to branch-based development. At my current client we are doing mostly branch-based development.

The huge downside I have is that all of our work is not integrated most of the time and it is a lot of pain to work with others who also work on branches.

The branching thing is good in theory, and git is up to the task. But in practice I at least don't see neatly separated tasks, where I don't have to use other people's current work. We don't always have small (couple of hours) tickets, so the window where the software is not integrated is always larger. And there's dependency hell, because I can't just depend on "SNAPSHOT" versions, I always need to build branches, or worse, integrate myself and have that dependency somewhere.

Ideally I would want everybody to work on master. Of course that also at some point becomes untenable when there are multiple teams or products with dependencies and different release cycles. Devops and CD would solve most of these problems, but none of the places I worked with have that, not even close.