Is CI/CD a myth

Question

Perhaps because I'm primarily a coder, I don't understand the benefit of CI/CD, or any related terms such as scrum, agile, etc.

• In my opinion, testing can be done locally on development computer;
• testing cannot replace manual code review, a lot of security features are not easily tested using automated scripts, there are lots of code paths whose functional correctness need to be examined, not to mention the many possible edge cases (speaking as a full-stack developer looking at back-end codes) (small addition) - these needs human review in addition to testing (in some case, test cases also need reviews). The essence of this point is to express my doubt on any exaggeration of benefit offered by CI/CD over security;
• the whole concept of CI/CD seem to be advertised towards managers and other non-coders, and generally for projects that aren't security-critical.

I've seen claims of benefit of CI/CD such as "faster xxx", "less bugs", "better code quality", etc. but except "faster xxx" (which I don't see why it's true), all other quality dimensions can be achieved though other means - style guide, use of linters, mentoring, etc.

So with all those said, what's the true benefit, application and non-applications of CI/CD?

Answer 1

In my opinion, testing can be done locally on development computer;

Of course it can. And it should be done. CI/CD is not "instead" of it, but rather "in addition" to it.

testing cannot replace manual code review

Testing and code review are two separate things. The main purpose of testing is to prevent regression.

The problem with code review is that the bigger the app is, the harder it is to spot mistakes. The piece of code you modify may affect something in a different part of app, that noone sees. This happens to me (and people around me) all the time. Humans are just very limited.

a lot of security features are not easily tested using automated scripts

Security is inherently hard to test, that's correct. But that doesn't mean that other things (features) are. It is not really the point of CI/CD to test security. Unless the project you are working on is actually about security. Vast majority of projects are not.

there are lots of code paths whose functional correctness need to be examined, not to mention the many possible edge cases

Indeed. And that's precisely where the tests are needed. Also, I'm not sure how is that related to CI/CD? You could literally say the same about any form of testing. So are you saying that automatic testing is not needed? Regardless of CI/CD? That's just plain wrong.

So with all those said, what's the true benefit, application and non-applications of CI/CD?

To understand that we need to establish a fundamental rule. And that is: do not trust developers. You may think that you are a coding god, and hey, you might actually be. Except that coding gods don't exist and everyone makes mistakes. And developers are lazy. I am lazy. And yes, I am a coding god. ;) Do you want to know how many times I pushed code to main branch without running tests locally? A lot. I'm sorry, that's the truth: you cannot trust developers. And that's why you need mandatory CI/CD platform, which will always run tests, regardless of developers' laziness.

Answer 2

In my opinion, testing can be done locally on development computer

In a minority of cases, it could be. Mobile apps can only be tested in emulators. Embedded apps can only be tested in emulators. Large and distributed apps can usually not be tested on a single dev machine.

Even in those cases, having a computer run the tests on the code actually shipping to prod is far more reliable than a human testing code that might be going to prod.

testing cannot replace manual code review

And manual code review cannot replace testing.

there are lots of code paths whose functional correctness need to be examined, not to mention the many possible edge cases

All the more reason to have a computer do it.

So with all those said, what's the true benefit, application and non-applications of CI/CD?

Making software development faster, less risky, better code quality…

CI/CD means that you can release today rather than waiting for your weekly/monthly/quarterly release. Which means that you can try stuff out. If it’s not quite right or the customer doesn’t like it, you can just change it tomorrow.

It also means that you can make changes more confidently. The test suite will let you know if you caused any regressions. That means that people new or unfamiliar with the codebase can work effectively in it more quickly, since the suite will cover the biggest risks rather than their own familiarity with the code.

Honestly, quick and painless build/deploy pipelines are one of the few things that I’ve actually seen help every software development team.

Answer 3

In my opinion, testing can be done locally on development computer

With more powerful workstations and advancements in virtualization and containerization technology, more testing can be done locally on a development computer. However, there is still plenty of testing that can't be done in this type of environment.

Regarding technical testing, performance and load testing come to mind as something that can't be done locally for many applications. Although local testing and simulation may highlight potential performance bottlenecks, testing at the scale of production systems often needs to be done in dedicated environments.

Usability and user acceptance testing are also difficult in a local development computer. Again, some testing, especially usability testing, could be done locally or using tools that scan against usability standards. However, making the software available on the devices where end users interact can make usability and user acceptance testing more robust.

These types of testing aren't generally done within a CI/CD pipeline. However, a CI/CD pipeline can make it easier to carry out these tests by executing various levels of automated tests and, when those tests pass, packaging and deploying the software to environments where performance and load testing can be done. The build servers used to execute CI/CD pipelines can also be used to manage any automated performance and load testing.

testing cannot replace manual code review, a lot of security features are not easily tested using automated scripts, there are lots of code paths whose functional correctness need to be examined, not to mention the many possible edge cases (speaking as a full-stack developer looking at back-end codes) (small addition) - these needs human review in addition to testing (in some case, test cases also need reviews). The essence of this point is to express my doubt on any exaggeration of benefit offered by CI/CD over security;

Testing and code review serve very different purposes. Previously, I've written about the purpose of a code review. Although some organizations do use code reviews to find errors and defects, I don't think this is something that humans should be spending their time on. Developing a shared understanding of the system, cross-training, and ensuring the readability and maintainability of the code are far better focuses for the time people invest in code reviews.

It is possible to inject security testing into code reviews. If you have a CI/CD build pipeline, you can and should integrate linters and static analysis tools into the pipeline and expose those results to code reviewers. When humans are carrying out code reviews, they can review the results of these static analysis tools and flag any high priority items for resolution as part of the review process.

I also think it's a mistake to believe that security features can't be easily tested. Many of the items on the Common Weakness Enumeration (CWE) can indeed be found by testing. Others can be found by static analysis. Both testing and static analysis can be incorporated into a CI/CD pipeline. Similar to testing, a CI/CD pipeline can also make it easier to integrate dynamic analysis into a development process by making it easier to build, package, and deploy software to an environment where dynamic scanning tools can be used.

the whole concept of CI/CD seem to be advertised towards managers and other non-coders, and generally for projects that aren't security-critical.

I don't agree with this at all. The foundational work on CI/CD, whether it's the work from Kent Beck's Extreme Progamming (book) or Dave Farley's Continuous Delivery (YouTube, book) or Paul Duvall's Continuous Integration (book) are geared toward practitioners. Dave Farley has also talked about the application of CI/CD on critical projects, such as his past work in the highly regulated finance space.

This isn't to say that it's not important to sell it to managers. It is. The investment needed to set up and maintain a robust CI/CD pipeline is not trivial, and there is plenty of business value. However, there are also huge implications for developers when it comes to rapid feedback and a great confidence in the ability to build and deploy working software quickly and efficiently.

So with all those said, what's the true benefit, application and non-applications of CI/CD?

A few examples:

• Rapid feedback. A CI/CD pipeline is built on fast feedback through automated tests. Having a controlled pipeline that integrates these automated checks with other types of checks and gives developers feedback regarding potential issues on the order of minutes is invaluable.
• Confidence in build and deployment. CI/CD pipelines are controlled and automated. Any configuration and scripts used are managed and controlled carefully to ensure that each pipeline run is correct. The automation ensures that it happens the same way each time, regardless of the developers doing the work that causes the pipeline to be executed.
• A foundation for longer-running tests. Although rapid feedback is key, not all testing and checks can be done quickly. The ability to automate deployment and run these longer-running tests more frequently reduces the overall feedback loops, giving the developers the information they need to make informed decisions about their work.

Answer 4

Perhaps because I'm primarily a coder, I don't understand the benefit of CI/CD, or any related terms such as scrum, agile, etc.

Yes, almost everything in software today is old and familiar ideas dressed up in new fancy jargon. If they were spoken about more plainly and historical experience was consulted, then their advantages and absurdities would often be clearer.

In my opinion, testing can be done locally on development computer;

Perhaps for certain kinds of tests. However, complicated multi-user systems tend to acquire properties that make them difficult to test completely except in normal use, and many systems have automatic test suites that are far too heavy to be executed locally on the developer's own machine without impeding the progress of the developer.

testing cannot replace manual code review

Yes, automated tests do not replace the analysis work of all kinds that programmers perform. Effective suites of automated tests typically result from manual analysis, and automate certain routine testing activities that the analysis shows would be desirable to do and would otherwise have to be performed by hand or go undone.

a lot of security features are not easily tested using automated scripts

Adequate "testability" is a property of machinery that is typically regarded as desirable, and results from design and engineering that is good and skillful.

So with all those said, what's the true benefit, application and non-applications of CI/CD?

I think one basic principle of CI/CD is that deploying software is a repetitive routine activity, and therefore it should be a click-button exercise to control, as opposed to something that needs complicated attention and lots of human labour like getting a steam roller into gear. So if you want to release every 5 minutes, it should be possible.

I think another principle is that it's a reaction to a fixed release cadence, and says that the default release time should be immediately when the development work is done (or else with no more delay than is justified for the situation), rather than everything being either rushed or accumulated for fixed release dates.

As usual though, managing software development itself requires professional expertise and judgment, and staff who are engaged with the problems and constraints of their particular environment. What's appropriate for a one-man, in-house development function, is not necessarily appropriate for Google and Microsoft.

Answer 5

It sounds like you've been presented with a narrow view of CI/CD that doesn't fully reflect its purpose.

In my opinion, testing can be done locally on development computer

CI/CD doesn’t discourage local testing; rather, it complements it with build-agent testing, which catches issues like bad merges more effectively.

Testing cannot replace manual code review, a lot of security features are not easily tested using automated scripts [..] these needs human review in addition to testing (in some case, test cases also need reviews)

CI/CD doesn’t replace manual reviews; it simply automates repetitive checks, reducing the burden on manual efforts.

there are lots of code paths whose functional correctness need to be examined, not to mention the many possible edge cases (speaking as a full-stack developer looking at back-end codes)

Automated testing focuses developers on creating tests for these paths, saving time over repeated manual checks.

The essence of this point is to express my doubt on any exaggeration of benefit offered by CI/CD over security

Security remains critical; CI/CD helps automate checks but doesn’t claim to replace in-depth security assessments.

The whole concept of CI/CD seem to be advertised towards managers and other non-coders

Management buy-in is essential because CI/CD implementation requires resources and cultural shifts (dev capacity as a startup cost, dev training, changes to release comm processes).

The reason this is advertised to managers is to get them to buy into it. The reasons you see little advertisement to developers is because most of them are already on board with a CI/CD shift as it directly benefits their work process.

and generally for projects that aren't security-critical.

For compliance-heavy or critical applications (e.g., NASA, medical equipment), CI/CD is often modified due to regulatory demands, not because it's inherently insecure.

---

In short, CI/CD enables frequent, smaller releases, reducing risk per release and simplifying troubleshooting. While initially it requires setup, automation minimizes ongoing manual effort, providing efficiency gains that are scalable across teams and repetitive releases.

Automating something costs more effort than doing it manually once. But if you have to do it 100 times, it's better to automate it once than it is to do it manually 100 times.

Answer 6

CI/CD is absolutely real. It exists, and it solves some real world problems very well, and it helps a bit with the things that most developers think CI/CD is there to fix.

In my opinion, testing can be done locally on development computer

Yes, it can. and when you are writing your code you should be stepping though it and verifying that the data and code flow are behaving as expected. I do not want you to be trying to do this in the CI/CD pipeline. In the CI/CD pipeline I want to run a large number of basic sanity checks from the unit tests that you have also already written. This is a fail safe mechanism to prevent something that is totally broken from being deployed onto the production servers.

testing cannot replace manual code review, a lot of security features are not easily tested using automated scripts, there are lots of code paths whose functional correctness need to be examined, not to mention the many possible edge cases (speaking as a full-stack developer looking at back-end codes) (small addition) - these needs human review in addition to testing (in some case, test cases also need reviews). The essence of this point is to express my doubt on any exaggeration of benefit offered by CI/CD over security;

The manual code review is often the trigger for the CI/CD process to start. When several people have reviewed your feature branch, and authorised a pull request, the action of closing the pull request and adding these changes to the existing code base doesnt replace the CI/CD process, but acts as the trigger point for a deployment onto a pre-production environment.

the whole concept of CI/CD seem to be advertised towards managers and other non-coders, and generally for projects that aren't security-critical.

Its more a case of CI/CD is a process that is not part of the core developer workload. It falls into the camp of the work that other people do, when the developer has actually made the product.

Here is what I value about CI/CD

1. Consistent releases. Its been 15 years since the last time I worked on a project and could tell which of the developers fixed the last bug and updated production. Why? Because developers do forget to commit all of the code. They test it on their machine. It works. They Deploy it. Then the next bug fix is handled by a team mate, who doesnt have that last fix on their laptop, so when they fix the next bug and deploy, it also undeploys the last bug fix. Thats really annoying.

2. Reliable releases. Having a scripted automated process that takes the current code, builds the application, installs it into the run time environment and then places that runtime environment onto the production servers and routes live traffic to the code happens smoothly, consistently and with predictably. Doing this by hand allows mis-clicks to occur, steps to be missed out, a server to be forgotten about. Never mind if you are operating at cloud scale and need to deploy to 200 servers.

3. Automated Rollback. As part of the deployment onto the live servers, the CI/CD process can only route 1% of the live traffic to the new deployment, keeping the remaining traffic on the existing code base. If the new containers show either a higher error rate or higher latency, then we can automatically abort the deployment and trigger an investigation ticket to the platform team.

In reality, the benefit of the CI/CD pipeline is really for the quality assurance processes - verifying that what is being tested is really the same as the thing being deployed and for the operations team - making deployments predictable, automatic, and whenever possible, something that can be done on a live site during working hours.

Answer 7

So with all those said, what's the true benefit, application and non-applications of CI/CD?

Frequent releases.

Everything else is hype.

Frequent releases means fast feedback loops. Which is golden. The salesmen often don't understand that and will tell you it can make toast if it will get you to buy. But just because they are full of crap doesn't mean this isn't a good thing. I never want to go back to a shop that needs 6 months of testing before a release.