6

Using continuous integration represents an (almost) mandatory requirement for a software project to be agile. And without CI there can be no continuous delivery/deployment.

But CI can face scalability challenges, see How can continuous integration scale for very large projects/teams?.

Splitting work on multiple smaller pre-integration branches would allow work on those branches to be agile, but then merging of those pre-integration branches needs to be handled. And those would be massive, unpredictable merges, rather difficult to plan and incorporate into an Agile framework. Until the merge is actually performed one can't really tell if it'll take 2 hours, 2 days or 2 entire sprints to complete.

So how can Agile methodologies still be applied to massive, monolithic software projects with 100s/1000s of developers?

030
  • 13,383
  • 17
  • 76
  • 178
Dan Cornilescu
  • 6,780
  • 2
  • 21
  • 45

4 Answers4

4

Well, you won't be able to apply an Agile methodology to a large team. One of usual principle is to work on Pizza Team (Less than 10 persons which can share a large pizza for diner together) because the lack of formalism advocated by Agile process makes it hardly applicable to a large team, the information would be scattered and some would be lost in the crowd.

So before thinking about the CI system, you'll have to review the organization around the software you build.
This mainly need to refactor a large project in smaller independent pieces, this could be separate softwares (and you'll take the path of microservices) or it could just be an architecture design within the software, where a class/object/library is given to a small team. This team has to document its service input and output (service contract/service orientation).

Once this is done you can split the responsibility of each independent piece to a smaller team which will be able to apply an Agile process.
This may solve the CI/CD problem, splitting the code base would allow each team to follow it's own pace, as long as it's interface is properly versioned and that the input/output are fixed for a type of call you can make parts of the product evolve while some other part keep a previous behavior.

The main drawback is that the service contracts from each part must be compatible between versions in use, which need a dependency system to know which part use which version of what before dropping an unused version of an API. This brings problems to make breaking changes and usually need a careful transverse planning between producer and consumers.

Tensibai
  • 11,416
  • 2
  • 37
  • 63
3

We went from a large, monolithic backend application worked on by 100s of devs into docker based microservices, deployments using ansible and similar DevOps fluff. Here are some notes on the process:

  1. Determine the domain boundaries inside your monolithic app. Make sure calls between boundaries happen as less as possible (by call I mean either in-process calls, or database access).
  2. Split the devs into teams, and assign each of them one slice of the domain. (It is possible that one domain will be worked on multiple teams as well)
  3. Specify cross boundary calls still happen, and define an API around it. Make sure only this intermediate library is used when doing cross domain calls.
  4. Have proper tests around these intermediate libraries, that check that they are working properly.
  5. Split up the code based on these domain.

Now, each of your teams should be both smaller, and work on their piece of code without worrying too much about the others. The tests you put into these intermediate libraries should cover regressions from outside callers, and the internal code is your responsibilty. Obviously if the intermediate libraries needs changes, or if your tests tell you that you are not adhering to the API specifications anymore (which can, and will happen occasionally), then you need to start talking to the other team(s) using these APIs.

Once you are happy you can try to replace these intermediate libraries to be HTTP API calls instead of in-process calls. Once this is done, you can actually split up your large app into real, smaller pieces (they will probably still not be "micro" services, but much-much smaller and more manageable). Also they will at least belong to one specific domain, and also have a clear, well defined API that others (including new microservices mostly free of your legacy big app) can use to obtain their data.

SztupY
  • 1,597
  • 1
  • 16
  • 18
2

Yes, splitting the bigger team into smaller, agile-size sub-teams is obviously necessary. But it's far from sufficient to also be performant overall, at the bigger team level.

With a highly scalable CI system in place it is possible even for very large teams to use trunk-based integration, in a single master branch, thus completely avoiding the intermediate integration branches and the unknowns/risks/delays/costs associated with their merges which would otherwise represent serious obstacles impacting the sub-teams. See How can continuous integration scale for very large projects/teams?

With these obstacles out of the way the sub-teams can really make the best out of the agile techniques as they're automatically all on the same page and benefit from all advantages of the CI methodology.

Such approach works equally well:

  • for monolithic codebases/producs which can't/won't be split into micro-services for whatever reason
  • for codebases/producs split into micro-services, even if for reasons other than independent deployment capabilities - which don't want to deal with inter-service dependency management and prefer to stick to monolithic deployments (they still share the same development branch)

Note: the actual SCM repository is irrelevant, it can be any combination of repos managed together as a single one, the term branch denotes just the common logical representation of whatever physical/actual repo branches exist in that combination.

Community
  • 1
Dan Cornilescu
  • 6,780
  • 2
  • 21
  • 45
1

I've been working on couple of Agile projects with over 100 developers without any problems, so from my personal experience, to deal with such big workflow, here are some suggestions:

  • Team definitely should to be split (max. ~12 people each).

    Make sure you've the right people with the right skills in each team, so they're independent (cross-functional teams). DevOps doesn't have to be part of development teams.

  • Each team should work on their own components/modules, so their work won't conflict with other teams.

  • Once feature/component or bug is completed and tested in separate branch, developer should be responsible for sending and maintaining a valid mergeable Pull Request to be approved by other team members.
  • Often commits and pulling from master branch will avoid any surprises about code conflicts.
  • Each Pull Request should be automatically tested in CI and once approved, should be merged (either development or master branch).
  • Deployment to development branch should be done on each Pull Request being merged and tested on development environment by automated tests and manually, so only stable code can progress upstream.

To summarize, the main point in order to avoid spending too much time on merging complex branches, introduce Pull Request, so each requester is responsible for mergeability of his own code and other developers are responsible for reviewing the change, so no any unpredictable branches or conflicts are possible in this case.

So yes, Agile methodologies can scale for massive scale projects.

kenorb
  • 8,011
  • 14
  • 43
  • 80