Continuous Integration : which frequency?

Question

I've always launched builds after each commit, but on this new project, the architects just asked me to change the frequency to "one build every 15 minutes", and I just can't understand why that would be a good reason vs "building on each commit".

First off, some details :

• Objective-C (iOS 5) project
• 10 developpers
• each build actually takes ~1 min, and includes build and unit testing.
• the integration server is a Mac Mini, so computing power isn't really a problem here
  • we use Jenkins with the XCode plugin

My arguments were that if you build at each commit, you can see right now what went wrong, and correct directly your errors, without bothering the other devs too often. Plus our tester is less bothered by UT errors this way. His arguments were that devs will be flooded by "build error" mails (which is not completely true, as Jenkins can be configured to send a mail only for the first broken build), and that metrics can't be done properly if the frequency of builds is too high.

So, what's your opinion on this ?

Answer 1

Fail fast is a good principle - the sooner you know the build is broken, the sooner the offending commit can be identified and the build fixed.

Build on every commit is the right thing to do.

Building every 15 minutes can be pointless if the project has a high volume of commits within such a timeframe - tracking down the bad commit would take longer and may be difficult to determine (one might also be in a situation where multiple commits have different things that break the build). There is also the possiblity that in quiet times (night time?) you end up rebuilding though no changes have been made.

If the build breaks so often that it is a problem, the answer it to re-educate the team on the importance of not breaking the build and in techniques that ensure this does not happen (frequent fetches, checkin dance, compiling and running unit tests locally etc...).

Answer 2

There is literally no point in doing a build every 15 minutes if nothing has changed. But equally there is no downside, afaik, jenkins will only e-mail on fail and then success and not everything in between (eg 10 fails).

We do it on every commit. However we do poll the repository every fifteen minutes and check for changes, maybe that is what yr colleagues are referring to.

You expect your 10 dev's to be committing more than once every fifteen minutes ? That sounds like rather a high estimation. 10 dev's means that after every 150 minutes the same person is committing again, so thats 2.5 hrs. So in your average day each dev commits 3 times. Personally I do one commit ~ 2 days ... sometimes more sometimes less.

Answer 3

It's going to flood developers with mail more if it's every 15 minutes only. That's because it won't know for certain who broke the build and thus mail more people.

As for metrics, if it's really a problem—which I can't tell because I don't know what metrics they think there is a problem with—, you can always make another job for collecting the metrics.

Answer 4

Perhaps the requirement shold be "build at most once per 15 minutes". This could make sense for projects with very frequent commit activity (i.e. multiple commits within few minutes) and perhaps long build times. Of course it depends also on how the builds are used. For testers it might be somewhat confusing to get multiple builds within 15 minutes ...

But I agree that it makes no sens to build if nothing has changed.

Answer 5

Some devs want to be allowed to do commits in a way where the files belonging to one functional change are not committed in one single, atomic procedure. They need two or three minutes to do a "logical commit", which consists of some "physical commits". This is typically the case when your devs directly commit to a central repository, not using a DVCS.

For this cases, it may be a good idea to let your CI server wait some time after the last commit before starting a new build. But 15 minutes seem to be a very high number, 5 minutes or less should be enough.

Or, better(!), try to guide your devs to work only in small portions, only one thing at a time, making it much easier to do only "functional complete" physical commits. Then a build after every commit will work seemlessly.

Answer 6

Even if you have Jenkins set up to build on a source control commit to a project or any of its dependencies, this does not prevent a developer from deploying to the artifact repository without first committing to source control. If they deploy an uncoordinated API change or a bug in a dependency to the artifact repository, this can break your build without triggering the Jenkins job. I personally would want to know about this ASAP.

Ideally you would build for every commit and also on a schedule to check for situations like I just described. This would conceptually mean you build at least once every 15 minutes.

If you have your Jenkins jobs set up to run on dependency artifact deploys (and if you do...Bravo), you can prob axe the scheduled build if your unit tests are solid (meaning they don't test the dependencies) and your integration/functional tests are not part of the Jenkins job.

As far as the "flood with email" problem, I say getting flooded with email on a broken build is a good thing. Make sure you force developers to respond to the email with a description and you will see your broken builds go way down, trust me.

Answer 7

You said you can't understand their reasoning, so you have to ask them. Don't just guess at what they want and try to implement that, and certainly don't just implement what they asked for without understanding why they asked for it.

That said, to answer the general question, it depends on the development philosophy you use. If every commit is expected to work, then every commit should be tested. If you use a DVCS with the philosophy that every push should work, then test every push.

In general, it's better to tell people something they don't know, then to repeat what they do know. For example, if they want to reduce the number of redundant emails they get, tweak the email settings, not the build frequency.