Agile is said to be iterative-incremental, but where is this "revisiting" and "refining" if we follow Definition of Done?

Question

First of all, I found it a bit strange because technically, iterative means something that is done as a recurring series of steps - so how can one be purely incremental, when each increment will be implemented through the same series of steps?

Anyway, my main confusion stems from the word "iterative" - if the meaning is enahnced to mean "refining and revising work", where does it fit in the classical story development scenario: you develop story B, and once it passes the tests and meets your DoD and Acceptance criteria, it is Done. You added it on the top of the existing baseline. In the mean time, some other stories are finished, ideally Done and hence certainly not meant to be refined or reworked... Where is the iterativeness?

So, to me it does not seem different from the decades old incremental models (which some claim are examples of Agile models, some say they are not). Just developing features one by one, with early testing and frequent delivery of working increments.

Another interesting point: Agile is apparently said to be "one of the incremental models". However, incremental models are defined by quite a comprehensive upfront documentation...

Answer 1

It seems like you are mixing several things together, which is why you're confused.

Iterative and incremental development refers to how the system as a whole is built. Contrast typical methods that are associated with Agile Software Development and associated techniques and principles like incremental design, YAGNI, and refactoring with "waterfall" techniques that embrace getting all of the requirements upfront, then spending time architecting and designing for all the requirements, then spending time turning the design models into code, then integrating and testing the system. Instead of trying to do everything for the whole system up front, you'll be repeating the activities for each slice of the system.

When you look at individual units of work, just because you've developed, tested, and shipped something doesn't mean that you won't revisit it. There are a few different reasons why you may need to. One would be that you get feedback that what you built is no longer quite right and you need to revise that feature. Or perhaps you are building a related feature and need to revisit your design decisions to support both, yet keep a maintainable system. Maybe the team's technical maturity grows and an evolving Definition of Done means that there's now technical debt to be resolved.

Answer 2

You're right that agile software development is not necessarily radically different from other software development approaches. The “radical” idea in agile is to Do Things That Work, which usually means adapting the process to make it work better, and to listen to the people doing the work (“people over processes”).

Under the agile umbrella, we have specific process methodologies such as Scrum. Scrum is definitely iterative and incremental, and suggests feedback loops of various levels. In a typical instantiation of a Scrum process, you'd see feedback loops like the following:

• retrospectives
• sprints
• standups
• TDD red/green/refactor cycles

In Scrum, the main feedback loop is the sprint. Each sprint has a defined duration and defined scope. At the beginning, we determine some tasks that should be Done during the sprint. At the end, we see what was Done. Did some things fail to get Done? During the retrospective, there's opportunity to understand why and adapt as necessary for the next sprint. Then we repeat. Since we might adapt between each Sprint – and in particular, the Product Owner may change priorities based on the experiences – it is not appropriate to plan in advance multiple sprints. One step at a time.

You are entirely correct that individual tasks (backlog items) in Scrum are not handled in an iterative manner. A task is either Done or it's not. This is considered important in order to manage clear expectations with the Product Owner, but also to demonstrate clear progress to the PO.

The entire point of Scrum is that many projects do not make clear progress because a lot of tasks are only mostly done. This delays value, and makes it difficult to close feedback loops. The idea is that with Scrum, there will be something Done each Sprint, thus delivering clear and predictable value to the PO. And since the PO can re-prioritize between Sprints, this maximizes the value delivered across the project duration.

When you say “incremental models are defined by a comprehensive upfront documentation” then this is wrong. There are processes without a focus on documentation that are clearly incremental, such as Scrum. It's also perfectly OK to prepare lots of upfront documentation in an agile project, if that documentation has value (though in many cases, a Spike is more useful to find a good design than to do lots of upfront analysis). This circles back to my original point: agile is about finding a process that works. In this viewpoint, being agile is about values that inform how you select and implement processes, and are not an inherent attribute of processes themselves. (Indeed, most Scrum instantiations are probably not agile in any meaningful way…)

Iterative/incremental processes were pioneered in environments with very high integrity requirements and thus a focus on documentation and testing. In particular, some incremental processes involve an entire Waterfall-like process per increment, including analysis and design phases. This isn't wrong, but this also isn't the defining feature of iterative/incremental. The defining feature is that functionality of the project is split across two or more increments, thus de-risking the project and allowing for early feedback to be integrated. This is broad enough to cover both defense projects that start with a 1000-page requirements document and a three-person web consultancy doing Scrumban.

Answer 3

I think your confusion stems from what you associate the word "incremental" with; I get a feeling that you're thinking about rote accretion of features (software simply growing by addition of new things, and piecewise delivery). That's not what's meant - it's about design, about deciding what to build and how.

The problem with waterfall is "big design upfront". This may work when the problem domain is very well-understood (e.g. there are years of research, how things work and interact is known, there are standard time-tested practices, etc.), but it's not iterative or incremental in the sense that pretty much everything is decided upon and specified in one phase. (Now, you could call it incremental in the sense that you're probably building it in multiple steps, in an additive process, but that's not in itself especially interesting.)

Agile recognizes that in many cases for business applications the problem domain is either sufficiently unique or isn't well-understood (in the above sense), and that doing "big design upfront" when you don't know enough about the subtleties of the domain leads to expensive projects that in the end don't do what clients wanted (you build "the wrong thing"), or do it poorly - assuming they weren't overly ambitious and were successfully built at all.

So agile opts for "some design upfront", and then some more design at each iteration, as the developers learn more and get feedback. Agile is trying to hone in on a good solution over time, starting small and retaining design plasticity for longer. Another aspect is that agile tries keep the product useful/usable at every iteration. It also strives to learn what the most important (core) aspects of the system are, and tries to push those to be implemented first (or as early as possible). This combination reduces the risk for clients because they can stop the project when they feel is appropriate (e.g, when they reach the point of diminishing returns, or when they run out of budget), and still get the most value out of it - they still end up with software that is actually useful for them. (Note that there's a business aspect to this as well, you have to have a business model that can support this approach; e.g. you can't have scope, cost and time all fixed, etc.)

The revising aspect is going to come from the the project team developing a growing understanding of the domain, and from frequent feedback - from uncovering new things, or identifying what was misunderstood from (or misspecified in) the requirements, or what was assumed and never explicitly said, etc. It's the moments that make you go: "Oh, so that's what you actually want!", or "Ooooh, this process actually works like this!", or "So, when you say 'shipment' you don't mean the same thing as when your colleague from a this other department says 'shipment'???".

So it's really about incremental delivery of value (clients find product usable/useful in some way after every iteration), and the ability to learn and revise on the go. (Or, if you want to say it differently, it's about being incremental and iterative simultaneously - which then implies that you have to be incremental in the particular way described).

In contrast, a waterfall project might spend a long time in a half-baked, work-in-progress state, and can proceed for a long time before any feedback is provided (or even possible).

Answer 4

Your story meets the acceptance criteria, so it’s done. You will never, ever go back to that story. Absolutely right. That doesn’t mean you never change the code, and you never change the application.

Say your acceptance criteria said there must be a green button on the screen. You do it, QA checks the button is green, story is closed. Next day a customer complains: The button should really be orange. Yes, they said “green” in a meeting, but now they want orange. You don’t re-open the closed story, you don’t re-visit the story. You create a new story “button should be orange, not red”.

You refine stories that are not closed yet. Once the story is closed, it doesn’t get refined anymore, a new story is created or another story is modified.

Answer 5

Incremental

so how can one be purely incremental, when each increment will be implemented through the same series of steps?

A dictionary definition will help here:

Increment - NOUN
An increase or addition, especially one of a series on a fixed scale.
‘all sizes from 4–30 mm in 1 mm increments’

In terms of software development, "incremental" is taken to mean "not all in one go". It is the antithesis of "if you build it, they will come", and it's more akin to "build what you need, don't build for possible future versions".

It means that if your application has a FooService class, that there is no expectation that someone will spend X time making the entire FooService class (let's assume it has features A, B, C, and D), at which point it is considered "finished".

Instead, someone working on task 1 might create a FooService class because they need the A feature, so they develop only the A feature. Then, at some later time, someone working on another task finds themselves in need of a B feature, and FooService is the right place to put it. So now, they develop the B feature. At another point in time, someone might need a D feature, so they add it. Maybe after a while, you notice that B's implementation was very simple but because it is now being heavily used, it needs to be more robust, so you upgrade B to B2.

Little steps, one after the other, as opposed to knowing precisely what you're going to build = incremental development.

If an analogy helps: if you're expecting to hold a blueprint of the entire house before you start building it, you're not doing incremental development.

Iterative

the word "iterative" - if the meaning is enahnced to mean "refining and revising work", where does it fit in the classical story development scenario: you develop story B, and once it passes the tests and meets your DoD and Acceptance criteria, it is Done. You added it on the top of the existing baseline. In the mean time, some other stories are finished, ideally Done and hence certainly not meant to be refined or reworked

If you look at a particular feature in your software (let's call it A), that was probably created because of a story.

But when that story is done, that doesn't mean that you will never touch A again. A new story might be written, which brings some requirements that lead to A needing to be changed, extended, or even made obsolete.

some other stories are finished, ideally Done and hence certainly not meant to be refined or reworked

A story should be finished and then not touched again. But the source code that was related to that story doesn't get that same expectation. Just because these lines of code were related to story A doesn't mean that there isn't (or won't be) a story B which also influences what this code does.

Stories are rigid, and will be finished and done with. The code that was written for that story, however, is a living entity which should gradually be changed/updated to keep adapting to the ever-changing needs of the application.

Agile

So, to me it does not seem different from the decades old incremental models (which some claim are examples of Agile models, some say they are not). Just developing features one by one, with early testing and frequent delivery of working increments.

There is a difference between developing one step at a time, and planning development one step at a time. Agile pushes more towards the latter, more so than older models.

The goal is to not be waterfall, i.e. not trying to immediately build a comprehensive solution. People are notoriously bad at guessing the future, and waterfall specifically asks you to guess what you will need for the foreseeable lifetime of your application.

The odds of you spending time and effort building something that you end up not needing, or needing in a different way than you once anticipated, are so high that it makes the initial effort (and mental load that accompanies it) not worth it. At least, according to Agile.

Keep in mind that what I describe here is how Agile sees the world. Not every model works for every scenario.
I've worked on software projects involving major infrastructure safety, and the waterfall model is paramount to ensure that all involved parties are interacting with each other correctly, down to the strictly versioned chipsets in each PLC controller.
It leads to a more rigid process and severely decreased flexibility in development, e.g. you're not allowed to quickly update the chipset's functionality, but when peoples' lives (and billions of dollars in repair costs) are at stake, flexibility is not the most important priority.

Answer 6

The iterative process isn't iterative over a single story, it's iterating over a series of sprints. We do a sprint, we evaluate how the sprint worked, we make adjustments, we do another sprint, we evaluate how the sprint worked, we make adjustments, and so on.

From the perspective of the customer, they have a list of product requirements. We break those requirements down into small chunks (epics, stories), we organize these chunks into a backlog of work, and then we iterate over those stories, delivinering incremental functionality with each iterations.