Starting with Scrum: 3 - Backlog Refinement

This is the third article in the 'Starting with Scrum' series. In the first article, we discussed how to get started with Scrum by determining what the Product Goal is. In the second article, we talked about how to translate the Product Goal into a backlog. In this article, we pick up where we left off and focus on the next step: refining the backlog so we can get started with the first sprint.

Want to read the full Series? See here for the link to: Article 1, Article 2

What is Product Backlog Refinement?

According to the Scrum Guide: 

"Product Backlog refinement is the act of breaking down and further defining Product Backlog items into smaller more precise items"

But what is the point of this? In my opinion, it has everything to do with managing complexity. 

Breaking work up into smaller parts reduces complexity. In the previous article, we saw how to break up the Product Goal into manageable parts. With this initial overview, we were then able to determine what the most important part would be to start with.

In the Product Backlog Refinement (I’ll just call it Backlog Refinement from here on) the team can focus on that most important part and break it down even further. The goal is for the resulting items to each be small enough to fit in a Sprint, allowing the Developers to complete work in short iterations, focusing on a small part of the work in each Sprint.

This will of course be familiar to most, as it lies deep in the heart of Agile. This is the empirical approach. Its benefits are best illustrated by this diagram:

Note that in the quote above, the Scrum Guide also mentions making work items more precise. This goes hand in hand with making work items smaller. In fact, it is a form of the famous military strategy of divide & conquer: Instead of tackling the full complexity of the product all at once, the team focuses on a smaller part of the work. This allows the team to concentrate their energy and expertise, making it that much easier to gather an understanding of the purpose of the work and to develop the necessary confidence to deliver.

These two aspects, making work items smaller and more precise, are what I like to call the rational argument for Backlog Refinement. However, another very interesting aspect is how having small precise work items is so important to experiencing flow.

The simple fact of being able to move work items to Done frequently and efficiently is surprisingly important to achieve a state of flow. We often focus on the big things, but sometimes just a little tweak in the process itself can have a profound effect.

In Practice

For me the key aspect of Backlog Refinement is self-confidence. As a Scrum Master, it may be my goal to separate the what (what problem are we solving) from the how (how do we solve the problem, something I’d prefer to leave for the Sprint Planning) in the interest of Just-in-Time (JIT) efficiency, but it is the self-confidence of the team that determines how this divide evolves.

Scrum Teams will often spend a lot of time in the Product Backlog Refinement discussing how they will deliver a story. They will agonise over all kinds of possible problems and challenges, trying to create certainty for themselves, trying to build the necessary confidence. And they will damn well want to figure that all out before they give you an estimate!

It is my job as a Scrum Master to help them discover that with complex work, the certainty they are looking for is an illusion, and true certainty can only be achieved by actually delivering the work. It takes time for a Scrum Team to become comfortable with uncertainty.

The trick is to continue the same approach I described in the previous articles, simply asking the team what needs to be done and using the 1–2–4-all technique to facilitate the process. Keep the team in the flow and make sure it all feels like a casual game.

We are basically trying to keep the team distracted from the lack of certainty, their bad experiences with Scrum and Agile, and even their misconceptions about Scrum. Incidentally, that is the reason I haven’t mentioned user stories yet.

But where does this end?

In my experience there will come a moment when we reach a level of detail where the what and the how coalesce: the what is so clear that the how becomes implicit. You will notice it by the way the Developers suddenly become very eager to get started. It is the moment when their stress about the uncertainty disappears, simply because the work becomes so clear.

This is the moment to choose a name together with the team for that last set of work items, or tickets, or even just PBIs. The most popular option is of course user stories. This will often trigger a discussion of whether the work items are adequate (it’s not written in the right format!). The key is to encourage the team to realise that it is the joint understanding that is most important. Everything else, all those words, names and concepts like INVEST — these are just tools that can be considered as the team gains experience with their process. 

One of these tools is estimation. As you may remember from my previous article, after each round of splitting up work items we have estimated the work items using Story Points. So what makes estimation special?

Estimation

Of course, I can’t discuss Backlog Refinement without talking about estimation. For many, Backlog Refinement is all about estimation. At the same time, many in the Agile community see estimation as evil, mainly due to how easily they can be misused.

Assuming a non-toxic use of estimates, I like them for two reasons:

First of all, they’re a way to test the shared understanding of the team. If the team agrees on an estimate, it builds confidence that there is a true shared understanding and that there are no hidden assumptions that still need to be unearthed.

Second, estimation helps the team learn to size work. This is an invaluable skill. At the most basic level, it helps me answer the question, “Will I be able to deliver this user story in one Sprint?”

That knowledge can be extrapolated, helping the team manage their work as a whole. Beyond the concept of flow we talked about in the beginning of the article, there is also the possibility of roughly dividing the backlog into sprint goals. And there is nothing wrong with a team working towards a deadline of their own making. In fact, it can be hugely effective for their focus and their sense of flow.

Again, this is all in the context of a non-toxic environment where the estimates are for the team’s own use. Beyond that, it is useful for making rough delivery projections for stakeholders, as long as all parties involved are very clearly aware of their inaccuracy. Think weather prediction instead of deadlines. 

Mike Cohn offers some very handy techniques for dealing with projections and visualizing error margins in his book Agile Estimating and Planning, if you would like to learn more about it.

As for me, the trick here is the same as before — you may be seeing a pattern here — keep it fun, make it feel like a game! To me, the beauty of Story Points is that they abstract estimation to such an extent that it is easy to turn it into a game. 

We all know that estimation is a relatively inadequate tool. I find that its inaccuracy means that its value is minimal and therefore investment in estimation should be minimal. Instead, I think it is more important to make it as relaxed and playful as possible. 

Through the years I’ve noticed that estimation is generally inaccurate enough that it makes casual estimation just as accurate as formal Scrum Poker. So why put in the time and so much of the team’s energy?

Conclusion

In the activity of Backlog Refinement, I see Daniel Pink’s three pillars of motivation in action: purpose, mastery and autonomy. Let the Scrum team clarify the purpose of the work autonomously, to develop self-confidence, a sense of mastery.

Maybe the most important thing I’ve learnt as a Scrum Master is that Backlog Refinement works best if approached from the highest level of detail. Having gone through the whole breaking up of the Product Goal into individual user stories will give a team a sharp and clear overview of all the work. They will see each user story in the context of the whole, and this seems to mitigate the lack of detail at the user story level, the uncertainty about the how.

Perhaps the reason is rather less rational than I think it is. Perhaps, by working from the highest level the cognitive capacity of the developers - their RAM, so to speak - gets saturated so that they do not have enough space anymore to get sucked up into the details of each individual user story. Precisely the opposite of what happens with teams that only focus on the individual stories, whereby their cognitive capacity is saturated by the details, leaving no space for the overview.

Whatever the case may be, in my experience, teams are usually quite relieved to find how easy the whole Backlog Refinement goes when starting at the higher level of detail. It also seems to give them a huge boost in self-confidence, which allows them to leave the details of the how, the implementation, for the Sprint Planning. 

Which incidentally, will be the subject of my next article.