Beyond Features: Reorienting to what matters most (Chapter VIII)

After several conflicts, Björn and Ivar understood the heavy burden of accountability. Creating output is easy and foreseeable, but delivering outcomes is daunting and unpredictable. 

Want to read the full Series? See here for the link to: Chapter 1, Chapter 2, Chapter 3, Chapter 4, Chapter 5, Chapter 6, Chapter 7

Teams Kattegat and Jorvik are learning the hard way what it takes to inspect and adapt, yet they struggle to find the balance between discovery and implementation. Meanwhile, stakeholders keep pressuring them for more. Luckily they have Lagertha to light their way. 

Three Sprints have passed, and both teams managed to put new features live. Now, tourists can recommend their friends in exchange for an exclusive VIP tour, and partners can self-onboard. Still, something was missing: RESULTS.

Despite the achievements, the customer acquisition cost is still skyrocketing, and partner acquisition remains slow, which leads to a lack of competition.

Impatient, Ecbert called a meeting with Lagertha, Björn, and Ivar. He kept it short and, emotional, he presented his dissatisfaction with the results. Loud and clear, he demanded: “I empowered you to deliver results, it’s been almost a quarter, and beyond features, I see nothing. I give you a month to change this scenario. If nothing changes, we’re back to our previous way of working. At least we got more features live. I don’t want excuses, I want the promised outcome.”

Lagertha, Björn, and Ivar were speechless. Björn used to react to such strong claims, but this time was different; he knew team Kattegat was empowered and still couldn’t achieve the desired result. Again, they had to step back and understand why the numbers were not changing despite validating their assumptions before implementing features.

Inspecting

Björn and Ivar hoped Lagertha would have another card up her sleeves. With such pressure, they had one last shot to prove the trust they were given wasn’t a waste. They went to a meeting room, where Lagertha drew a line on the board and asked Björn and Ivar to walk her through how they got there.

Björn took the initiative, went to the board, and adjusted the timeline. He kept nodding as though something was wrong, but he couldn’t figure out what it was.

Lagertha asked Ivar to share what team Jorvik did before she presented her questions. Ivar didn’t hesitate and shared the timeline for Team Jorvik. He was honest; as he pulled the timeline, he remembered how chaotic the result of the first Sprint was, but anyway, he put that as a milestone.

Lagertha asked them to look at the timelines and reflect on the following:

• Was the evidence convincing enough to implement?
• When you validated your assumptions, did you try to confirm or uncover what you didn't know?

Silent dominated the room for the next few minutes. Björn seemed thoughtful and Ivar uneasy. Lagertha let the silence sink until someone would break it. And that happened when Björn said:

“I screwed up! I asked clients if they would recommend us, showed them the prototype and asked how they would use it. I confirmed what I wanted to know. I made them biased. Then, I got excited and led the team to implement the whole solution.”

Ivar: “At least you talked to customers. After I messed up in the first Sprint, I talked to our partner managers and sent surveys to our current and potential partners. I didn’t interview any of them as the survey confirmed my assumptions. We developed self-onboarding after that.”

Lagertha: “It seems you created weak evidence and a biased audience. An interview or survey may give you direction, but it’s not enough to justify implementation. And the validation was flawed, as you mentioned, Björn. The question is, what can you do now? We’ve got a month to change Ecbert’s perspective.”

Björn: “I don’t know. I’m failing as a Product Owner.”

Ivar: “I won’t say, ‘I told you so.’ We must be creative and open-minded. The problems happened because of our confidence, and we may have given too much attention to what we wanted to see.”

Björn: “You’re right, Ivar. I wanted customers to recommend us, but I didn’t try to learn from them, and I don’t even know if they understood our flow. I think we need to observe them somehow and learn from their actions instead of their words.”

Ivar: “That’s it! If we learn what’s holding them back from using our new features, we may figure out what’s happening and adapt to what makes sense for them.”

Lagertha: “I like the direction you’re going. What about you both get together with your teams and define actions to uncover the hidden opportunities?”

Björn: “Yup, I’m going to do that. I already have some ideas: screen recording, heatmap, link tracking, etc.”

Ivar: “Me too. Let’s understand what’s wrong and change it.”

Adapting

Once again, confidence got in the way of creating value sooner. Weak evidence and strong biases fooled both Björn and Ivar. Now, they have no room for mistakes; they had better figure out how to change the situation, or return to their old Waterscrumfall working mode.

Björn and Ivar spoke with their teams; they wanted to understand why customers weren’t using the new features. As Björn brought this up to the team, Torvi immediately challenged him: “Do you want to understand why they don’t use our feature or what is important for our customers?” Björn realized that the focus was not on making the feature work but on helping customers get their job done.

Team Kattegat decided to do the following:

• Add a heatmap to the recommendation feature
• Track each step of the flow of recommendation
• Whenever customers are about to bounce, ask them: “Did you give up on earning a VIP Tour? Could you share with us the reason for that?”

They decided to run this experiment for two days and adapt according to their learnings.

Ivar took a similar direction with Team Jorvik and received the same pushback from Ragnar: “Do you want us to understand how to force clients to self-onboard or learn what matters for them?” Ivar also understood the importance of reorienting to their goal of partners being able to be onboarded without anyone else’s involvement. Team Jorvik took the following measures:

• Track the time to fill the onboarding form
• Measure the conversion rate
• Whenever potential partners are about to bounce, ask them: “Sad to see you go, what could we do differently to keep you?”

Team Jorvik decided to run the experiment for two days, and Ivar talked to Haraldson to figure out how to get more traffic. That would help create more insights.

Both teams understood how critical the situation became. They stopped everything else, and the experiments were running in a matter of hours. After a day, some patterns emerged:

• Team Kattegat: Clients didn’t see the recommendation feature. The heatmap showed that only 5% got there. The ones landing on the page didn’t understand what that was about; the common question was: “How do I get a VIP Tour? I want that.”
• Team Jorvik: Instead of taking 3 minutes to fill out the form, partners took 19 minutes on average. The bounce rate was as high as 75%. A common wish presented: “It would help me stay if you could read my data from my company registration. Filling out this form takes too long.”

Were these insights signals or noise? Could Björn and Ivar adapt immediately? 

The previous experiences taught both teams something. Don’t invest too much when uncertainty is high.

Björn said, “Let’s do something quick and dirty. Clients don’t get to our recommendation feature. We need to make it prominent. They don’t understand it, so we need to make it simpler. Can we hack something and go live?”

Torvi: “Yes, we can. But let’s do a split test and learn incrementally.” 

Björn agreed, and the team would hack some changes, quick and dirty.

Ivar was even more radical. He understood clients wouldn’t self-onboard with the form; that was a flawed assumption. But they were still interested in Odin Enterprises. Ragnar said: “We learned something. Partners have this company registration. We just don’t know which format that is and if we can import data from there. What if we do a self-backed solution? They send us the registration, and we manually put the data in the system. With that, we will understand if they will do it and if we can automate it later.” Ivar loved the idea.

Poor Solutions; Great Learnings

Teams Kattegat and Jorvik concluded their changes and gathered more data to learn from their experiments. Surprisingly, the numbers started to change.

• Within a more prominent referral problem, 40% of customers arrived on this page
• The bounce rate reduced 50% after team Kattegat added a banner, “Get a free VIP Tour by bringing your friends to us. No hidden stipulations.” 
• Partners sent 10x more applications with company registrations than filling out their forms
• Reading company registration from partners would require some effort, but manual input was doable by Haraldson’s team. The goal wasn’t to fill out the form but to get the partner in

Neither team Kattegat nor team Jorvik had a complete solution, but they hacked their implementation, and the results improved. They learned how to observe actions and adapt to what customers do instead of what they say.

After a quick shock to the system, Björn and Ivar understood what it takes for teams to focus on the outcome. 

Change the solution if it doesn’t serve your audience. Look for signals instead of confirmation biases.

As they may reach their initial goal in one or two Sprints, what will they do next? Have they figured out the secret to creating outcomes beyond features? Are they ready to walk on their own?

Wait until the next chapter to understand if teams Kattegat and Jorvik are ready to remain empowered or if Ecbert will fall back to the old command & control style. 

‍