Skip to main content

Experimental Cycles

Friday Ship #317 | September 23, 2022

daniele-franchi-GbAEJUJKJ88-unsplash

This week we set up new squads for a new trimester. Each squad kicked off its first sprint to explore its problem space.

Based on Parabol’s Squad Model, we’ve just finished our first a cycle of working as multi-disciplinary squads. It’s time to share what we’ve learned.

Each squad worked on a clear goal aligned with Parabol’s strategy. Squads were either figuring out how we could make the meetings more meaningful or trying out more product possibilities.

As members of squad 2, our goal was to quickly experiment with new ideas to increase user conversions. We learned some lessons on how to better experiment quickly, starting with the goal of “Early Drop off Reduction”, and reducing frictions to adopting features that drive usage.

What We Learned

1. Analysis: We reach out to our users

We really like the feedback we get from our users and consider it an honor. 🏆

We learn how to see users, listen to users, connect with them, and iterate and optimize based on their feedback. The silent majority may not find a place to express themselves, so we also look at the data we have accumulated over time and from targeted user surveys.

Early churn feedback survey results (grouped by category)

But the data should not fully representative of users, we did user interviews, sent email surveys, and watched their session replays as well:

  • We did face-to-face user Interviews in the initial drop-off research, which included two types of users: churned users and non-users.
  • We also did surveys via email, asking them sincerely for what reasons they left us.
  • We looked at users’ session replays to observe and determine where people get stuck. This helped us see if there’s a common area where users get stuck the most.

2. Ideation: We clarify our hypotheses

We then leveraged the data we collected to create experimental ideas with clear hypotheses and supporting evidence.

Our goal was to focus on retention and enabling users to experience as many of Parabol’s product features as possible.

This included:

  • Minimizing friction
  • Building a funnel model

The most important thing was to reduce the barriers that users encountered before they experienced the core functionality (”aha” moment). To minimize friction as much as possible, we itemized all the actions to be done before a user converts. Then we added objectives and hypotheses.

  • Objective: How does this idea contribute to improvement?
  • Hypothesis: What is the expected outcome you predict from this idea?

For example, we like to allow users to experience our demo meeting as soon as possible, so we displayed the demo meeting placeholder card instead of an empty state on the /meetings page, so that users can click on this when they enter the product.

Early churn experiments backlog in Notion

3. Prioritization: We scored experimentations based on OKRs

With the team’s goals, we were able to choose our own squad goals. For example, we defined our own OKRs and created an Amplitude Dashboard for each key result, so we could continuously track how well the newly launched experiments were working.

Early churn testing dashboard in Amplitude

4. Testing and measurement: We learned a faster way to run experiments

The goal of this stage was to see if an idea brings any positive changes after implementation.

Measure Weekly:

We reviewed the results of previously live experiments on a weekly basis and recorded the data in Notion’s database view. This helped us evaluate experiment outcomes on an ongoing basis.

As we experimented, we also discovered that we needed a faster way to do experiments. We discussed the basics of preparing an A/B testing framework. To bring the best ideas to the surface, we had to reduce the cost of putting them to the test.

In the next trimester, we will bring it all to fruition, using proven product experimentation capabilities to realize the value of Parabol’s product.

Repeat

After the 4 steps above were finished, we did a retrospective analysis to make a closed cycle.

“The more experiments you run, the more you learn.”

With the start of the new trimester, we have re-established our goals and have re-formed our squads. The goals also are realigned so that we can be more focused.

We want to see what the barriers are for users to use the product and want to find out what surprises and satisfies them the most. We want to build products that are enjoyable and make work more meaningful.


Metrics

After some astounding growth, our monthly web sessions and new users dropped back to levels we saw one month ago. Meanwhile total registered users and MAUs continued increasing at a steady clip. Weekly meetings were slightly down this week, with the trend replicating across almost all meeting types.

This week we…

…published a blog on Scrum Values featuring an interactive self-assessment quiz.

…published our random icebreakers tool as a Figma widget!

…continued hiring for our open Inside Sales Rep and Head of Public Sector roles

…began interviewing Parabol users about their experiences using our product

…held a retreat for the Exco team in San Diego

Next week we’ll…

…welcome the Exco team back from their retreat

…continue interviewing Parabol users to learn about their experiences

Jimmy Lv

Jimmy Lv

Jimmy specializes in Web development. He worked at Thoughtworks, is passionate about Agile software development and training, and built collaboration tools of DingTalk at Alibaba. He loves helping others learn while learning and is a true believer in the tools of thought, exploring and building the future of work. Jimmy lives and photographs in Chengdu, China, but would like to travel the world with his wife.

All your agile meetings in one place

Run efficient meetings, get your team talking, and save time. Parabol is free for up to 2 teams.