#398 – Team Insights Model Development

This week we evaluated the prototypes we developed during last week’s Product Team retreat to San Diego for our upcoming team insights feature.

It’s hard for us to believe but folks have held more than 6,000,000 discussions on the Parabol SaaS platform. That’s an awful lot of reflecting, brainstorming, and improving the way teams work. Since very early on in our company, we dreamt of having the capability to analyze these data at scale to give teams and leaders insights into the systemic issues that are challenging teams from being able to do their best work. The astounding rate of advancement of large language models (“LLMs”) has made what was once a difficult and costly engineering problem to something well within the realm of our small company to tackle on its own.

Last year, we began developing the very first LLM-based prototypes for transforming a team’s data into insights. Over the past few months we shipped a generalized architecture for transforming our data into formats that are more readily used by today’s LLM technologies (embedding vectors and plaintext representations, for the technical among our readers).

Last week our team gathered in La Jolla, California (near San Diego) to develop several prototypes on top of this architecture to evaluate approaches for extracting team-level and organization-wide insights from these data (we previously wrote about planning this retreat in Friday Ship #396). We rented a big house and spent nearly the entire week developing varied approaches and putting them to the test with real customers. Several folks at Parabol are fans of the card game “Dominion” so we offered a copy of the game as a prize to the top-performing model.

Models were developed on a common set of training data (including the Parabol organization’s some 8 years’ worth of meeting data). When it came time to evaluate model performance, novel data the developer’s hadn’t previously had access to was ran through each model and the model outputs were gathered and sent to customers for evaluation in a survey instrument.

While we’re not quite ready to reveal example insights publicly yet, we can show you what a piece of our model performance evaluation looked like. Below is a graph of 7 models developed during last week and their mean aggregate score as collected by our survey instrument. Here, the model codenamed “Radiance” was the top performer, just edging out “Skylark” and “Whisperwind.”

One noteworthy output of last week’s experimentation was experiencing just how advanced GPT-4o “omni” is as a general AI tool. Our top 3 performing prototype models had 1 thing in common, they all fed data directly to GPT with minimal preprocessing. Other models used more “traditional” processing pipelines involving the use of embedding vectors/distance searches, clustering, etc. None of these models outperformed “just sending all the data to GPT with a prompt.” Fascinating!

We were long overdue for some time to catch up and bond as a team. While we had a good time hacking away on our laptops at the Airbnb, we also made time for a little recreation. There are plenty of good options in and around San Diego:

Our time together was quite energizing and refreshing. It was a good reminder of how important it is to get together to work on a project, fellowship over meals, and have some fun. We feel the quality of our work and time together was outstanding.

We see some down metrics this week after a high in weekly meetings last week. It’s good to see Standups having steady growth over the past few weeks. We’ll keep an eye on traffic for the rest of the month to ensure our signup growth continues.

…we kicked off a TACFI with the USAF. 🚀 ****A small team of us met up with leaders in San Antonio to kick off the project. We have some great deliverables that we are excited about developing and shipping to our customers.

…we shipped a favorites feature for our Activity Library. ❤️ Now users can mark which activities are their favorite to run with their teams and projects. They can easily access these from the top navigation.

…hold a strategy planning session as an executive team. 🗺️ It’s time to refresh our strategy for T3. We’ll include input that we’re gathering from everyone at the company. The good news is we feel like we’re on the right track overall, but may need to make some small adjustments to our course.