#374 – Moving Parabol to a new infrastructure

In the past weeks, we’ve moved Parabol to a new infrastructure and created a new release process. This project has been in the works for quite a while. With moves like this, we need to prepare by packaging up our old “stuff”, throwing away what isn’t needed anymore, and bringing the rest to our new “home”.

Parabol has been around for awhile now, the previous infrastructure had existed since 2015 actually. While functional, it wasn’t the easiest to work with for not only our developers, but our operations team. Like all technology stacks, over time it became more difficult to release new features and at times felt clunky and slow. In other words, our infrastructure got kind of … old.

Being a release owner for the week was something our developers didn’t enjoy. The past months it was becoming harder and harder. Issues came up again and again. Releases got postponed from week to week to fix issues that were standing in our way. No one enjoyed that process.

Our technology sprawl was quite large. We were spread across multiple cloud and hosting providers – most of it living in Digital Ocean, some in Google Cloud Platform, and the rest in Amazon’s AWS. CI/CD leveraged CirceCI and container orchestration was done via a Dokku instance. Great tools, but how we had things configured didn’t perform up to our standards. In themselves, all great tools. But how we had it set up didn’t perform well. Add on top of that lots of backup stages and manual approvals and it made the whole release process a long and arduous process that sometimes dragged on for hours.

Introducing Rafael, who joined in July 2022 as a Senior DevOps Engineer. He ventured out on a mission. A mission to make this situation better for all of us.

Diving into a project like this requires an immense amount of preparation and understanding. Slowly making smaller pieces better. We started to create Helm charts to make general deployment easier, set up pipelines in Gitlab and introduced Terraform to manage all of it.

The goal was also to concentrate as much as possible in one hosting tool, so that managing it would become easier. We are still a small company, so allowing for an easy way to manage access. We chose the Google Cloud Platform (GCP) for it, and have been happy so far 😊

Next on the list was more and better monitoring, to understand what is currently happening on our servers, and what we have to look out for.

We were already using Datadog, but we took it to the next level, adding more metrics, more monitors and better visibility for our infrastructure.

While all of that was happening, we also hired Dale, a third DevOps engineer to focus mostly on our Public Sector work with Aaron, our first DevOps, but to also help out Rafa when needed. Soon we realized that a lot was going on, both in our Saas offering and public sector work. All DevOps engineers connected, and talked, but there was no real space to get aligned. Alignment in a team like that is crucial, as we all need to work from the same foundation that Parabol is built upon.

Therefore we created the DevOps circle. Helping everyone to understand what’s currently happening, giving space to ask for help and questions, and speeding up communication. Check-In’s and Retro’s were set up, and we continued on our venture to move to a new infrastructure.

Having a DevOps circle also enabled us to give some work that was need or already happening at home. Matt was fixing a lot of Developer Experience issues, fixing memory leaks, migrating some of our databases – so it was an easy thing to have Matt work in the DevOps circle and help out where needed.

This actually resulted in a huge speed bump and improvement for everyone. Rafa had someone to count on making changes in the application whenever needed. And with all the knowledge Matt has (as he has been around since the beginning) this become kind of our secret superpower in this whole process.

Here we are now. New infrastructure. New release process. And have things changed?

Oh yes, they did. Deployments now take less than 10 mins. Down from 2 hours and longer. What an achievement. Developers are happy to do releases now, which is the way it should be. We all want our work to be out there for our users.

Right now, our deployment process require us to merge a Github PR generated by Release Please. This will then generate a Docker image, create a Github Release and even prepare our Gitlab repository for the deployment, which just means clicking a button.

To quote Nick, one of our developers:

More work is of course happening, to automate all of it even further, so everyone who wants to release something doesn’t even have to leave Github.

But we know the future in a startup is always cloudy and chaotic, which is part of the fun! So there is always stuff to work on and make even better. There are lots of challenges for the DevOps circle and the company in general, but we now have a much better foundation to work from 🙌

A few things we are going to look into in the near future:

Trends in top-of-funnel continue to fluctuate, as we see the effects of the latest Google algorithm update. We’re also seeing meetings fluctuate with the start of the holiday season.

But: we’re delighted to see signups ticking up! We believe this is the result of concerted focus & effort over the past few months, and are excited to see how this develops as we head into the holiday season.

…we had a small virtual retreat. A day filled with workshops, an AMA and a fun virtual game.

…had a Design Share meeting. Our design team invited people from across various circles to share ideas, or early prototypes and get feedback.

…finally got approved for our GCal integration. The new Google Calendar integration is starting to roll out to some of our users 🙌

…have a company holiday on Thursday and Friday. The whole week will likely be a bit slower, as almost half the company is celebrating the US holiday of Thanksgiving 🦃.