Google outage: how Dotdigital mitigated the effects on customers

This time of year is a conflicting one. It’s my favorite season in the Northern hemisphere – pumpkin spice lattes, crisp air, maybe a little too much food, and all the fun of catching up with friends and family – even if it’s virtual. However, there’s also a lot of stress in our industry, with pressure to meet KPIs and revenue goals with targeted content that doesn’t overload recipients. There’s so much careful planning and building of strategies in the run-up, but it can so easily be challenged by a variable we have no control over. We have to be adaptable; senders have to be able to pivot, and platforms like Dotdigital have to rely on contingency plans and quick thinking.

Google outage and what it meant for our customers

We were all put to the test this week when Google experienced notable outages with widespread impact. As frustrating as this has been for ESPs and senders, I have no doubt the team at Google has felt similar pressures to what we feel during this busy time. My heart goes out to that team.

The outages caused senders to see false errors returned stating the recipients being sent to didn’t exist.

This posed a challenge: under normal conditions, all senders, including dotdigital, have a commitment to make sure that they don’t continue sending to recipients that don’t exist. Suppressing recipients that bounce for this reason is usually the right thing to do, to prevent sender reputation damage from repeatedly trying to mail non-existent email addresses. Seeing the spike in bounces and additions to suppression lists means that the platform was working correctly – it’s a sign that dotdigital is committed to making sure only recipients that want and expect to receive emails are being sent.

How did Dotdigital respond?

As soon as we realized that these were not genuine bounces and instead reflected a problem with Gmail, dotdigital sprang into action. The teams identified three key areas for action:

• Stopping processing of the incorrect bounces during the outage

We added a rule to our bounce processor that matched the false-positive error we were seeing from Gmail and ignored these bounces, instead of processing and suppressing the contacts. Once we’re confident that Gmail has fully resolved its issues, we can easily remove this rule and continue to process these bounces again.

• Reversing the suppression of recipients that actually do exist and want to receive emails

We ran a database update to unsuppress Gmail and Googlemail contacts who had been marked as suppressed in customer accounts during the time period of the outage.

• Restoring these contacts to our customers’ address books

If we had just left the contacts unsuppressed, our clients would have had to do all the hard work to get them back in the right address books – that’s the last thing they need to be doing so close to Christmas, plus we wanted to help more. We used a database backup from before the outage to identify which address books the contacts had previously been assigned to. We then used this data to restore the subscriptions, putting the right contacts back into the right address books for our customers.

How could we respond so quickly?

Mobilization was swift thanks to our proactive monitoring and close working relationships between multiple teams, each comprised of amazing people with decades of specialized experience and intimate knowledge of Dotdigital and email delivery.

We were able to work on all three key areas simultaneously. This meant that in less than 24 hours of being notified that the latest outage had been resolved, dotdigital was able to complete the unsuppression of affected recipients and restoration of the Gmail and Googlemail contacts into customers’ address books.

Here at Dotdigital, I am proud of the teams and their ability to roll with what was thrown at them at the moment. They quickly found a solution and hopefully, we’ve taken away some of the added stress that I’m sure our customers were feeling over this.

We know this time of year brings a certain amount of pressure along with the added fun, and we are here to help.

If you would like to see the platform in action, and learn more about how we were able to mitigate this issue so quickly, please request a demo here.