Spread security incident

TL;DR

Spread recently experienced a minor security incident related to increased email login attempts. However, no user data was compromised during the incident.

Introduction

In this blog post, I’ll discuss a recent security incident at Spread and the steps taken to address it. I’ll provide an overview of our authentication approach, explain the incident details, outline the actions I took, and discuss the future directions.

Authentication Approach

When designing Spread, I adopted a passwordless approach for user authorization and authentication. Spread doesn’t store any credentials or passwords. Instead, users can choose to authenticate using OAuth 2.0 with Google or any other email. In the latter case, Spread sends an email containing a “magic link” with a short-lived token. Upon clicking the link, the token is validated, and a longer-lived JWT access token is issued, signed with Spread’s RSA key.

Incident Details

On April 26th, I received an alert from Sendgrid, our email service provider, notifying us that our monthly quotas were almost exceeded. Given our focus on cost-effectiveness1, I had chosen a very modest plan that had been sufficient until now. My concern was that a potential leak of our API key could result in an unauthorized usage, affecting our reputation as a trusted sender.

Steps taken

To investigate the situation, I reviewed the logs and noticed a significant increase in email login attempts. While this was not ideal, it indicated an attempt to overload the endpoint responsible for sending login emails, rather than credentials leak or a sophisticated attack.

To mitigate the issue, I took the following steps:

  • Revoked the Sendgrid API token to prevent further unauthorized usage.
  • Temporarily disabled the “magic link” login/sign-on functionality until the API token was rotated.
  • Developed a solution to handle similar attacks by implementing an IP-based “jailer” decorator that limits the number of login attempts within a specific timeframe.
  • Collaborated with Juan to improve the decorator implementation, resulting in a more efficient and testable state machine-based approach.
  • The solution can be found in the following code.

1: sadly the academia is notoriously underfunded.

Written on April 26, 2023