System-wide Performance Degradation

• 02:54 pm PST (UTC-8) December 11th, 2024

  At approximately 2:30 PM PST, we’ve been seeing stable recovery and functionality should be restored. We will continue to monitor the system to ensure stability.

  08:14 am PST (UTC-8) December 11th, 2024

  While removing Chartable prefixes ahead of the service shut down on 12/12, we experienced a high volume of traffic from Spotify due to their using the Episode URL rather than the GUID to identify episodes, which is not compliant with the IAB’s Podcast Measurement Guidelines. We have incrementally scaled out our systems throughout the past week in order to handle the anticipated spikes in traffic. On 12/11 at approximately 4 am PT, our load balancer started running into out-of-memory issues, which impacted our transcoder system. This has affected our ability to serve new ad decisions and new episode releases. The ART19 incident response team is actively working to increase our infrastructure capacity and bring stability to the system. We will provide a detailed post-mortem of customer impact once we are back to normal operations.