ART19 CMS is not available

• 19:51 pm PDT (UTC-8) December 1st, 2023

  What happened?
  On 2023-12-01 around 14:00 PM PST, when rotating our TLS certificates, there was a missed API call to sync the production monorail backend with Fastly’s next gen WAF that we adopted a few months back.

  What was the impact?
  From 14:19 PM PST to 15:02 PM PST, A small number of customers couldn’t login ART19 CMS. Listening traffic was not impacted.

  How did we resolve the issue?
  As of 15:02 PM PST, the HAProxy rollback was performed and the ART19 CMS was back to normal operations.
  At approximately 18:00 PM PST, we did an API call to sync production monorail with Fastly’s next gen WAF; then around 19:00 PM PST, we re-rotated our TLS certificates.

  What steps are we taking to prevent future incidents?
  To prevent this issue from occurring again, we updated our standard operating procedure to include this additional API call when performing this type of system maintenance.

  15:02 pm PDT (UTC-8) December 1st, 2023

  At approximately 14:19 PM PST, login functionality was impaired.

  This was caused by annual system maintenance when rotating our TLS certificates supplied to Fastly.

  As of 15:02 PM PST, we were able to restore back to normal system functions.

  The impact of this incident was isolated to impaired login, and we are gathering further data across our systems to further validate impact.

  As soon as we know more we will send a follow up. We’re sincerely sorry for the inconvenience and appreciate your patience.

  If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.