Office cat gone wild and tripped some wires, causing improper power down the server and database.

We're sorry for the inconvenience! 🙇

💡 Fun fact: we have worked for 26 minutes to resolve this server issue.

Example

We are unable to allocate VMs to run Databricks jobs. This will cause all analytics to be delayed. We have identified the root-cause to be a bug on Azure Billing side and have engaged with the Azure support team to resolve this as soon as possible.

4/13/2022 15:30hrs PT Still working with Azure on resolution. We are running a pared down data pipeline to unblock everyone, while in parallel we figure out a full solution.

20:00hrs PT The Azure issues have been resolved. Analytics are being processed and are catching back up. Incident impact was limited to processing delays; no data was lost.

4/14/2022 15:00hrs All services back to normal.