The issue has been resolved and the services are back to normal. The team is working on deploying a permanent fix.
All impacted clients who need to reconfigure their origins (storages and servers) have been notified via email with the necessary steps. We are also working on restoring data states for the impacted clients.
UPDATE (0525 IST):
All the uploaded images and cached images of impacted clients have been restored
PostMortem of this incident will follow.
Posted Mar 27, 2018 - 04:23 IST
Update
We have restored services to all accounts except for accounts that have been registered or updated on or after 28 February 2018. We are manually trying to restore services to those accounts as well.
Posted Mar 27, 2018 - 02:29 IST
Monitoring
Services restored on all data centres for 95% clients (all clients registered on or before 27 Feb) and some clients post that.
Posted Mar 27, 2018 - 01:10 IST
Update
Services have been partially restored in all regions. We are continuing work to bring all the services back up for all clients.
Posted Mar 27, 2018 - 00:28 IST
Update
The issue is with one of our databases not running correctly resulting in downtime for all services across all regions.
Posted Mar 26, 2018 - 23:49 IST
Identified
The issue has been identified and a fix is being implemented. All images cached on CDN are being served with no error. This issue is affecting new transformations and miss image requests.