We experienced a Partial Outage affecting access to the Moderation console from 2:00AM to 11:47AM EST on March 2.
On-site Moderation remained functional
Problem identified and resolved by 11:47AM
All events replayed to Moderation Services, full restore by 9:32PM
Incident Root Cause
Issue identified with configuration in the Amazon Machine Image for the ElasticSearch nodes
This configuration led to the master nodes losing the ability to communicate which eventually caused the outage with Moderation Console
Incident Impact
Access to the Moderation Console was unavailable, On-site moderation was unaffected
Resolution Details
Configuration of the AMI has been corrected
All events were replayed to the moderation services to fully catch up and restore data from the outage
Incident Prevention Actions
We have improved our error handling to be more resilient in this situation
Measures have been put in place to detect the error and action immediately should it occur in the future
Audit of Emergency Escalation procedures to uncover potential improvements
Once again, we thank you for your continued partnership, and please reach out should you have any questions.
Posted Mar 04, 2022 - 16:19 EST
Resolved
Full comment import has now been completed with all systems fully operational. Thank you for your patience.
Posted Mar 03, 2022 - 23:25 EST
Monitoring
"Flagged" and "Awaited Moderation" comments between the hours of 2:00AM - 11:47AM EST are currently being imported. We anticipate that this will take several hours for completion.
Posted Mar 03, 2022 - 17:18 EST
Update
We have resolved the issue related to the partial outage affecting the Moderation Console, and full access has been restored including front and back end moderation. Please note: comments that were posted between 2:00AM - 11:47AM EST and put into "Awaiting Moderation" or "Flagged" status, can not be reviewed from the Moderation Console at this time. Our teams are actively working to restore this visibility to the Console.
Posted Mar 03, 2022 - 15:16 EST
Update
We have resolved the issue related to the partial outage affecting the Moderation Console, and full access has been restored including front and back end moderation. Please note: comments that were posted between 2:00AM - 11:47AM EST and put into "Awaiting Moderation" or "Flagged" status, can not be reviewed from the Moderation Console at this time. Our teams are actively working to restore this visibility to the Console.
Posted Mar 03, 2022 - 15:14 EST
Identified
We continue to experience the partial outage affecting the Viafoura Moderation Console. The issue has been identified and our team is actively resolving this matter. We will continue to release status updates to this page. On-site Moderation continues to operate as expected.
Posted Mar 03, 2022 - 12:26 EST
Investigating
We are currently experiencing a partial outage to the moderation console with intermittent trouble logging into the Moderation Console. On-site Moderation continues to operate as expected. We are actively investigating this matter and will continue to release status updates to this page.