Impact Statement on Last Night’s Emergency Downtime
Dear EyeWirers,
Last night, while setting up an upcoming competition, a malformed database query was run against our production database and corrupted nearly all of our reconstructions. Thankfully, we have daily backups. As soon as we understood what had happened, we took EyeWire down to refresh it to its previous save point at midnight from the previous day, Sept. 21, 2015, starting around midnight Sept. 22, 2015. This took EyeWire offline for three and a half hours while its database was unavailable.
The good news is EyeWire is back online, the bad news is that we lost a day of progress. It’s pretty lame, and on behalf of the EyeWire team I sincerely apologize. Your contributions to the reconstruction effort are indispensable and we care deeply about ensuring they are respected.
Before we initiated our save point refresh, I saved all of your account information such as new accounts, points, and awarded badges and reapplied it around 3AM after the restore operation completed. Your points and awards are safe. There were some inconsistencies in scythe complete points (a few players had small negative numbers for that day) and I invalidated them since there was no corresponding record in our reset log.
To reduce the risk of this occurring in the future, I’ll be examining the chain of events that lead to the errant query being run against our production database and giving training as necessary. Thanks for playing EyeWire. Sometimes there are setbacks on the road to discovery, but we will do what we can to prevent them.
Sincerely,
Will Silversmith
EyeWire Lead Developer
Important Takeaways
- EyeWire was offline for several hours after midnight.
- A day of reconstruction progress was reset.
- Your normal points and achievements are unaffected.
- Scythe Complete points for the day are set to 0 (but no one had positive points).
Correction: A previous version of this article erroneously used the date Sept. 23 as the day of the restore point when the correct date was Sept. 22.
Correction: Added information about scythe complete points.