We’re Back to Normal

Dear EyeWirers,

I’m happy to report that our systems are back to normal. There have been a variety of problems, some structural and some bad luck, that have been making EyeWire very slow and occasionally crash over the past few weeks. Thankfully, that should all be over now. I want to specially thank nkem for spotting the key to solving the latest problem with our image data. You may notice some minor slowdown over the next day or so as our caches fill up.

Because this instability has caused so much stress for everyone in the EyeWire community, we are taking the following steps:

1) We are immediately putting a code and system reconfiguration freeze into effect. This means that no non-emergency changes will be put into production between 2014-03-01 and 2014-03-09 inclusive. Normal activities will resume on Monday, March 10, 2014. Feature development will continue in our test environment.

2) We have patched the bug that was inflating the size of our image data. Cubes are again 7-12 MB downloads instead of 40-70 MB.

3) We have changed the configuration of chat connections so that they are no longer being passed through our front-end cache. For unknown reasons, this may have been causing the accelerator to crash from time to time. This change constitutes a reversion to a last known stable configuration.

4) We have conducted a post-mortem and are implementing a system robustness action plan that accounts for technological and human factors. 

Thanks for bearing with us during these slow downs and outages. I know that some of them came at unfortunate times and caused you frustration. EyeWire launched a bit over a year ago as an experiment to see if we could democratize the quest to understand the brain – something that had never been done before. It’s been successful and become dear to you in ways we never imagined. EyeWire is still an experiment, but it’s not a toy anymore.

We’re back now, and as always, there’s science to do.

Sincerely,
Will Silversmith
EyeWire Developer