Weighted Votes

EyeWire’s goal is to map neurons and understand how they form information processing circuits through synaptic networks. We achieve this together by mapping millions of individual cubes containing tiny volumes of neurons. Multiple people map each one. We compare submitted results after reaping to generate a consensus reconstruction. This correct final result allows us to calculate each player’s accuracy, as you see in your profile. Submitting a cube is equivalent to voting. In EyeWire, all votes are not equal.

Your vote’s weight is based on your accuracy on previous cubes.

If a player’s accuracy is above 70% in the Mystery Cells or above 80% in starbursts, then when she submits a cube, the volumes she adds have weight 1. If a player’s accuracy is below these thresholds, her weight is 0.1. We call this system “enfranchisement” because when you get good enough at EyeWire, you become enfranchised and earn a vote substantial enough to tip the majority.

If we ignore ability and let anyone who has passed the tutorial submit cubes equally, our cells would be growing all over the place and branches would constantly get stuck. That’s why we added enfranchisement.

Time machine

Back in the day, before the enfranchisement system, each cube was done by 11-18 people. Oh, the inefficiency. Top players often ran out of cubes. Cells took ages to grow. Brand new players could trailblaze.

EyeWire was unable to map starbursts because the average player could not find the narrow branch all the way across the cube. We coupled the starburst challenge with a weighted voting system, and thus enfranchisement and faster mapping was born!

Why wasn’t this advertised? Well, this new system was rolled out way before profiles, so players had no way to know their accuracy. We thought it would be frustrating to highlight something you couldn’t see or do anything about, so it became a backend implementation.

Rethinking accuracy

We’re all here to have fun and do science.  Hopefully you are having fun whether you are a super expert or just learning, but user weighting is all about doing more science and better!  The first step to improving weighting is to collect more data. Ignacio Tartavull, an awesome visiting developer from Argentina, built a system that continuously updates accuracy for every cube you submit.

This will allow us to study how our players are doing in different situations. Maybe players are more accurate in the morning or at night. Maybe some players’ accuracy decrease when playing too many cubes in a day.  We hope to use this information to build a better estimator of a player’s expertise at submission time which will lead us to produce more science from your time on EyeWire. Finally, with accuracy per validation in addition to weekly accuracy, we can give immediate feedback to players about their accuracy, which will be very helpful for new players.

We deployed a version of this accuracy system this week and as you saw, it didn’t work quite as planned. For now we are tracking real-time accuracy but have reverted back to the weekly promotion when accuracy is updated every Monday. We will continue to work on this and welcome your feedback – as Ignacio says, “this cool new data can be used to improve the future of EyeWire.”

Thanks for playing and as always, we’ll see you online at eyewire.org.

 

— final food for thought —

How many players should map each cube?

Should easy cubes be looked over the same number of times as a challenging bouton with a branch and several nubs?

If we could know the difficulty of a given cube and the expertise of a player mapping that cube, then we could assign players to cubes in an optimal manner to get a aggregated tracing with improved confidence.