In one of the previous posts, Matthias pointed out that we need to focus more on getting people to vote on open moderations. I’d like to brainstorm a little on this…
I have plans for a more elaborate editor rating system that would replace our black and white automoderator system. This new system will create many levels of privileges and automatically adjust an editor’s rating based on their editing/voting history. As an editor moves up in the rating, fewer of their edits will be put to a vote or perhaps require fewer votes to pass. Many of the details of this new system have not been worked out.
However, until we get to this system which ought to reduce the overall number of edits required to be voted on, we should somehow attempt to encourage people to participate in the peer review voting process. So far we know:
- Voting i-frame: A “vote on this edit” i-frame shown at the top of each artist/album page didn’t increase the voting much but encouraged people to just start clicking the no button causing lots of pain for many editors. Voting no become too easy. Once we turned this off by default and made it optional the problem of random no voting went away.
- Forcing people to vote: Forcing people to vote never works. People will vote anything just to get past the obstacle. This is not a viable solution.
- Showing top voters: The top voters list encouraged more people to vote, but that wasn’t enough. This shows that giving people credit for their work encourages others to participate more.
We’ve probably learned more, but these are the big lessons I remember. Please post a comment if you remember something I’ve forgotten.
The only thing that has worked to a degree was to put active voter’s names in lights and give credit where credit is due. For quite some time I’ve thought about using an eBay like system to put more informative icons before users name to indicate the user’s status. eBay uses stars and other icons to show the status of a user to give people immediate feedback about the trustworthiness of the seller.
What if we create a new icon for users that graphically shows the editor status and the voting status of the user?
Editor status:
- Newbie registered less than two weeks ago
- Less than 100 edits
- Less than 500 edits
- Less than 1000 edits
- More than 1000 edits
Voting status:
- 0 votes cast
- less than 100 votes cast
- less than 500 votes cast
- less than 1000 votes cast
- less than 5000 votes cast
- more than 5000 votes cast
These numbers/breakdowns are just randomly made up to illustrate the point — I’d like to have the community suggest actual values and graphical representations of these values if we proceed with this plan. What do you think of this approach?
Finally, I’d like to play devils advocate: What if our magic number of ~6000 open edits is our steady state? What if none of our actions ever change the number of open edits?
Technorati Tags: collaboration, communities, musicbrainz
MetaBrainz Blog 
It would be nice if we had an easier way to know what languages are ‘mastered’ by the moderators/voters/users.
I tried once to vote all the open edits (lots of abstain), and then managed to pass a month voting all edits, but it was too much work. An extensive search is needed to actually vote (yes, no).
I think most of this ~6k edits share this property: they are not absolutely, 100%, correct, and voters don’t know what is better: to vote down an edit because of a little mistake, or to allow a not correct info to be applied. So, most voters abstain, hoping that other user decides. (In this scenario I think it is a good idea to show the number of the votes when browsing.)
for4saken
> I think it is a good idea to show the number of the votes when browsing.)
exactly, that was my idea too when i changed this (eg show votes even if i haven’t voted yet). but the problems that come with this are obvious: if there is one yes vote by a voter who hasn’t checked the edit first (or a no vote) and people blindly follow the lead, its a case of potential injustice. But i am still quite convinced, that this would reduce the number of edits which linger in the system for one week + the additional 7 days grace period. If it looks good, i’m usually voting yes on edits which can be reversed again (not on merges and removes, though). This is another area which can be reviewed, to model these “dangerousness” levels into a system which passes edits a bit faster.
i like the editor rating ideas, i have said so before. I also think that these icons should be added… First steps have been done to introduce the icons, although not in a funcational way yet, but only visual. It will be difficult to concinve the users who have turned off the icons to enable them again, or to just turn all the icons on by default (since they carry an important information after this is implemented), but its worth it.
About the ~6000 open edits number: I think it stems also from a healthy growing process in terms of active users. There are always new users registering on MB, and only a few are made auto editors. Therefore the number of edits will be reduced if the system is adjusted in the area which decides which edits have to be kept open like they were, and which ones dont
keschte
This issue has had my thinking for quite some time. I believe that it’s not about quantity, so much as quality. Someone could have 1000 edits, but 100 voted down, so the icon system purely based on how many edits they’ve done isn’t as effective as it could be. And such a simple system could bring with it other problems, such as blind voting, as keschte pointed out.
This is just my opinion, but firstly there should be a threshold system, where they must be a member for a certain amount of time and/or reached a certain amount of edits (eg. 2 weeks and/or 50 edits). Once they have reached that small step, then some sort of trust system kicks into place where, they have to stay under a particular percentage of voted down edits, and as they increase there editing/voting they are granted more privileges. However, this also has it’s flaws. For example if someone puts out multiple edits and they all get voted down by accident or because of another unfair reason, then mathematically it would take the editor a long time to regain that percentage, and would ultimately deter them from using/contributing to MusicBrainz.
Although, there are so many factors involved. Like one person may have more edits than another user, but does that mean that they are a better user? or more valuable to MusicBrainz? That question is a little harder to answer.
With the correct planning and input from other users, I believe this can work very well and setup a fairly solid system for current and future users.
I think that one of the main resaons of people not voting is because you have to go and sit for it and make time to spend some time on voting. I go through the daily edits of artist I’m subscribed to and that’s for some reason less of a ‘make time to vote’ approach. This is probably because I get confronted by it by means of an e-mail which makes it more like a ‘quick inbetween action’ approach. At least for me it makes it more justifiable to do it.
From this point of view I would suggest that we put a bit more focus on getting people to subscribe to artists. This will also ensure a bit more that people actually vote for stuff they at least know a bit about.
This could be achieved by displaying a notification telling users that there aren’t any subscribers for a specific artist yet. This can be done at the moment someone is just browsing an artist or only when someone makes an edit to an artist or a release of a specific artist.
The notification can also be displayed until a certain amount, lets say 10(?), of subscribed people has been reached.
To get more subscribers we can also display a notification about similar artist not having any subscribers yet when someone subscribes to a specific artist.
A second, or additional, solution could be a set of plugins or extensions for different programs. E.g. a Firefox extension which displays random edits at the top or bottom of the browser window. People can ignore them when they’re not up for it at a specific time or have a look every once and a while to place a vote. A different colour scheme could be used for the type of edits to indicate its importance or to indicate if it’s an edit of their interest. At this stage the interest levels can only be determined by being subscribed to the artist or not and similar artists to which they are subscribed already, but I can imagine that more categories can be created in the future (genre, etc.).
I like the idea of icons next to a persons name, even if it’s just pointless eyecandy they’re like medals and people will want to get as many as they can. Some online gaming sties revolve around this idea. So does the Military come to think of it.
I’m also like Prodoc in my voting, I’ll do my subscribed artists every day, other votes not so much. Encouraging people to subscribe to artists will get people with alot of knowledge of a certain artist putting their time into a place it’s best used.
Perhaps we can merge both ideas so that after they make x number of votes to an artist the system suggests they subscribe to them.
Incentivate people to subscribe more artists it’s a good idea.
As a first (and dramatic) step, I would suggest to display a pop-up in picard when somebody look up for an artist that doesn’t have a minimum number of subscribers (the min could be also an ‘optimum’ ratio between edits and subscribers), asking if he wants to subscribe to that artist.
Also, when using picard we could be alerted that the artist/album we are looking up has open edits, and ask us to vote. IMHO, this would help the people who only uses the tagger to realise that we need them to vote.
By the way, about picard, it really needs to alert when an album (or track in the album) has open edits (could be the same sistem of the site).
Historically I haven’t done much voting, I sort of feel there is a basic flaw in the system in that there are only 3 (or 4 if you consider not voting a choice), yes/no/abstain.
I submit that adding changing the voting options to something like the following might encourage voters and also reduce the open edit count:
– Yes, I have this release and/or have exhaustivly investigated this edit and agree with it as stands.
– Yes, I do not have this release, but it looks good and appears to conform to MM guidelines
– Abstain, I know nothing of this release and do not feel qualifed to vote on its validity
– No, I don’t have this release but more information/documentation is needed and/or there are some corrections that need to be done
– No, I have this release and this edit is completely wrong and/or is a duplicate of existing data (link to existing data)
Possibly some other vote options could be used as well.
Yes I understand that someone could already vote yes/no/abstain and include some of the comments above, but this of course leaves the black and white situation of strict yes/no votes.
I’m with Prodoc: I’m very much likely to vote on edits for my subscribed artists, but I rarely, if ever, go into the wider pool of open edits. It’s just all too much to consider at once.
Two thoughts:
1. A Slashdot-meta-moderation-like voting system, where all users who agree to do so get 10 random edits emailed to them once a day to consider. Break it up into small chunks, and it won’t seem so bad. I could and would generally make time for 10 edits a day. [And keep records: if people sign up and don’t actually follow through, pull them out after a while. No unnecessary CPU cycles!]
2. Working to encourage regular submitters into becoming subscribed editors. Maybe there’s a way to automate this: if I’ve entered five edits into the system for artist N, invite me to become a subscribed user for that artist.
I think the key thing behind MB is that it provides users with the opportunity to share their knowledge of the artists that they know well. If I’m entering edits in about an artist, it typically means I have some attachment to them. The quality of my voting is going to be far higher when I care about it; honestly, I don’t care too much about bad data being in the system for artists that I’ve never heard of, so I’m not going to spend a lot of time rooting out the bad data.
Does this make sense?
In addition to the mentioned subscription encouragements I think it would also be good to add a ‘Subscribe to this artist’ checkbox on the add artist pages. This makes it easier to subscribe to an artist right away when someone adds a new artist. Usually when I add a new artist, I’m too focussed on the relations I want to create with it. This makes me forget the subscribe to it since that’s a completely separate process at the moment.
Prodoc: There is a preference that will automatically subscribe you to any artist you create.
In General: I wonder if the sheer size of the list isn’t intimidating some people by itself. Opening that ‘vote on edits’ page and seeing it’s page 1 of 685, is a bit scary.
I think we could probably utilise approving of things that are ok too. I know I rarely approve an edit, but pretty often if I look up an edit that just passed, it passed with three yes votes, all by autoeditors. If the autoeditors are the ones doing most of the voting anyway (and it does appear that way), why not push through more of the easy edits and leave the voting and discussion for the trickier ones. Not, of course, approving anything even remotely contentious, but straightforward edits like adding wikipedia links, or amazon ASIN’s.
Or maybe something like automatically applying things that have three yes votes from autoeditors and zero no votes after 24 hours (to allow slowing down people rallying support for a contentious edit) instead of waiting for the whole week.
Encouraging subscription also fits into this. Not only are you confronted with a list in email, as someone mentioned, but when you go look at the list, it’s short and to the point, and when you get to the end of it, it’s done. A lot of people are happy to give up ten minutes for something that gives a feeling of accomplishment because it has a finite end.
Lauri: maybe you want to review the documentation on voting, edits with three yes votes and zero no votes are applied the next full hour. 😉
Adding the approve function didn’t have the effect we expected. In the beginning the edit queue went down a bit but then back to the average number. Maybe because approving still is some more effort than voting (in some cases it’s quite annoying that one can’t mass-approve).
What I really dread is the general growing acceptance of a system where people get more rights just by doing lots of edits. The community idea gets completely dropped; behaviour of a person, quality of their edits, trust of other community members, engagement and considered value for the community don’t seem to matter any more.
And if you drop community values, the community might fall apart…
Hey all, just want to restate some suggestions I thought were excellent and put out a few more which I think are also excellent.
1. Show how many artists editors are subscribed to – I think this will help advertise the subscription tool and encourage people to subscribe to more artists.
2. Show how many Yes/No votes an editor has received on their edits – Instead of a straight “+/- Edits Approved” score, why not count the number of Yes/No votes an editor’s edits have received? This would encourage good editing and referencing edits so that people could increase their “score”. Also, another idea is to use the “Thumbs Up/Thumbs Down” idea that http://www.torrentspy.com uses for its torrents. If you are browsing edits and someone provides a piece of evidence that convinces you to vote their way, click the “Thumbs Up” button for them to increase their score.
3. Suggest other artists to subscibe – This will help advertise the subscription tool and help people with lots of subscriptions find more artists they can edit.
4. Random edits emailed daily – I really like this idea since the “View open edits” page(s) is VERY overwhelming. Having the option to receive 10, 20, or 50 edits emailed to you allows it to be broken up into chunks and seems much more managable in spare time! 🙂
5. Show “You (are|are not) subscibed to this Artist” on artist pages – this will help us find out artists we edit frequently but aren’t subscribed to. It will also advertise the subscription tool.
Cheers,
-Aaron
I actually think that the system is fine the way it is (with many open edits), and that people should not be encouraged to vote on things they don’t know about and are not willing to research.
When I first got here I was very concerned about the quality of moderation I was getting. People who knew nothing about the artist or release would bother me with all sorts of irrelevant questions and demands. I found that if I wrote anything in the edit note it was guaranteed to attract dissenters (better to explain nothing!) Others I know have actually tried MusicBrainz and abandoned it for this reason. Now, I guess my handle is recognized (or overall habits have changed, and it is true also that my edits have changed) and so the random static has magically disappeared.
I think the quality of moderation is as problematic as the quality of edits. (More so, really, since an inaccurate edit can always be undone later, but once you piss someone off they are very likely to leave MB.) Rewarding people for quantity of votes seems much more dangerous than rewarding them for quantity of edits.
@bklynd: I agree that rewarding quantity of votes is a dangerous idea – we could easily get people spamming yes votes to up their “score”. But would you not agree that rewarding/”punishing” people based on the number of Yes/No votes their EDITS receive? I think this would be a very good way to judge who is making good edits.
-Aaron (cooperaa)
What I meant in my last comment was:
“But would you not agree that rewarding/”punishing” people based on the number of Yes/No votes their EDITS receive is a good idea?”
Heh. I’ve read the “how voting works” page a million times, apparently it doesn’t stick.
I’d also like to see something less mathematical and more community based, perhaps like Advogato’s trust metrics. Even if it only kicks in after a certain threshold of good edits (and good votes) was passed, to avoid the usual cliquish behaviour that evolves. Then a matrix so that you end up with “you have x% good edits, you have an average community trust rating of +2, these types of edits are now autoedits for you”.
The one simple feature that would get me to happily vote more would be to be able to get emailed Open Edits for /related/ artists to those I’ve subscribed to.
The functionality is already there when you’re browsing artists, but the email system will only email you those you’re subscribed to. I’m VERY likely to vote on related artists as I know the genre and am interested in the artist broadly speaking.
I don’t frequently broach the open pool as I know nothing about foreign language, classical or the many other subgenres of music that have a good following on here. Wading through abstentions on those votes is prohibitively expensive on my time.
What about adding a tick box to preferences with “Email me open edits for artists related to my subscriptions” and implementing the ability to include related artists in what is emailed?
I back the motives on increasing subscription and also I really like the idea of having 10/20/n open edits emailed to you each day. This doesn’t make it seem like a chore that we have no chance of tackling.
When I’ve tried going through the edits it’s been very hard, and just when something looks ok, there’s 600odd more pages and I just don’t feel I’m getting anywhere.
Here’s my take on how it could work. Maybe if the system first gave you 10 open edits. You are able to vote on 8 of these, but unable to on 2 and leave comments. The next day, you receive 12 edits – 10 new and 2 from yesterday. This gives you chance to check any messages, but also carry on. Months pass and no one has replied to your messages so you choose “Remove this edit from my edit digest” and you will never be notified again because you don’t feel you can add a useful vote.
RSS? Or it would increase the server load.
I hate when I receive an email with my subscriptions and the edits that I had the knowledge to vote on were already accepted. 😉
RSS? Or it would increase the server load.
I hate when I receive an email with my subscriptions and the edits that I had the knowledge to vote on were already accepted. 😉
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq