Crowdsourced reputation/voting

About

Crowdsourced reputation is a possible solution to the problem of solves::online voter fraud. There are a number of different ways to do it, but the key features are:

• Every voter may indicate whether any other voter is known to them personally.
• Voters who do not have sufficient recognition by other voters are (essentially) prevented from voting.

There's a simple way to get each user's score, and a simple exploit for that method. There are a number of more complicated ways that are less vulnerable to that exploit.

Context

This discussion makes some assumptions about the context in which voting is taking place:

• The voting software will be just one of a particular set of tools designed to work together.
• The various tools, the web site they run on, and the group of users at that site, constitute a social node.
• The key software for running a social node is designed to make it easy for disgruntled users to fly off and open another branch.
  • software is free/open-source
  • sufficient user data is exposed to at least invite all users to a new node
• By the time these processes have gained enough usershare to have any significant influence, there will be many different social nodes in use (comparable to the number of wikis currently in existence).

Methods

Simple Average

In this method, each user's recognition score is the average of all other users' ratings of that user.

Math, for those who speak it:

Ex = (SUM(Ax:y) for [all y != x]) / COUNT(y)-1

Where: Ex = overall recognition score for user x Ax:y = user y's (recognition) rating of user x

The obvious problem is that a whole bunch of bots joining the network and giving each other positive recognition will overwhelm the influence of the existing legitimate users. If we allow negative recognition as well, they could even de-legitimize those legitimate users.

Weighted Average

This is where it gets tricky. Instead of taking each user's rating of another user (A_x:y) as being of equal value, we use that user's rating as a value-multiplier. The ratings of users with low recognition will be counted less than users with higher recognition.

Again, some math for those who like that sort of thing:

Assuming ratings are from 0 to +1: Ex = (SUM(Ax:y x Ey) for [all y != x]) / SUM(Ey)

(I have a strong urge to allow negative ratings, so as to allow users to flag known bots and sockpuppets, but at the moment I have no idea how we would interpret those numbers without accidentally giving actual bots and sockpuppets the ability to "vote backwards". Maybe "bot" and "sockpuppet" should be separate ratings.)

As with any kind of feedback network, the question now is how to handle the recursion. Each answer depends on the answers calculated the previous round. With enough iterations, the values will settle down -- but how many iterations do we need to do each time someone changes their rating of someone else in order to get a stable answer? There are a lot of different ways to do this. We can start by doing a large number of iterations when the network is small, because they each will not take much time, but eventually we will probably have to have the recalculation going continuously.

This method handles the "outvoted by bots" scenario in the Simple Average method with the assumption that the network will first be populated by "good" users, and that new users start with zero ratings and (therefore) no ability to influence anyone else's score. Real users won't recognize the bots, and won't give them any ratings. Problem solved -- as long as the bots don't succeed in convincing a legit user that they are real -- which may be a problem.

In order to minimize the temptation to "be nice" and legitimize users you don't actually know, one solution may be some kind of penalty to one's own reputation if a user you have legitimized is later discovered to be a ringer. This is inelegant, however.

Another solution might be to penalize users whose ratings (of other users) "buck the system" by legitimizing a user others have explicitly rated low. This suggests the question of under what circumstances one user should "downrate" another, rather than not rating them at all (equivalent to saying "I don't know this person, but I have no reason to think they're fake").

Local Centralization

A simpler solution that addresses all of the above issues is simply to have many smaller social nodes, with each node having a supreme dictator-sysop deciding which users are real and which are not, deploying whatever assistance or tools s/he feels are helpful in making that determination. Any user would be free to find a node whose userbase includes people known to them, who would then grant recognition on that basis.

The obvious exploit here is that a single user could gain recognition at multiple social nodes and thereby have more than their "fair" share of influence, especially if they are a gregarious person known to many people with different interests. While this would be a problem if we want to keep strictly to a rule of "one person, one voteshare", it seems to me that the extra effort involved in getting to know more people and spending more time discussing things means that each extra voteshare of influence would be paid for in terms of participation-time, which is something we want to encourage. This is something to watch, but I do not yet see any mechanisms by which it could actually cause a problem.

This page is a draft; editing is still in progress. Information may not be accurate or current. If you have comments to add, please contact Woozle.