Difference between revisions of "Crowdsourced reputation/voting"
(→Local Centralization: modified in light of "context" section) |
(→Weighted Average: xref to chain of trust) |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 7: | Line 7: | ||
There's a simple way to get each user's score, and a simple exploit for that method. There are a number of more complicated ways that are less vulnerable to that exploit. | There's a simple way to get each user's score, and a simple exploit for that method. There are a number of more complicated ways that are less vulnerable to that exploit. | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
==Methods== | ==Methods== | ||
===Simple Average=== | ===Simple Average=== | ||
| Line 34: | Line 26: | ||
This is where it gets tricky. Instead of taking each user's rating of another user (A<sub>x:y</sub>) as being of equal value, we use ''that'' user's rating as a value-multiplier. The ratings of users with low recognition will be counted less than users with higher recognition. | This is where it gets tricky. Instead of taking each user's rating of another user (A<sub>x:y</sub>) as being of equal value, we use ''that'' user's rating as a value-multiplier. The ratings of users with low recognition will be counted less than users with higher recognition. | ||
| − | Again, some math for those who like that sort of thing: | + | Again, some math for those who like that sort of thing (where ratings range from zero to some positive number): |
| − | |||
| − | |||
{| align=center | {| align=center | ||
|- | |- | ||
| − | | | + | | E<sub>x</sub> = (SUM(A<sub>x:y</sub> * E<sub>y</sub>) for [all y != x]) / SUM(E<sub>y</sub>) |
| − | |||
|} | |} | ||
| Line 52: | Line 41: | ||
Another solution might be to penalize users whose ratings (of other users) "buck the system" by legitimizing a user others have explicitly rated low. This suggests the question of under what circumstances one user should "downrate" another, rather than not rating them at all (equivalent to saying "I don't know this person, but I have no reason to think they're fake"). | Another solution might be to penalize users whose ratings (of other users) "buck the system" by legitimizing a user others have explicitly rated low. This suggests the question of under what circumstances one user should "downrate" another, rather than not rating them at all (equivalent to saying "I don't know this person, but I have no reason to think they're fake"). | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | Looked at one way, this method uses [[chains of trust]] to broaden its reach and provide a cross-check. | |
| − | |||
Latest revision as of 13:29, 5 June 2011
About
Crowdsourced reputation is a possible solution to the problem of solves::online voter fraud. There are a number of different ways to do it, but the key features are:
- Every voter may indicate whether any other voter is known to them personally.
- Voters who do not have sufficient recognition by other voters are (essentially) prevented from voting.
There's a simple way to get each user's score, and a simple exploit for that method. There are a number of more complicated ways that are less vulnerable to that exploit.
Methods
Simple Average
In this method, each user's recognition score is the average of all other users' ratings of that user.
Math, for those who speak it:
| Ex = (SUM(Ax:y) for [all y != x]) / COUNT(y)-1 |
Where:
|
The obvious problem is that a whole bunch of bots joining the network and giving each other positive recognition will overwhelm the influence of the existing legitimate users. If we allow negative recognition as well, they could even de-legitimize those legitimate users.
Weighted Average
This is where it gets tricky. Instead of taking each user's rating of another user (Ax:y) as being of equal value, we use that user's rating as a value-multiplier. The ratings of users with low recognition will be counted less than users with higher recognition.
Again, some math for those who like that sort of thing (where ratings range from zero to some positive number):
| Ex = (SUM(Ax:y * Ey) for [all y != x]) / SUM(Ey) |
(I have a strong urge to allow negative ratings, so as to allow users to flag known bots and sockpuppets, but at the moment I have no idea how we would interpret those numbers without accidentally giving actual bots and sockpuppets the ability to "vote backwards". Maybe "bot" and "sockpuppet" should be separate ratings.)
As with any kind of feedback network, the question now is how to handle the recursion. Each answer depends on the answers calculated the previous round. With enough iterations, the values will settle down -- but how many iterations do we need to do each time someone changes their rating of someone else in order to get a stable answer? There are a lot of different ways to do this. We can start by doing a large number of iterations when the network is small, because they each will not take much time, but eventually we will probably have to have the recalculation going continuously.
This method handles the "outvoted by bots" scenario in the Simple Average method with the assumption that the network will first be populated by "good" users, and that new users start with zero ratings and (therefore) no ability to influence anyone else's score. Real users won't recognize the bots, and won't give them any ratings. Problem solved -- as long as the bots don't succeed in convincing a legit user that they are real -- which may be a problem.
In order to minimize the temptation to "be nice" and legitimize users you don't actually know, one solution may be some kind of penalty to one's own reputation if a user you have legitimized is later discovered to be a ringer. This is inelegant, however.
Another solution might be to penalize users whose ratings (of other users) "buck the system" by legitimizing a user others have explicitly rated low. This suggests the question of under what circumstances one user should "downrate" another, rather than not rating them at all (equivalent to saying "I don't know this person, but I have no reason to think they're fake").
Looked at one way, this method uses chains of trust to broaden its reach and provide a cross-check.