Go to bottom

How should entries be ranked?

category: parties [glöplog]
One thing I didn't see yet:

Give people points to hand out over a compo (e.g. 2 stars per entry in compo -> you can give max N stars to a single entry to avoid excessive self voting).

This way maybe there'll be less noise, as people are forced to choose instead of just give 5 stars to their most liked entries and 0 to the others?
Yup, me too. With a 5-star UI, I think most people have something like -2 / -1 / 0 / +1 / +2 in mind
(ie "if I don't vote, it's neutral ; if I vote, I skew the mark on one side or another)

Indeed. This seems like a good system to me. I think it's just natural that if you're a lazy voter you only vote for your friends entries and for stuff that really stands out for you.

The only downside I see is if you stick to the 5 star system star #3 becomes obsolete this way. An even number of stars would probably "feel" better for those who take their time to vote for all entries, because then every vote has a true impact.
added on the 2019-08-23 09:09:46 by spiderj spiderj
Another idea:
At most parties the releases are presented in an order decided by the orgas (from worst to best). What if visitors would only say if the current entry was better than the previous or not (something like selection sort or bubble sort)?
added on the 2019-08-23 09:23:11 by pohar pohar
I'm perfectly fine with the current method.
added on the 2019-08-23 10:15:58 by teo teo
I would prefer avg or that sum based with neutral. Though i'm not weeping if current mostly used one stays and nothing changes :)
added on the 2019-08-23 10:30:25 by Serpent Serpent
Well, thank you KeyJ for confirming my fears that Spiral by Resistance should have beaten me and Gasman.. :P

It's really good to see the changes on the actual data, and seeing it for a party with smaller attendances would be great and help this discussion. If I get a vote on how we all vote in future, I'd choose the +2..-2 idea at the moment, but with spiderj's modification to an even split away from no vote so that you can have slightly more resolution for how you feel about the entry. Call it +2.5..-2.5 ?

Overall we are always going to be relying on the social side to get people to vote, and that means those who are good at the social side, and can get lots of people they know to vote in general, will do better. I don't have a problem with this, in order for it too happen you need to have been contributing to the scene for a while and bringing more people to parties. I want more demos and more people at parties, so I'm happy for them to get a small advantage.
What teo said.

Not a fan of the neutral for "not voted" - every click is an investment of time, if I don't like something, I don't want to invest any time to vote for it. The current system in partymeister would probably be a little more transparent if there was a 0 to 5 score instead of "no stars" to "5 stars". But at the end of the day, that's pretty much the same.

I think, most issues we see with demoscene votes are mainly caused by not enough visitors voting at all or bias introduced by human error, not by the used voting system.

And while it's rather clear that the evoke is biased, the result is actually what the people voted. Of the people who used live voting, probably roughly 20 didn't place their last vote. That's bad IMHO.
added on the 2019-08-23 10:54:47 by las las
KeyJ: Can you please add an option for giving unvoted entries 2 stars (or, equivalently, shift the range to -1..3)? This would come pretty close to the old Breakpoint voting system that (IIRC) had a range from -1 (SUCKS) to 5 (RULES).
added on the 2019-08-23 11:00:13 by Kabuto Kabuto
I like that we're getting into the discussion of voting - maybe we will have a better + easier to use system in the end.
If we stay with the current system I like the idea of trying out to take the average, but I see the point that this probably could only work on bigger parties and has other flaws.

Looking at the discussion I think we can safely summarize that we have the following goals for a voting system:

  • Easy to sign-up
  • Easy to use at demoparty, especially during live-voting
  • Doesn't punish non-voted prods (newcomers, unknown groups, compofillers)
  • Minimises possibilities to harm/skew results through "Fake-Voting"

Proposed System:

  • Possibility to Sign-up via Scene-ID to make it as easy as possible
  • Voting Scheme : +1; 0; -1 (similar to pouets ruletz, ok, sucks) (This has the following live-voting advantage: It doesn't matter what comes next, either you like a prod or not, or you don't care / it's ok. You don't have to save up your "5 stars" for a demo which is yet to come, or to adjust your votes constantly)
  • The default vote is 0
  • All Prods start with no of registered users = Points -> Everybody starts at the same base -> possibility to go down to 0 or to go up to 200%
  • Super easy live-voting interface (see mock below)
  • (Optional) Being able to add a comment at the party, ability to allow it to be shown during prize giving or to be transferred automatically to pouet/demozoo/both (Opt-in!)
  • (Optional) On-Party comments will be marked on Pouet as being such
  • (Optional) All votes will made public (anonymiyed) to enable statistics and for transparency, eg. to make patterns visible like 20 people upvoted only one prod and downvoted all others

BB Image
added on the 2019-08-23 11:01:38 by v3nom v3nom
v3nom: Giving negative points is really a terrible idea.
added on the 2019-08-23 11:11:29 by hfr hfr
For the sake of curiosity I'd like to see the outcome of concordet-style ranking as well (a > b if both have been voted for and a has more stars than b).

It could have similar issues as averaging for entries with few votes but it would also reduce the impact of people giving their own entry 5 stars and everyone else's 1 star or no vote at all.
added on the 2019-08-23 11:13:29 by Kabuto Kabuto
Oh, while looking at KeyJs "probably nicer" results, keep in mind that the experiment is a little flawed. It's a bit trying to compare apples and pears. Voters might vote totally differently depending on the voting system used. Throwing data gathered with one voting system through a filter to generate the data from another voting system - unfortunately - just does not work.

If we wanted to pick the best voting system based only on KeyJs experiment we would have to pretend that we changed nothing to the voting system and filter the data in post without telling anyone.
added on the 2019-08-23 11:20:25 by las las
hfr: fair point. It works the same with 0 +1 +2 though, with having +1 the default if you don't like negative votes.
added on the 2019-08-23 11:29:38 by v3nom v3nom
I prefer +0 as default. I don't want to invest any energy into voting something which I don't care enough about to vote (i.e.: your entry is bad enough that I probably don't want this anywhere near the winning entries and I really don't care about it).
added on the 2019-08-23 11:33:09 by las las
It would still be simple and not discriminating non-voted prods (eg because a screenshot is missing, or because nobody can remember the first demo of a 28-entry democompo.
added on the 2019-08-23 11:33:15 by v3nom v3nom
One more data point to consider: With Breakpoint's -1...5 voting system, there was probably only one entry that ever achieved a negative score, and that was because it somehow ended up in the wrong compo.
las: In my opinion the need to press one button to express your opinion is not too much to ask for when getting a fairer overall result.
added on the 2019-08-23 11:35:03 by v3nom v3nom
We had -1..5 at Solskogen for a while. Don't remember negative end results, plus there's something to be said about having an entry so bad that it gets negative results.
added on the 2019-08-23 12:00:36 by Gargaj Gargaj
v3nom: On a +/-1 voting scale you have no possibility to differ between two good entries.
added on the 2019-08-23 12:08:24 by hfr hfr
v3nom: I agree that it's not too much to ask, but the question is whether people (overall) are going to A) understand that this is what they're meant to do, and B) actually do it

it would be better imho to design the vote system around the assumption that people aren't going to press a button for every single entry, because that (in my experience) is the reality.
added on the 2019-08-23 12:08:48 by farfar farfar
hfr/farfar: sure, every system has it's pros and cons. For this proposal I had easy in use and 'actively selecting the worst option' != 'Not voting' in mind, with the added Charme of being pouet compatible.
I have to say I like a -1...5 or -1...3 with 0 being the default as well
added on the 2019-08-23 12:14:33 by v3nom v3nom
Thanks KeyJ for compiling the results!

Even if I agree with:

Voters might vote totally differently depending on the voting system used. Throwing data gathered with one voting system through a filter to generate the data from another voting system - unfortunately - just does not work.

And that's also apply to the current system: are you sure that all voters understand how the current voting system works?
Especially new-comers? People who are used to non-partymeisters demoparty? Drunk people?
With this incertitude, you can't say the results reflected what the visitors wanted, including the care / didn't care factor.
(To have a better idea on how much people understand the current system, we should make... a poll :p)

I like v3nom UI & the -1..3 range proposal (equivalent to current system with 2/5 for non-voted prod).
It's more fair to non-voted prods.
And it's better cognitively (sucks / neutral / why not / good / great) while still giving a fair expressiveness. Especially in a live-voting context.

(The more I think about it, the more I believe there is a bad mix between the current partymeister system & the livevoting. Two important biases in livevoting have been shown in this discussion: the voting-window bias, and the "I give 4 stars (and not 5) to a great prod shown early"-bias.).
added on the 2019-08-23 12:55:09 by wullon wullon
Voters might vote totally differently depending on the voting system used.

Absolutely. That's why I noted that the -2..2 results are just a simulation based on the 1..5-based input data, neglecting the psychological effects of the different scale.

However, regarding average-based ranking with the current 1..5 voting scheme, I think the results would be more or less the same, regardless of whether people know what happens with uncast votes or not. Those who think that an uncast vote is a neutral vote will be represented better, and those wo want to downvote entries with the least possible score can continue to do so; it's just 1 instead of 0 points then.

In general, I'm surprised that so few people see the appeal of a truly neutral vote option. If I missed an entry because I came too late to the compo, had to pee, had a phone call, had a too interesting conversation, or worked on my own prod, I don't want to assign any vote to that entry. I certainly don't want to downvote it, but I don't necessarily want to give it a seemingly neutral "was OK, nothing special" default vote either, because I don't fucking know how good or bad it was. I might have just missed "the product 2" -- or another boring Färjanmaker compofiller.

Sure, you may say now that I can review all the entries later on -- but seriously, who does that? It involves downloading all the entries (except those which screenshots in the graphics compos), relistening to music compos with headphones in a loud hall, maintaining a zoo of emulators for all possible and impossible platforms to run entries on ... that's a lot of work I can't imagine many people doing just because they want their votes to be as fair as possible.
added on the 2019-08-23 13:35:40 by KeyJ KeyJ
FWIW, there are preferential voting systems where you can rank candidates equally; you don't have to distinguish between entries you don't care about. So if you have a compo with 20 entries, you can rank the (say) five on top that you care about, and then put all the others on split 6th (or split 6th–20th, depending on how you look at it). Or possibly with that troll entry you really hate at the 20th, alone :-P
added on the 2019-08-23 14:37:23 by Sesse Sesse
With AVG scoring system, "Bold" would be the "best" demo on pouet (highest upvotes on all perfect 1.00 avg entries), disregarding all the nonvoters, and of course only if potential trolls resist to give it a pig right now ;)

And finally i know why i don't win compos more often ... the voting system! i knew it all along ^^
added on the 2019-08-23 14:48:11 by HellMood HellMood


Go to top