How should entries be ranked?
category: parties [glöplog]
Quote:
Quote:-- if votekey not used -> implicitely contributes to compos with mode "reverse compo order"
wouldnt this basically be a "organizers vote" then? given that (at evoke) about 10% of the votekeys registered weren't used for any vote at all i'd think that would be a very strong bias towards whatever the compoteam thinks should be placed last. or am i misunderstanding you?
using all votekeys in the calculation, no matter if they were used or not, is the "encuragement to vote the fuck" part to get more people to actually register and vote.
of course that can be reduced to actually used votekeys by people registered, as it is today, the point is about enforced ranking of things.
and yes, the reversed compo order would in general give a predefined ranking.
you could call that biased to the orgas/compoteam, but honestly:
- i've rarely seen compos with more than 5 top entries (subjective opinion)
- chances are very high that you find all of these at the end of the compo (because orgas know a good from a "bad" demo)
all i am saying is that you use that as starting point for the ranking, so it actually reduces complexity in my book.
you are free to shuffle nothing, only the first entry, or the top 5, or all the ranking, thats up to you.
the bottom half of entries, lets be honest is prone to random noise and "dont care" factor anyway.
Quote:
FWIW, the party that sparked this whole conversation, Evoke, used to have top-3 voting as well, until the early 2010s (don't remember when exactly it was replaced with what we have now). Seems like you can't please everyone no matter which way you turn it. ;)
Haha, indeed :).
Do you know if this year it was clear that if you didn't vote, you gave a 0/5 star?
(Also, for those who didn't read the Evoke thread: this year there was also a bias because of the livevoting ; the livevoting stopped immediatly after the compo, which was bad for the last shown entries!)
Anyway, I find it very problematic that the "I don't care attitude" (=0/5) has more weight than "I care enough to give a shitty rating" (=1/5) in a sum ranking.
Not sure if having the 0/5 is consistent with a 5-star ranking.
If you rate only one prod in a given compo, it should be like you gave at least 1/5 to all others (and not 0/5).
If I read the partial raw data correctly, it may have pushed Eisenerz to the top 3 at least and reduced the livevoting issue bias (at least 147 voters in the compo - probably more -, only 117 voters for Eisenerz, so at least +30 points).
Quote:
Do you know if this year it was clear that if you didn't vote, you gave a 0/5 star?
I think to most people who don't give it an explicit thought it might not be clear whether not voting on an entry means 0 stars or 2.5 stars or whatever. I'm pretty sure there are both people who use 0 as a strategic "worse than 1 star" vote and people who are not aware of the consequences if they only vote on some entries.
It's definitely not something that is communicated in the PM voting interface.
Quote:
Quote:Quote:Quote:wysiwtf has an interesting point, showing the livevote results (and number of voters per visitors ratio) while the compo is happening (up to 5 mins after) might be nice to call people to go vote.
If you've ever had the misfortune to sit through the voting stage of a livecoding round, you might have a feeling why that idea is a disaster.
care to elaborate your point to something one can actually discuss?
Sigh. Ever wondered why most elections don't announce exit polls until full poll closure? Results influence results.
Yes in between results would def. influence the other voters: the question is if that is a bad thing (in this case, in democratic elections it most certainly is). I mean, if theres a result at the end of the day "everyone can life with" thats basically the best case scenario, isnt it? Its not like people or groups are voted for a 4 year period into some sort of parliament.
Comparing it to clap-voting as done in the shader showdowns is a little bit unfair since the repetition is mostly due to the voting-style and being hard to judge by the compo orga while you can count digital votes very accurately. But yes its an example of people changing minds/voring behaviour on the fly.
That doesnt mean this is a good method and there wont be other problems connected to it. People may for example be tempted to vote at the last possible moment if a prod they want high up is "on the edge", so other voters cant react to that anymore.
There are possibly other disadvantages as well but also positive things (transparency being one of them).
I think I once saw some really old The Party video-footage where a current-results slide was on the bigscreen so experiences with this have obviously been made (and since other parties didnt jump on the bandwagon they probably havent been too positive, either).
Quote:
I think to most people who don't give it an explicit thought it might not be clear whether not voting on an entry means 0 stars or 2.5 stars or whatever. I'm pretty sure there are both people who use 0 as a strategic "worse than 1 star" vote and people who are not aware of the consequences if they only vote on some entries.
It's definitely not something that is communicated in the PM voting interface.
OK thank you.
Actually it was not clear for me until this thread (but I haven't be to a demoparty recently).
Well, it's a bit problematic ^^.
FWIW I like wysiwtf idea of a "voting as an event" where the results are displayed live (this would probably increase voting-engagement a lot) & benJam's winner-take-all (would probably make the party winner more significant).
Maybe some parties will experiment them in the future!
To increase engagement but not spoil results, maybe display a live-updated graph of the amount of visitors who have voted?
Quote:
Comparing it to clap-voting as done in the shader showdowns is a little bit unfair since the repetition is mostly due to the voting-style and being hard to judge by the compo orga while you can count digital votes very accurately. But yes its an example of people changing minds/voring behaviour on the fly.
Quote:
voring behaviour
voring = a cool word for votewhoring ;)
(yes yes, its a typo alright)
(yes yes, its a typo alright)
connect the amount of beers you're allowed to order at the bar to the number of votes you've cast, problem solved!
That already seems to be the case.
Wow. That's a long thread already.
TL;DR: I think we want to avoid pretty any bias that can be avoided and I don't think the current partymeister system is bad at all.
While I feel honored by gophers suggestions, I've been surprised multiple times by what people like. For example, the fractal ride intro wasn't even in my top 5 and I honestly don't like it. There must have been a lot of fractal lovers at evoke.
The "ordering system" doesn't fix self voting at all. You just selfvote your entry as first and then you order from bad to good. This just doesn't fix anything and I sometimes just don't feel like voting for everything.
Additionally, while gophers idea might not be that bad, I'm not a fan of it. Sometimes the play order doesn't even reflect an subjective quality ordering (if you have 3 equally good entries, it's already bad enough that I have to put them in any order at all, sometimes I just go random). There's also a tendency to put newcomer entries somewhere in the middle of the compo. Sometimes there's a technical reason to put a certain entry at a certain play position (reboots?).
Often you pick an intro as "last of compo" of which you think it fits the party mood best, even if there's a slightly better intro on your top 3 list.
I'm aware that playing entries in any order already suggests a certain result and introduces an inevitable bias (let's assume here we all agree that playing entries in a completely random order is a bad idea.). I really really don't want this as a voting baseline and it would certainly put even more pressure on the people doing decision making.
These discussions on voting system are probably as old as voting systems.
Back to the evoke incident (I was the 4k/64k compo organizer and very surprised by the final result, I was not in any way responsible for the voting):
If we had noticed the issue right after the compo, which we didn't (I really don't want to know any results if it can be avoided), I think we would have encouraged people to vote and additionally asked them to check their votes.
Figuring out what exactly happened took some time after the party and at the end of the day we can't be sure how the results would have looked like if there hadn't been any bias due to the live voting incident or if the people cared more about their votes being counted. It's not like people didn't know that their vote wasn't counted or anything like that.
To the best of my knowledge all these live voting issues are handled a better in very recent versions of partymeister, it's pretty unlikely that we see exactly something like this again. Nevertheless: Take the time to vote. Maybe we should do a "you voted" == free beer/non alcoholic beverage voucher thing? IMHO we don't need a completely different voting system. We need to encourage people at the parties to vote and to care about their decisions.
TL;DR: I think we want to avoid pretty any bias that can be avoided and I don't think the current partymeister system is bad at all.
While I feel honored by gophers suggestions, I've been surprised multiple times by what people like. For example, the fractal ride intro wasn't even in my top 5 and I honestly don't like it. There must have been a lot of fractal lovers at evoke.
The "ordering system" doesn't fix self voting at all. You just selfvote your entry as first and then you order from bad to good. This just doesn't fix anything and I sometimes just don't feel like voting for everything.
Additionally, while gophers idea might not be that bad, I'm not a fan of it. Sometimes the play order doesn't even reflect an subjective quality ordering (if you have 3 equally good entries, it's already bad enough that I have to put them in any order at all, sometimes I just go random). There's also a tendency to put newcomer entries somewhere in the middle of the compo. Sometimes there's a technical reason to put a certain entry at a certain play position (reboots?).
Often you pick an intro as "last of compo" of which you think it fits the party mood best, even if there's a slightly better intro on your top 3 list.
I'm aware that playing entries in any order already suggests a certain result and introduces an inevitable bias (let's assume here we all agree that playing entries in a completely random order is a bad idea.). I really really don't want this as a voting baseline and it would certainly put even more pressure on the people doing decision making.
These discussions on voting system are probably as old as voting systems.
Back to the evoke incident (I was the 4k/64k compo organizer and very surprised by the final result, I was not in any way responsible for the voting):
If we had noticed the issue right after the compo, which we didn't (I really don't want to know any results if it can be avoided), I think we would have encouraged people to vote and additionally asked them to check their votes.
Figuring out what exactly happened took some time after the party and at the end of the day we can't be sure how the results would have looked like if there hadn't been any bias due to the live voting incident or if the people cared more about their votes being counted. It's not like people didn't know that their vote wasn't counted or anything like that.
To the best of my knowledge all these live voting issues are handled a better in very recent versions of partymeister, it's pretty unlikely that we see exactly something like this again. Nevertheless: Take the time to vote. Maybe we should do a "you voted" == free beer/non alcoholic beverage voucher thing? IMHO we don't need a completely different voting system. We need to encourage people at the parties to vote and to care about their decisions.
Quote:
Quote:Quote:Quote:wysiwtf has an interesting point, showing the livevote results (and number of voters per visitors ratio) while the compo is happening (up to 5 mins after) might be nice to call people to go vote.
If you've ever had the misfortune to sit through the voting stage of a livecoding round, you might have a feeling why that idea is a disaster.
care to elaborate your point to something one can actually discuss?
Sigh. Ever wondered why most elections don't announce exit polls until full poll closure? Results influence results.
you're talking about voting systems where you can only vote once, demoparties voting system you can still change your vote after the livevote poll closes.
Yes, being able to change your vote after seeing the results surely keeps you from being influenced, and especially keeps you from acting out on that influence! :)
Seriously, did you read what you wrote?
Seriously, did you read what you wrote?
"Oh we don't have enough votes yet, let's do something about it!"... I'm with Gargaj on that one.
I think the current system is just fine. If someone comes up with something seriously better, why not. But for the time being I'm not seeing that.
I think the current system is just fine. If someone comes up with something seriously better, why not. But for the time being I'm not seeing that.
gargaj: by that logic all polls should be banned. i agree a livepoll will influence the final result. i don't agree that that is necessarily a bad thing though. and i'm not advocating it to be public throughout the entire voting period, just used as a reminder after the compo that you should go vote.
i would still prefer to test out a system of proxy voting through other sceners though.
the voucher for free stuff also sounds like it might help more people to bother voting.
i would still prefer to test out a system of proxy voting through other sceners though.
the voucher for free stuff also sounds like it might help more people to bother voting.
Quote:
The "ordering system" doesn't fix self voting at all. You just selfvote your entry as first and then you order from bad to good.
Well, actually i think it would fix it.
If many people vote that way, inherently the "bad" entries will get ranked higher, potentially even surpassing the selfvoting intros, so essentially rendering the intended tactic void.
i believe this would enforce fair voting (at least in the long run), maybe after one or two "surprise" results based on that :D
To extend on gamifying voting:
another possible incentive(which seems to be rather unattractive among some):
ensuring that people will actually bring decent releases for you to enjoy in the future.
And while we're overthrowing the status quo, let me suggest three more options:
Please continue serious discussion below.
- glöps for votes
- voting compo(number of votes cast pP., live updated, constantly being displayed and ranked)
- chance to win free merch by luck of the draw(every vote turns the digital fortunewheel)
- reduced entrance fee(payback at the end of the party)
- a free drink every completely voted compo
another possible incentive(which seems to be rather unattractive among some):
ensuring that people will actually bring decent releases for you to enjoy in the future.
And while we're overthrowing the status quo, let me suggest three more options:
- totalitarian voting, I decide who wins
- illegal thumb-wrestling in the backyard
- we make an octopus decide
Please continue serious discussion below.
Quote:
If many people vote that way, inherently the "bad" entries will get ranked higher, potentially even surpassing the selfvoting intros, so essentially rendering the intended tactic void.
i believe this would enforce fair voting (at least in the long run), maybe after one or two "surprise" results based on that :D
I don't think your math is correct on that one.
Quote:
It's not like people didn't know that their vote wasn't counted or anything like that.
It depends on what you mean by "their vote wasn't counted".
If you (live-)vote on all entries, except the last one, you give a 0/5 rating to the last entry.
I bet most people don't realize that.
It guess it was never obvious on the regular UI.
But it's even less obvious in a live voting context, where you are focused on one prod, where you don't see the global list, where you have absolutely no incentive to "rate them all" (BTW I remember some scener mass-rating 1/5 to prods from groups/persons he didn't like ; he was actually giving them points haha).
This is a genuine concern imho, especially with the generalisation of live voting.
I don't mean you should change the system per se, but at least make some education about how the system works.
For example, on the UI:
displaying "+1", "+2", "+3", "+4", "+5" icons, instead of the stars.
Quote:
And while we're overthrowing the status quo, let me suggest three more options:
- totalitarian voting, I decide who wins
- illegal thumb-wrestling in the backyard
- we make an octopus decide
Please continue serious discussion below.
your first idea, we already had some editions of inércia where there was no public voting and i just decided special awards getting special prizes :) worked fine.
your second idea, we already had smash arm wrestle navis, no clue why that didn't get universaly adopted though :D
third idea is definitely worth considering ;) octopuses are reknown for their unbiased voting behavior!
I wonder if it would change something if all prods get 3/5 unless a user gives them a different rating (so if someone decides to only vote for 1 prod in a compo all other entries of the compo get 3/5 from that user).
Yup, me too. With a 5-star UI, I think most people have something like -2 / -1 / 0 / +1 / +2 in mind
(ie "if I don't vote, it's neutral ; if I vote, I skew the mark on one side or another)
(ie "if I don't vote, it's neutral ; if I vote, I skew the mark on one side or another)
You get n CDC's per compo and hand them out accordingly. No reason to rank the demos, just say "these n deserve my praise".
As promised, I computed some simulated party results with different ranking algorithms, but otherwise genuine "real-world" voting data.
Three algorithms have been tested:
Long story short, the results can be viewed here: https://keyj.emphy.de/partyvoting/
The left half is the result as it currently is, the right half shows the results of the alternate ranking algorithm. (You can switch algorithms with links at the top of the page.) The column in the center of the table indicates how many ranks the entry to the right would have gained or lost if the alternate ranking method had been used.
Executive summary: both of these simple changes to the existing voting procedure shake up the results quite a bit, with many entries gaining or losing one rank, and up to six ranks of maximum deviation compared to the current system. Average-based ranking has a larger overall effect than the -2..+2 method, but both tend to work in favor of the weaker entries in compos with less overall votes.
So far I only got data for Evoke 2019, but I'd really like to add more parties! If you're a party organizer and have access to any recent Partymeister database dump, please send it to me. (I only need the tables 'competitions', 'entries', 'votes', and perhaps 'accesskeys'.)
Three algorithms have been tested:
- Sum-based ranking, as currently implemented by Partymeister. Not voting means 0 out of 5 points. This is the baseline for comparison.
- Average-based ranking. Same voting procedure, but not voting doesn't influence the result, at the expense of being susceptible to "noise" if too few people vote for an entry.
- Sum-based ranking with a neutral option. Would be a slightly different UI, with -2/-1/0/+1/+2 points per entry, and 0 being the outcome for unvoted entries. Mathematically, this is equivalent (up to a constant offset that doesn't influence ranking) to the 1..5 system, except that 3 is the default instead of 0. Note that I just re-interpreted the 1..5 voting data as -2..+2 to simulate this; the psychological effects of having the -2..+2 range presented as such while voting may change the outcome in reality.
Long story short, the results can be viewed here: https://keyj.emphy.de/partyvoting/
The left half is the result as it currently is, the right half shows the results of the alternate ranking algorithm. (You can switch algorithms with links at the top of the page.) The column in the center of the table indicates how many ranks the entry to the right would have gained or lost if the alternate ranking method had been used.
Executive summary: both of these simple changes to the existing voting procedure shake up the results quite a bit, with many entries gaining or losing one rank, and up to six ranks of maximum deviation compared to the current system. Average-based ranking has a larger overall effect than the -2..+2 method, but both tend to work in favor of the weaker entries in compos with less overall votes.
So far I only got data for Evoke 2019, but I'd really like to add more parties! If you're a party organizer and have access to any recent Partymeister database dump, please send it to me. (I only need the tables 'competitions', 'entries', 'votes', and perhaps 'accesskeys'.)
I vaguely remember parties having the following policy: if you don't vote for some entries within a compo, your votes within that compo are not being counted.
IIRC at the same parties, if orgas spotted someone voting 1,1,1,1,5,1,1,1,1, they also discarded those votes which I liked, but I know it's rather hard to apply it to bigger events and probably even harder to implement the automatic check (not to mention conversations on where to draw the line: "so 1,1,1,1,1,5,1,1,1 is not ok but what about 1,2,1,2,5,1,2,1? is it ok? yes? no? why yes? why no?")
Anyway, as previously mentioned, it's only treating symptoms, not the cause (the low participation rate). Having said that, I really like how civilised this discussion has been so far.
IIRC at the same parties, if orgas spotted someone voting 1,1,1,1,5,1,1,1,1, they also discarded those votes which I liked, but I know it's rather hard to apply it to bigger events and probably even harder to implement the automatic check (not to mention conversations on where to draw the line: "so 1,1,1,1,1,5,1,1,1 is not ok but what about 1,2,1,2,5,1,2,1? is it ok? yes? no? why yes? why no?")
Anyway, as previously mentioned, it's only treating symptoms, not the cause (the low participation rate). Having said that, I really like how civilised this discussion has been so far.
