Thoughts on anonymized compos
category: general [glöplog]
What about other voting methods, such as the Schulze method? The method itself requires voters to give an order of preference of entries, but that could be deduced from number of stars (and no mutual preference for entries on the same star level each).
While the method itself looks promising I don't know how well it would fit to demoparty votes - it's usually used for votes on persons. Someone with access to real-demoparty data should give it a try, there's sample code on that wikipedia page. But for sure it would resolve the "-1 for everyone but my friend" problem.
While the method itself looks promising I don't know how well it would fit to demoparty votes - it's usually used for votes on persons. Someone with access to real-demoparty data should give it a try, there's sample code on that wikipedia page. But for sure it would resolve the "-1 for everyone but my friend" problem.
(or Ranked pairs for that matter)
Quote:
I am not going after statistics here, maybe Mercury would have won this year with more points or not all all. We do not know exactly how voting is influenced. But I think we can agree that is has "some" impact.
If you don't know *how* voting is influenced, how can you know it's influenced at all?
Quote:
Now imagine for a moment the 64k result would have been the other way around and Conspiracy would have won by 2 points. Since the last position is assumed to have an advantage, I am sure Mercury (or some supporters) would have started a discussion about the order. Using (controlled) randomization would put you in a safe position here.
Now imagine for a moment that I could code, made a 64k and won by 4 points!!!
And after that, I rode away on a firebreathing dragon while the Conspiracy and Mercury underlings lied on the ground crying in fear!!!! And no supporter of any group would dare utter a word, lest they'd awake the wrath of the mighty King lug00ber of the North!!!!!
There'd be no safe positions anywhere, because my army of 20ft tall wolves would roam Saarbrücken in search of warm human flesh!!!!!
I mean, if we're gonna make shit up, I like that story way better than what you came up with.
I wouldn't mind ranking, but it'd be a bit hard to do live voting ranking since you don't know all the entries ahead of time and you might forget stuff about ones that came earlier if you wait. You could go "Is it better than [song n - 2]?" etc but the further you are along in the compo the less you might remember about ones before, even if you liked them a lot. It would really only work afterwards when Partymiester lets you go and listen to them again in the voting area.
Quote:
I mean, if we're gonna make shit up, I like that story way better than what you came up with.
Yea, I'll leave that to you. You have much more talent there.
Would it be unbearable to accept the voting results as an artefact that reflects unknown, complex and uncontrollable, human aspects of the voters and their interactions and the occassion, as much as, or even more than it reflects the prods. Without trying to filter out some "unwanted" aspects of human behaviour and distil some kind of a reproducible "measurement" that you can analyze. Is it too frightening if it can't be analyzed and fine-tuned and managed like a business process or something.
I don't think that's the case - I simply wondered if the use of anonymous voting is a good thing or not.
The discussion has since broadened a bit but I think the initial question still stands - at this point we also have at least one Revision organizer who expressed their dislike of hidden author names, so that's something, right? :)
The discussion has since broadened a bit but I think the initial question still stands - at this point we also have at least one Revision organizer who expressed their dislike of hidden author names, so that's something, right? :)
JTZ, given who is in Mercury and Conspiracy, I am absolutely positively certain that Mercury would not have complained about the outcome or started a thread on it and that this thread was not started to complain about the outcome of the 64k compo. Hell, that compo isn't even anonymous!
So no, quit trashing Gargaj, Conspiracy, and the friendly guys behind Mercury.
So no, quit trashing Gargaj, Conspiracy, and the friendly guys behind Mercury.
I don't feel he was trashing? Maybe a bit of presumption, but that's fine.
Exactly. And I thought I made that clear. If not I will try again: I am NOT assuming that any member of the two groups would complain. I don't know the people involved into those great prods, so how should I assume something like that?
I was using their names because of the very close result of this year's PC64K compo. It could be any prod from anyone. I doesn't even have to be demoscene related.
To make it even more abstract:
In a competition (demo, ESC or whatever) the entries are presented in a manually selected order defined by group or person A. Entry Y is shown last and wins by a small margin over an entry X which was shown earlier. In such a case, there is a potential risk that X (or any person preferring X over Y) may accuse A of influencing the final result by putting the voter bias towards Z.
And to prevent this ever happening to A, I suggest using some controlled, limited(!) randomization of the entries.
I was using their names because of the very close result of this year's PC64K compo. It could be any prod from anyone. I doesn't even have to be demoscene related.
To make it even more abstract:
In a competition (demo, ESC or whatever) the entries are presented in a manually selected order defined by group or person A. Entry Y is shown last and wins by a small margin over an entry X which was shown earlier. In such a case, there is a potential risk that X (or any person preferring X over Y) may accuse A of influencing the final result by putting the voter bias towards Z.
And to prevent this ever happening to A, I suggest using some controlled, limited(!) randomization of the entries.
Quote:
...the voter bias towards Z.
Y (not Z)
JTZ: But randomization will still create that favor (if it exists); no matter what you do someone is going to be played last. Placing the best entry last is at least understandable, in the sense that someone with expertise / knowledge made an assumption that that specific entry has the best chance to win anyway, and as such it deserves the "home turf". It's like how in boxing there's an assumed scoring bias towards the champion.
As a whole, it would make more sense to me from the audience point of view to structure the competition like a DJ set or a music album. You don't play the "best" songs last, you try to keep up the flow and the proper mood: slowly raise the tempo and then end up with something that's emotionally releasing. I guess in a context like a demo compo it might sometimes be a challenge though. I think an approach like this would also give each demo a proper context, ie. the ones with less untz untz untz might not be forgotten so easily in the middle of the fireworks.
Quote:
Now perhaps. When you originally wrote what I referenced, not really:JTZ:
And I thought I made that clear.
Quote:
I was indicating that I'm pretty sure they wouldn't. I already gave reasons why not. On top of those, you had two very strong demos in a row. People around me were still at least mentioning the penultimate demo.JTZ:
Now imagine for a moment the 64k result would have been the other way around and Conspiracy would have won by 2 points. Since the last position is assumed to have an advantage, I am sure Mercury (or some supporters) would have started a discussion about the order.
None of the previous invalidates the conversation that the order of a compo might make a difference in voting. Just that those involved in this particular case would likely not raise issues about the ordering. It would be very petty.
Quote:
I figure with most of us adults, at least physically, we can look beyond the fact that someone we know released a prod, or that "stuff from hoffmann is typically good".
Everything we know about cognitive biases disputes this. Sure, you might be able to reframe your decision after the fact, but your brain is in control when it comes to making judgements like that.
Quote:
But randomization will still create that favor (if it exists); no matter what you do someone is going to be played last. Placing the best entry last is at least understandable, in the sense that someone with expertise / knowledge made an assumption that that specific entry has the best chance to win anyway, and as such it deserves the "home turf". It's like how in boxing there's an assumed scoring bias towards the champion.
There's also the traditional way of ranking teams in sports, by grouping the best ones together in playoffs. It would be unfair to group "LOL!COMPOFILLER!!111 by LOLz" against "Dreamscape by Andromeda Software Development" for example.
Where many people object to "Why do the compo crew place the demo they judge to be the best at the end?!", in compos with more than one good prod it's more often about "creating blocks of demos that are of equal or higher level". So don't think about the absolute last demo, but rather, for example, the last three or something. It just makes sense to put good demos next to each other.
in the end that is more or less my idea. Within the groups the order would have to be somewhat random, else grouping would make no sense.
And it also eliminates part of the bias ("someone with knowledge has put this last, so this must be the best...") for the very last entry.
And it also eliminates part of the bias ("someone with knowledge has put this last, so this must be the best...") for the very last entry.
Also there are no champions here in the same sense as in boxing i.e. defending the title. So, just be careful with all this sports metaphors;)
Of course there are :)
Quote:
("someone with knowledge has put this last, so this must be the best...")
In that case the person should probably just refrain from voting. I mean, in the case where they just rely on others to tell them what to vote (either directly or indirectly (by the assumption you just declared)), they should just not vote.
Quote:
in the end that is more or less my idea. Within the groups the order would have to be somewhat random, else grouping would make no sense.
And it also eliminates part of the bias ("someone with knowledge has put this last, so this must be the best...") for the very last entry.
I'd rather watch a compo in an order that makes sense than one that might mildly compromise the my entry's eventual placement in the results. At Revision 2016, United Force/DD was shown as the last demo, they ended up seventh. At Assembly 2015 our demo Emix was shown as the second demo in the compo (for reasons I'm not gonna go into, it's unconfirmed), yet we ended up fifth, a fairly reasonable end result considering the compo. Cases like this I think show (note: "show", not "prove") that where you end up being in the line up doesn't automatically dictate the results to any great extent. I'm sure it does to some small degree, but I'm not seeing any great bias from just showing a demo the last in the compo. And I'll echo it again that in any case, the enjoyability of a compo matters to me more than the results.
Sure, they defend "world domination" title!
Whoops, quoting kinda failed there... the first line is not mine obviously.
Quote:
At Revision 2016, United Force/DD was shown as the last demo, they ended up seventh.
Yeah, this was an odd scheduling choice, because everyone watching expected "Oooh, here comes the blockbuster!" IMO this hurt their final placement.
Quote:
Yeah, this was an odd scheduling choice, because everyone watching expected "Oooh, here comes the blockbuster!" IMO this hurt their final placement.
I'm not entirely sure I'd go that far, but it's certainly possible, and a good point that that kind of an adverse effect can happen too.