Comments on: To rate or not to rate http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/ Wed, 10 Mar 2010 13:54:13 +0100 http://wordpress.org/?v=2.8.4 hourly 1 By: nfreischlad http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-940 nfreischlad Thu, 11 Mar 2010 14:06:00 +0000 http://blog.jovoto.com/?p=5969#comment-940 Hi Cheshire, Thanks for your thoughts here. I think jovoto is quite a unique place. our contest model bases on the idea of the community-driven prize as well as orientation towards the needs of the client. We usually encourage people NOT to trash 'outlier' ideas which are cool but not 100% on brief, but rather pesent them to the community and see how they go. The unique part of our platform is that the clients has no say in distributing the community prize. After the contest has closed, they're free to choose any design from the pool of designs, it does not need to be the winner. Discussions about the fairness of the system and possble improvements come up all the time. Read also here http://blog.jovoto.com/2009/10/more-about-rating-criteria/ or here http://blog.jovoto.com/de/2009/07/voting-transparency-on-jovoto/, if you're inetersted. We're lagging behind in evaluating and testing all the valuable feedback we have received from community members during these discussions, but... it's coming! Thrdr ideas range from making rating even more transparent to ditching the ideas of negative votes altogether, or implementing a more differentiated rating score composed of serveral other values. Szay tuned. I think we will be testing several new rating models on jovoto early next year. Hi Cheshire,

Thanks for your thoughts here. I think jovoto is quite a unique place. our contest model bases on the idea of the community-driven prize as well as orientation towards the needs of the client. We usually encourage people NOT to trash ‘outlier’ ideas which are cool but not 100% on brief, but rather pesent them to the community and see how they go.
The unique part of our platform is that the clients has no say in distributing the community prize. After the contest has closed, they’re free to choose any design from the pool of designs, it does not need to be the winner.

Discussions about the fairness of the system and possble improvements come up all the time. Read also here http://blog.jovoto.com/2009/10/more-about-rating-criteria/ or here http://blog.jovoto.com/2009/07/voting-transparency-on-jovoto/, if you’re inetersted.

We’re lagging behind in evaluating and testing all the valuable feedback we have received from community members during these discussions, but… it’s coming! Thrdr ideas range from making rating even more transparent to ditching the ideas of negative votes altogether, or implementing a more differentiated rating score composed of serveral other values.

Szay tuned. I think we will be testing several new rating models on jovoto early next year.

]]> By: Cheshire http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-928 Cheshire Thu, 11 Mar 2010 19:14:00 +0000 http://blog.jovoto.com/?p=5969#comment-928 My 2 Cents.... Hello everyone, I am fairly new here...I joined because a friend directed me to one of the contest being currently hosted in this site. I have joined many contest before and this sort of "rating" system is pretty new to me as well. There are a ton of problems with it. I assume that the main reason people have it is because it seems logically a fair way to asset design work here, since it is indeed, rather hard to join this site. Someone mentioned above that some designs make it to the top and we don't know why, when they don't follow the brief. People are forgetting that ultimately we are working for a client not the community. You don't make something to please the community, its not pride or ego at stake, its the client request....so rating should be according to the brief. This is why we have comments, to tell the person they did a good job. Apllied them self but it doesn't fit. Just like we tell the people lower in the pool, "your design has potential but its missing something. " Artist take things to much to heart when it comes to contest. I've watched some designs that deserve placement and I've seen the light of day...and why? This so called rating system. I've been watching it. Not everyone votes. Not everyone votes according to design. It wouldn't surprise me if some people even posted a designs and it didn't get far so they just stop checking the contest. When i entered the contest, the top players had 30-40 ish votes already. While the average of person's voting was about 10-19. You can't tell me that community voted fairly here.... Also you want to speak about design and voting for it fairly? I've only see a few people here and there in the contest I entered, actually give artistic criticism. Not to mention that I've seen things get omited from the contest, people still submitting work, and people even submitting more than the max designs requested..... So you say you want fair? First you need to make things more anonymous....This whole Likes and Dislikes thing is bad...There were designs I liked- a lot- but i didn't feel like they should get a 9 or 10., but they get no recognition in my profile, their dashboard or the community dashboard. And there there designs I did not like, but because they broke the brief or common sense rules like "copy written stuff" and they got "bashed" as people call it here. It also makes people so fussy about it. Then because artist takes things to heart, they know if someone didn't like their work and thus, they may change how they vote for someone else.... I know this is comment is really long..people may or may not want to read it all the way...I could go on but in reality i don't know how to fix the problem because your asking a massive group of people to decide what best fits the breif, when normally its a small room with like maybe 4-5 people max [with tons of industry experience] or even just one guy saying "This works" or "This doesnt work, start over". My 2 Cents….

Hello everyone,

I am fairly new here…I joined because a friend directed me to one of the contest being currently hosted in this site. I have joined many contest before and this sort of “rating” system is pretty new to me as well.

There are a ton of problems with it. I assume that the main reason people have it is because it seems logically a fair way to asset design work here, since it is indeed, rather hard to join this site.

Someone mentioned above that some designs make it to the top and we don’t know why, when they don’t follow the brief. People are forgetting that ultimately we are working for a client not the community. You don’t make something to please the community, its not pride or ego at stake, its the client request….so rating should be according to the brief.

This is why we have comments, to tell the person they did a good job. Apllied them self but it doesn’t fit. Just like we tell the people lower in the pool, “your design has potential but its missing something. ”

Artist take things to much to heart when it comes to contest. I’ve watched some designs that deserve placement and I’ve seen the light of day…and why?

This so called rating system. I’ve been watching it. Not everyone votes. Not everyone votes according to design. It wouldn’t surprise me if some people even posted a designs and it didn’t get far so they just stop checking the contest. When i entered the contest, the top players had 30-40 ish votes already. While the average of person’s voting was about 10-19.

You can’t tell me that community voted fairly here….

Also you want to speak about design and voting for it fairly? I’ve only see a few people here and there in the contest I entered, actually give artistic criticism. Not to mention that I’ve seen things get omited from the contest, people still submitting work, and people even submitting more than the max designs requested…..

So you say you want fair?

First you need to make things more anonymous….This whole Likes and Dislikes thing is bad…There were designs I liked- a lot- but i didn’t feel like they should get a 9 or 10., but they get no recognition in my profile, their dashboard or the community dashboard. And there there designs I did not like, but because they broke the brief or common sense rules like “copy written stuff” and they got “bashed” as people call it here. It also makes people so fussy about it.

Then because artist takes things to heart, they know if someone didn’t like their work and thus, they may change how they vote for someone else….

I know this is comment is really long..people may or may not want to read it all the way…I could go on but in reality i don’t know how to fix the problem because your asking a massive group of people to decide what best fits the breif, when normally its a small room with like maybe 4-5 people max [with tons of industry experience] or even just one guy saying “This works” or “This doesnt work, start over”.

]]> By: Jesko http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-921 Jesko Thu, 11 Mar 2010 15:50:00 +0000 http://blog.jovoto.com/?p=5969#comment-921 Well for me, i would not like to see the bashed ideas in my dashboard. Just personally. Well for me, i would not like to see the bashed ideas in my dashboard. Just personally.

]]> By: dizzy http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-920 dizzy Thu, 11 Mar 2010 15:03:00 +0000 http://blog.jovoto.com/?p=5969#comment-920 So would you like to get a message simply saying “your idea was rated with 2 stars”? In effect, what would be your benefit? yes i would like to have this. your right its no problem for someone to dislike something if he feels that way no problem. but it would make things more transparent and ...let me give it the name "strategic voting" could be more easily spotted. ...sometimes i wonder... So would you like to get a message simply saying “your idea was rated with 2 stars”? In effect, what would be your benefit?

yes i would like to have this.
your right its no problem for someone to dislike something if he feels that way no problem.
but it would make things more transparent and …let me give it the name “strategic voting”
could be more easily spotted.
…sometimes i wonder…

]]> By: dizzy http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-919 dizzy Thu, 11 Mar 2010 14:59:00 +0000 http://blog.jovoto.com/?p=5969#comment-919 true true

]]> By: Jesko http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-912 Jesko Thu, 11 Mar 2010 23:48:00 +0000 http://blog.jovoto.com/?p=5969#comment-912 The redesign does not show the amount of loved and bashed ideas anymore. It was a quite good feature to get an idea of somebodys rating behaviour, if you see the proportion of loved and bashed ideas. The redesign does not show the amount of loved and bashed ideas anymore. It was a quite good feature to get an idea of somebodys rating behaviour, if you see the proportion of loved and bashed ideas.

]]> By: will http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-911 will Thu, 11 Mar 2010 20:16:00 +0000 http://blog.jovoto.com/?p=5969#comment-911 Something I've come across while rating is when you see a design that is clearly breaking the brief but people are giving it high scores. I think it has something to do with not wanting to put down their effort. I've seen really great work that doesn't fit the brief but I don't know what to rate it. The "description" text that goes along with a 0 is something like "trash it, start over" Well, you don't want them to "trash" that work because it's good. What you want to say is, this shouldn't win because you didn't follow the brief, but your art is really great. Maybe just a button that says "doesn't follow brief" and then you can rate it for what it deserves, which might mean you give a 10. Or maybe 2 scores than influence the main score "how well does this follow the brief?" Something I’ve come across while rating is when you see a design that is clearly breaking the brief but people are giving it high scores. I think it has something to do with not wanting to put down their effort. I’ve seen really great work that doesn’t fit the brief but I don’t know what to rate it. The “description” text that goes along with a 0 is something like “trash it, start over” Well, you don’t want them to “trash” that work because it’s good. What you want to say is, this shouldn’t win because you didn’t follow the brief, but your art is really great.

Maybe just a button that says “doesn’t follow brief” and then you can rate it for what it deserves, which might mean you give a 10. Or maybe 2 scores than influence the main score “how well does this follow the brief?”

]]> By: frieda http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-906 frieda Thu, 11 Mar 2010 10:30:00 +0000 http://blog.jovoto.com/?p=5969#comment-906 Hey Dizzy, the more ratings an idea has the less vulnerable it is for sudden changes. Obviously the 11th and 12sth ratings do have more impact than rating number 61 and 62. Each submitted rating has effects on all ideas - because each idea starts with the average rating of the contest. From their on it fights its own way, the initial (start) rating gradually looses it's influenec. So in a way ideas are rated proportinally - an idea starts with a higher average rating if the average rating in a contest is high/ the ideas in a contest are of high quality. An idea is then judged upon in perspective to the other ideas in a contest and rated accordingly. What exactly do you want to be informed about on your dasboard? We do not want to pillory a community member by sending messages like "xxx dislikes your idea" because it also should be ok to give a low rating - if an idea is not good. So would you like to get a message simply saying "your idea was rated with 2 stars"? In effect, what would be your benefit? Really interested in all you guys opinion! Please don't stop commenting! Hey Dizzy,
the more ratings an idea has the less vulnerable it is for sudden changes. Obviously the 11th and 12sth ratings do have more impact than rating number 61 and 62.
Each submitted rating has effects on all ideas – because each idea starts with the average rating of the contest. From their on it fights its own way, the initial (start) rating gradually looses it’s influenec. So in a way ideas are rated proportinally – an idea starts with a higher average rating if the average rating in a contest is high/ the ideas in a contest are of high quality. An idea is then judged upon in perspective to the other ideas in a contest and rated accordingly.

What exactly do you want to be informed about on your dasboard? We do not want to pillory a community member by sending messages like “xxx dislikes your idea” because it also should be ok to give a low rating – if an idea is not good. So would you like to get a message simply saying “your idea was rated with 2 stars”? In effect, what would be your benefit?

Really interested in all you guys opinion! Please don’t stop commenting!

]]> By: dizzy http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-899 dizzy Thu, 11 Mar 2010 17:24:00 +0000 http://blog.jovoto.com/?p=5969#comment-899 mhh hab mir den algorythmus schonmal angesehn... dennoch hab ich das gefühl, das arbeiten, die lange dabei sind und entsprechend viele votings haben etwas festgefahren sind. werden arbeiten im verhältniss zueinander bewertet? konkret: meine drei arbeiten in ihrer wertung extrem eingefallen, ich kann mir kaum vorstellen , dass bei allen drei arbeiten so viele negative bewertungen bekommen habe. ...ich würde im dashboard gerne auch über schlechte bewertungen (1 und 2) informiert werden... nur so ....hofft... mhh hab mir den algorythmus schonmal angesehn…
dennoch hab ich das gefühl, das arbeiten, die lange dabei sind und entsprechend viele votings haben etwas festgefahren sind. werden arbeiten im verhältniss zueinander bewertet? konkret: meine drei arbeiten in ihrer wertung extrem eingefallen,
ich kann mir kaum vorstellen , dass bei allen drei arbeiten so viele negative bewertungen bekommen habe.

…ich würde im dashboard gerne auch über schlechte bewertungen (1 und 2) informiert werden… nur so

….hofft…

]]> By: Gábor http://blog.jovoto.com/2009/12/to-rate-or-not-to-rate/comment-page-1/#comment-867 Gábor Thu, 11 Mar 2010 23:53:00 +0000 http://blog.jovoto.com/?p=5969#comment-867 Den Vorrednern möchte ich mich teilweise anschließen. In dem Punkt dass eine Punkteverteiltung nicht transparenter sein sollte stimme ich Jesko zu. Hier geht es ja auch um Demokratie, die am besten angesehene Idee gewinnt usw. und Wahlen laufen im Allgemeinen im Geheimen ab ... und das jovoto-System ist ja ähnlich angelegt. Wenn das System total transparent sein würde, liefe man möglicherweise Gefahr dass die Votings an sich nicht nur weniger werden könnten, sondern dass es auch nur noch "Kuschelvotings" geben könnte, da man ja selber nicht schlecht bewertet werden möchte. Das aktuelle Votingsystem finde ich in im Grunde gut. Man sieht die Lieblingsideen und die als schlecht bewertete Ideen. Jede einzelne Bewertung zu begründen (und zu sehen) halte ich für übertrieben. Eine optische "Massenbegründungsbewertung" würde den Betrachter überfordern, wer hat schon Zeit und Lust 300 Votings in einem Profil nachzulesen? Hingegen finde ich es sinnvoll (da gehe ich teilweise mit "tyler" überein) dass als schlecht bewertete Ideen einer kurzen Begründung bedürfen sollten, die man dann logischerweise auch beim "Basher" nachlesen kann. Wenn man an einem Contest teilnimmt, sollte man auch selber abstimmen dürfen. Wäre dies nicht so, hätte man optisch viel eher ein "gegeneinander" als ein "miteinander" in einem Contest. Sowohl in den Kommentaren als auch in der Bewertung untereinander. Was ich persönlich sehr schön finde, ist dass man seit längerer Zeit seine Bewertung ändern kann, je nachdem wohin sich eine Idee entwickelt. Den Vorrednern möchte ich mich teilweise anschließen. In dem Punkt dass eine Punkteverteiltung nicht transparenter sein sollte stimme ich Jesko zu. Hier geht es ja auch um Demokratie, die am besten angesehene Idee gewinnt usw. und Wahlen laufen im Allgemeinen im Geheimen ab … und das jovoto-System ist ja ähnlich angelegt.

Wenn das System total transparent sein würde, liefe man möglicherweise Gefahr dass die Votings an sich nicht nur weniger werden könnten, sondern dass es auch nur noch “Kuschelvotings” geben könnte, da man ja selber nicht schlecht bewertet werden möchte.

Das aktuelle Votingsystem finde ich in im Grunde gut. Man sieht die Lieblingsideen und die als schlecht bewertete Ideen. Jede einzelne Bewertung zu begründen (und zu sehen) halte ich für übertrieben. Eine optische “Massenbegründungsbewertung” würde den Betrachter überfordern, wer hat schon Zeit und Lust 300 Votings in einem Profil nachzulesen?

Hingegen finde ich es sinnvoll (da gehe ich teilweise mit “tyler” überein) dass als schlecht bewertete Ideen einer kurzen Begründung bedürfen sollten, die man dann logischerweise auch beim “Basher” nachlesen kann.

Wenn man an einem Contest teilnimmt, sollte man auch selber abstimmen dürfen. Wäre dies nicht so, hätte man optisch viel eher ein “gegeneinander” als ein “miteinander” in einem Contest. Sowohl in den Kommentaren als auch in der Bewertung untereinander.

Was ich persönlich sehr schön finde, ist dass man seit längerer Zeit seine Bewertung ändern kann, je nachdem wohin sich eine Idee entwickelt.

]]>