One of our prime directives for BoardGaming.com is to provide quality content and encourage quality content to be submitted by others.
In general, allowing users to “like” or give a “thumbs up” to another person’s review is a great way to encourage quality. On BoardGaming.com we call this action “grading” because that term fits nicely into the theme of the Professor skill class. You grade a review or tip by responding to the question “Was this review (or tip) helpful?”
We are concerned, though, that providing incentive for grading can lead to people not really thinking about the grade they are giving. We want to avoid having a person just click through all the reviews and give a random “yes” or “no” without thinking. As it is, the “grading” quest tends to encourage quantity only.
We definitely want to encourage people to grade reviews and tips, but how can we reward people for doing it thoughtfully, while preventing people from abusing the system by giving random “yes” or “no” responses?
We’re going to open this up for discussion and see if the community can collectively come up with a solution to this problem. For now, here are a few ideas to consider (none of which are ideal):
Solution #1: Daily limit – Like some other quests, we could place a limit on the number of grades you can give each day. We’ve used this technique for some quests, but for grading reviews and tips there are plenty of reasons why this would not be a good idea. We want to encourage people to read reviews, and the natural time to grade the review is immediately after reading it.
Solution #2: Less Valuable Rewards – At first, this might look like a good solution, but it could also backfire. Lower rewards could actually encourage exploitation (because it would tend to encourage quantity over quality).
Solution #3: Limited Rewards – Instead of continuing to give rewards for grading, we could cut off the rewards after you’ve graded X number of reviews or tips (such as cutting the quest series off at the “Amateur Grader” level). So far, this is the best idea we’ve been able to come up with to prevent the exploit … but it also means that we couldn’t reward the people who really DO put thought into their grading.
Solution #4: Pay to Grade – We could make it cost BG gold to grade a review. But, I don’t want to use that approach, because we’re already planning on implementing a different system for rewarding really good game reviews and tips … okay I’ll say it … paying BG gold to give a review or tip a “heart”.
Please remember to be polite to other users as you discuss ideas.
Update 8/16/2011
Here is a quick summary of some of the ideas from the comments …
- Adding a Neutral option
- Diminishing returns
- Sort reviews by Yes/No (really a separate topic)
- Including comments/feedback with the grades
- Limit based on Professor level
- Limit based on # of likes you’ve received yourself
- Adding a time-delay before the Yes/No option appears
- Only reward XP for a review after a threshold % or # of likes is received
- Add quick impressions to the Dicussion page instead [Done!]
- Issues with recent reviews getting bumped and never seen
Looks like our automatic filter didn’t want to let you put deity in the comment… Yet another wonderful bug. I was telling one user that I want our illustrator to create an image of our developer completely covered with little bugs, where only his hand is sticking out, desperately typing on a keyboard 🙂
There is no place to submit “short game impressions”. Thus, many people are writing “reviews” that are, in reality, their impression of the game. That is one definition of a review. It’s not the kind of review that we’re used to seeing.
LOL! Yes, I will temper my excitement. I think the site is great, and it looks like I’m not alone. I have come to find that the gaming community is amazingly passionate… which can sometimes lead to harsh judgements, but is almost always entertaining and useful. We seem to have and endless capacity for discussion of this passion, and I’m certain this site will never lack useful information because the community is relentless. May God be with you. 🙂
@ericbjohnson ~ I’m not suggesting each review should be a book. Your reviews of Dominion & Small World are both succinct and helpful. There is a big difference between “succinct and helpful” and “short and unhelpful.”
Anyway. We’ll see how it shakes out, as Jim says.
@ericbjohnson This is why we want to let things sit for awhile before we make any big changes… Seeing how the community reacts and changes as the site gets more use is very important.
Really appreciate the comments. Our developer of the site is insanely busy, so we would ask your patience when you see a bug, or the fact that rating things as helpful is limited to the home page for now. The game side of the site has a two fold mission for us. Making it fun to use the site to attract more people to the world of board games, and second, to encourage quality content on the site. Yes, there are people just doing things for points, but the system for rating reviews and tips should help the good stuff stand out.
The main thing is that everything takes time to implement. Hopefully we’ve done a good job at making the site accessible and easy to use, but unfortunately that doesn’t mean that what we do on our end is at all easy or quick. If we had a staff of 10+ people, we wouldn’t have that issue 🙂 Thanks for your support!
I think it’s important to realize that everyone is new to this site. When I first saw the “reviews”, it appeared to be more like a comment area… because that is how it was being treated by the first few reviewers. Now, it appears people want more elaborate reviews… or maybe opinions are mixed. For instance, I do not mind clear and concise. So, maybe there should be a comment area for ratings, and a more extensive Review section. That way people who want to make a quick comment on a game they like do not get chastised for it, and those who want to review the game are required to answer questions in a form based process that will help provide a richer review experience. A form based system would and a more objective system for reviewing reviews as well.
Reviews that get no positive feedback should get no reward. At all. Most of the reviews I read so far assume I’ve already played the game. Many are one or two lines.
Reviews like that are just clutter, and not a credit to the site.
/2cents
Perhaps a solution that would add value to the site, and help to spot/stop the current abuse is once you have graded, a text box comes up prompting in brief why was or was not the review/tip helpful? I think by making people respond to this it show down the process and brings back to focus why grading occurs. I am not sure what would be done with the additional data generated, but someone who fills it out with gibberish is most likely ganging the system.
I think the solution would be a combination of Solution #1 (Daily Limit) and sorting the reviews, tips, etc. by user ratings. That way the ones up top (higher ratings reviews) tend to get more views because of their higher quality content (as graded by readers). That and only awarding xp once a threshold of “likes” is achieved.
I haven’t been judging on grammar and spelling, because I have to look at my stuff 2 or 3 types to catch all the errors and some still trickle through. I wish the edit button was there, because I need to fix over dozen of my entries already:/
I thought about the flip side of the grading issue that might encourage people to write better reviews or tips. Give the reviewer the XP only when they get a set number of likes by graders (i.e. need 5 likes or +5 rating for credit).
I agree with adding a neutral button. When I grade a review, I look for content and details (as well as grammar,spelling and the like). There are some that are honestly in the middle for me. They are on the right track to writing a good review but not there yet.
As I was thinking about this, the question that came to me is, how can we get people to spend the time to read and think about what they’re reading? Could you make it that a review has to be “open” by the user for a certain amount of time before the grading option becomes active?
If you couldn’t grade something unless it was open for 10 seconds (or whatever time you choose), it could make it more likely people will read it. I know there are reviews that are too short and this can be noted quickly, but hopefully those will become the exception on the site.
Once a certain period of time has passed since the item has been expanded, the Yes/No/(Neutral?) box pops-up/activates.
One could still “game” the system, but then they’ll have to put in more time to do it, which could help deter this activity.
Of course, I say this as I wouldn’t have to implement the code to make it work…
Actually, the whole point is to enjoy gaming.
I think the good reviews just need to sink to the bottom at some point (certain number of “no” ratings… or negative sum). Poor scores will cause reviewers to rethink their reviews and rewrite them once the editing feature gets put into place. People who don’t care about reviews,etc will eventually dissipate. Let’s have fun and not get too serious. The site is about games after all. 🙂
I disagree with Lawcomic. Exploiting the quests adds no value to the site. The whole point is for the reviews, tips, etc is to help make informative buying decisions. The whole badges and perks is a sidebar.
Open up grading to all reviews, and don’t worry about the exploit. People are exploiting the number of reviews, the number of tips, etc. Let them. The badges are just for fun anyway.
I also think that the reviews are sub-par due to people trying to get xp. Something should be done to address that aswell. I’ve read many one-line reviews that don’t add to the site.
#1 all the way across the sky.
I find it a bit disheartening that there are those out there that would abuse the system. However, I believe that BoardGaming is taking a step in the right direction. I believe that a slow and steady rate is a great way to reward casual and dedicated browsers alike. Nobody should be able to do too much in a single day.
Eric, I’ve noticed this too, and I’d like it to be fixed, but it’s not an issue. Go ahead and click yes or no (I did it thinking maybe my review didn’t go through). If you’ve already submitted your response to a review, then it will not update and just say “Your response has been saved”. So there’s no taking advantage of that feature.
Gracias Dolus. I see, I see! BTW This proves the need for forums, yet again. – I Beat Dead Horses Mercilessly. >:(