Quest Discussion – Grading Reviews and Tips

Posted by admin24 | 15-Aug-11 | 74 comments

One of our prime directives for BoardGaming.com is to provide quality content and encourage quality content to be submitted by others.

Novice Grader In general, allowing users to “like” or give a “thumbs up” to another person’s review is a great way to encourage quality. On BoardGaming.com we call this action “grading” because that term fits nicely into the theme of the Professor skill class. You grade a review or tip by responding to the question “Was this review (or tip) helpful?”

We are concerned, though, that providing incentive for grading can lead to people not really thinking about the grade they are giving. We want to avoid having a person just click through all the reviews and give a random “yes” or “no” without thinking. As it is, the “grading” quest tends to encourage quantity only.

We definitely want to encourage people to grade reviews and tips, but how can we reward people for doing it thoughtfully, while preventing people from abusing the system by giving random “yes” or “no” responses?

We’re going to open this up for discussion and see if the community can collectively come up with a solution to this problem. For now, here are a few ideas to consider (none of which are ideal):

Solution #1: Daily limit – Like some other quests, we could place a limit on the number of grades you can give each day. We’ve used this technique for some quests, but for grading reviews and tips there are plenty of reasons why this would not be a good idea. We want to encourage people to read reviews, and the natural time to grade the review is immediately after reading it.

Solution #2: Less Valuable Rewards – At first, this might look like a good solution, but it could also backfire. Lower rewards could actually encourage exploitation (because it would tend to encourage quantity over quality).

Solution #3: Limited Rewards – Instead of continuing to give rewards for grading, we could cut off the rewards after you’ve graded X number of reviews or tips (such as cutting the quest series off at the “Amateur Grader” level). So far, this is the best idea we’ve been able to come up with to prevent the exploit … but it also means that we couldn’t reward the people who really DO put thought into their grading.

Solution #4: Pay to Grade – We could make it cost BG gold to grade a review. But, I don’t want to use that approach, because we’re already planning on implementing a different system for rewarding really good game reviews and tips … okay I’ll say it … paying BG gold to give a review or tip a “heart”.

Please remember to be polite to other users as you discuss ideas.

Update 8/16/2011

Here is a quick summary of some of the ideas from the comments …

  • Adding a Neutral option
  • Diminishing returns
  • Sort reviews by Yes/No (really a separate topic)
  • Including comments/feedback with the grades
  • Limit based on Professor level
  • Limit based on # of likes you’ve received yourself
  • Adding a time-delay before the Yes/No option appears
  • Only reward XP for a review after a threshold % or # of likes is received
  • Add quick impressions to the Dicussion page instead [Done!]
  • Issues with recent reviews getting bumped and never seen

Comments (74)

Gamer Avatar
6
Pick a Favorite LGS
Robots on the Line fan
Miniature Painter
I play blue

Looks like our automatic filter didn’t want to let you put deity in the comment… Yet another wonderful bug. I was telling one user that I want our illustrator to create an image of our developer completely covered with little bugs, where only his hand is sticking out, desperately typing on a keyboard 🙂

Gamer Avatar
3
Gamer - Level 3
Critic - Level 2

There is no place to submit “short game impressions”. Thus, many people are writing “reviews” that are, in reality, their impression of the game. That is one definition of a review. It’s not the kind of review that we’re used to seeing.

Gamer Avatar
9
USA
Platinum Supporter
Petroglyph
BoardGaming.com Beta 1.0 Tester

LOL! Yes, I will temper my excitement. I think the site is great, and it looks like I’m not alone. I have come to find that the gaming community is amazingly passionate… which can sometimes lead to harsh judgements, but is almost always entertaining and useful. We seem to have and endless capacity for discussion of this passion, and I’m certain this site will never lack useful information because the community is relentless. May God be with you. 🙂

Gamer Avatar
4
Viscount / Viscountess
Advocate
BoardGaming.com Beta 1.0 Tester

@ericbjohnson ~ I’m not suggesting each review should be a book. Your reviews of Dominion & Small World are both succinct and helpful. There is a big difference between “succinct and helpful” and “short and unhelpful.”

Anyway. We’ll see how it shakes out, as Jim says.

Gamer Avatar
6
Pick a Favorite LGS
Robots on the Line fan
Miniature Painter
I play blue

@ericbjohnson This is why we want to let things sit for awhile before we make any big changes… Seeing how the community reacts and changes as the site gets more use is very important.

Gamer Avatar
6
Pick a Favorite LGS
Robots on the Line fan
Miniature Painter
I play blue

Really appreciate the comments. Our developer of the site is insanely busy, so we would ask your patience when you see a bug, or the fact that rating things as helpful is limited to the home page for now. The game side of the site has a two fold mission for us. Making it fun to use the site to attract more people to the world of board games, and second, to encourage quality content on the site. Yes, there are people just doing things for points, but the system for rating reviews and tips should help the good stuff stand out.

The main thing is that everything takes time to implement. Hopefully we’ve done a good job at making the site accessible and easy to use, but unfortunately that doesn’t mean that what we do on our end is at all easy or quick. If we had a staff of 10+ people, we wouldn’t have that issue 🙂 Thanks for your support!

Gamer Avatar
9
USA
Platinum Supporter
Petroglyph
BoardGaming.com Beta 1.0 Tester

I think it’s important to realize that everyone is new to this site. When I first saw the “reviews”, it appeared to be more like a comment area… because that is how it was being treated by the first few reviewers. Now, it appears people want more elaborate reviews… or maybe opinions are mixed. For instance, I do not mind clear and concise. So, maybe there should be a comment area for ratings, and a more extensive Review section. That way people who want to make a quick comment on a game they like do not get chastised for it, and those who want to review the game are required to answer questions in a form based process that will help provide a richer review experience. A form based system would and a more objective system for reviewing reviews as well.

Gamer Avatar
4
Viscount / Viscountess
Advocate
BoardGaming.com Beta 1.0 Tester

Reviews that get no positive feedback should get no reward. At all. Most of the reviews I read so far assume I’ve already played the game. Many are one or two lines.

Reviews like that are just clutter, and not a credit to the site.

/2cents

Gamer Avatar
4
Gamer - Level 4
Advocate

Perhaps a solution that would add value to the site, and help to spot/stop the current abuse is once you have graded, a text box comes up prompting in brief why was or was not the review/tip helpful? I think by making people respond to this it show down the process and brings back to focus why grading occurs. I am not sure what would be done with the additional data generated, but someone who fills it out with gibberish is most likely ganging the system.

Gamer Avatar
4
Gamer - Level 4
Junior
Novice Reviewer
Knight

I think the solution would be a combination of Solution #1 (Daily Limit) and sorting the reviews, tips, etc. by user ratings. That way the ones up top (higher ratings reviews) tend to get more views because of their higher quality content (as graded by readers). That and only awarding xp once a threshold of “likes” is achieved.

Gamer Avatar
8
Professional Reviewer
BoardGaming.com Beta 1.0 Tester
Silver Supporter
Sentinels of the Multiverse fan

I haven’t been judging on grammar and spelling, because I have to look at my stuff 2 or 3 types to catch all the errors and some still trickle through. I wish the edit button was there, because I need to fix over dozen of my entries already:/

I thought about the flip side of the grading issue that might encourage people to write better reviews or tips. Give the reviewer the XP only when they get a set number of likes by graders (i.e. need 5 likes or +5 rating for credit).

Gamer Avatar
3
Advanced Grader
BoardGaming.com Beta 1.0 Tester
Amateur Reviewer

I agree with adding a neutral button. When I grade a review, I look for content and details (as well as grammar,spelling and the like). There are some that are honestly in the middle for me. They are on the right track to writing a good review but not there yet.

Gamer Avatar
10
Critic - Level 5
Professional Advisor
Expert Reviewer
Marquis / Marchioness

As I was thinking about this, the question that came to me is, how can we get people to spend the time to read and think about what they’re reading? Could you make it that a review has to be “open” by the user for a certain amount of time before the grading option becomes active?

If you couldn’t grade something unless it was open for 10 seconds (or whatever time you choose), it could make it more likely people will read it. I know there are reviews that are too short and this can be noted quickly, but hopefully those will become the exception on the site.

Once a certain period of time has passed since the item has been expanded, the Yes/No/(Neutral?) box pops-up/activates.

One could still “game” the system, but then they’ll have to put in more time to do it, which could help deter this activity.

Of course, I say this as I wouldn’t have to implement the code to make it work…

Gamer Avatar
9
USA
Platinum Supporter
Petroglyph
BoardGaming.com Beta 1.0 Tester

Actually, the whole point is to enjoy gaming.

I think the good reviews just need to sink to the bottom at some point (certain number of “no” ratings… or negative sum). Poor scores will cause reviewers to rethink their reviews and rewrite them once the editing feature gets put into place. People who don’t care about reviews,etc will eventually dissipate. Let’s have fun and not get too serious. The site is about games after all. 🙂

Gamer Avatar
4
Gamer - Level 4
Junior
Novice Reviewer
Knight

I disagree with Lawcomic. Exploiting the quests adds no value to the site. The whole point is for the reviews, tips, etc is to help make informative buying decisions. The whole badges and perks is a sidebar.

Gamer Avatar
3
Rated 10 Games
Intermediate Grader
Tinkerer

Open up grading to all reviews, and don’t worry about the exploit. People are exploiting the number of reviews, the number of tips, etc. Let them. The badges are just for fun anyway.

Gamer Avatar
4
Gamer - Level 4
Junior
Novice Reviewer
Knight

I also think that the reviews are sub-par due to people trying to get xp. Something should be done to address that aswell. I’ve read many one-line reviews that don’t add to the site.

Gamer Avatar
6
Gamer - Level 6
Vanguard
Professional Grader
I'm a Gamin' Fiend!

#1 all the way across the sky.

Gamer Avatar
4
Tasty Minstrel Games Fan
Pet Lover

I find it a bit disheartening that there are those out there that would abuse the system. However, I believe that BoardGaming is taking a step in the right direction. I believe that a slow and steady rate is a great way to reward casual and dedicated browsers alike. Nobody should be able to do too much in a single day.

Gamer Avatar
9
USA
Platinum Supporter
Petroglyph
BoardGaming.com Beta 1.0 Tester

Eric, I’ve noticed this too, and I’d like it to be fixed, but it’s not an issue. Go ahead and click yes or no (I did it thinking maybe my review didn’t go through). If you’ve already submitted your response to a review, then it will not update and just say “Your response has been saved”. So there’s no taking advantage of that feature.

Gracias Dolus. I see, I see! BTW This proves the need for forums, yet again. – I Beat Dead Horses Mercilessly. >:(

Add a Comment

You must be to add a comment.

× Visit Your Profile