Quest Discussion – Grading Reviews and Tips

Posted by admin24 | 15-Aug-11 | 74 comments

One of our prime directives for BoardGaming.com is to provide quality content and encourage quality content to be submitted by others.

Novice Grader In general, allowing users to “like” or give a “thumbs up” to another person’s review is a great way to encourage quality. On BoardGaming.com we call this action “grading” because that term fits nicely into the theme of the Professor skill class. You grade a review or tip by responding to the question “Was this review (or tip) helpful?”

We are concerned, though, that providing incentive for grading can lead to people not really thinking about the grade they are giving. We want to avoid having a person just click through all the reviews and give a random “yes” or “no” without thinking. As it is, the “grading” quest tends to encourage quantity only.

We definitely want to encourage people to grade reviews and tips, but how can we reward people for doing it thoughtfully, while preventing people from abusing the system by giving random “yes” or “no” responses?

We’re going to open this up for discussion and see if the community can collectively come up with a solution to this problem. For now, here are a few ideas to consider (none of which are ideal):

Solution #1: Daily limit – Like some other quests, we could place a limit on the number of grades you can give each day. We’ve used this technique for some quests, but for grading reviews and tips there are plenty of reasons why this would not be a good idea. We want to encourage people to read reviews, and the natural time to grade the review is immediately after reading it.

Solution #2: Less Valuable Rewards – At first, this might look like a good solution, but it could also backfire. Lower rewards could actually encourage exploitation (because it would tend to encourage quantity over quality).

Solution #3: Limited Rewards – Instead of continuing to give rewards for grading, we could cut off the rewards after you’ve graded X number of reviews or tips (such as cutting the quest series off at the “Amateur Grader” level). So far, this is the best idea we’ve been able to come up with to prevent the exploit … but it also means that we couldn’t reward the people who really DO put thought into their grading.

Solution #4: Pay to Grade – We could make it cost BG gold to grade a review. But, I don’t want to use that approach, because we’re already planning on implementing a different system for rewarding really good game reviews and tips … okay I’ll say it … paying BG gold to give a review or tip a “heart”.

Please remember to be polite to other users as you discuss ideas.

Update 8/16/2011

Here is a quick summary of some of the ideas from the comments …

  • Adding a Neutral option
  • Diminishing returns
  • Sort reviews by Yes/No (really a separate topic)
  • Including comments/feedback with the grades
  • Limit based on Professor level
  • Limit based on # of likes you’ve received yourself
  • Adding a time-delay before the Yes/No option appears
  • Only reward XP for a review after a threshold % or # of likes is received
  • Add quick impressions to the Dicussion page instead [Done!]
  • Issues with recent reviews getting bumped and never seen

Comments (74)

Gamer Avatar
9
USA
Platinum Supporter
Petroglyph
BoardGaming.com Beta 1.0 Tester

I just think there is a nice synergy between creating a review for XP, getting feedback on your review and getting XP for “reviewing” reviews. Ultimately, it doesn’t hurt anyone if someone does just “click” through unless everyone is doing it. If they aren’t really reading the reviews, they aren’t really using the site, and will eventually not come back (sad but true). Ultimately, the Quest is gone at some point, and we’ll keep reviewing reviews or another Quest will be added… maybe with a unique comb requirement that isn’t as easy to game.

Gamer Avatar
8
Senior
Critic - Level 3
Junior Reporter
Explorer - Level 4

Another option is to tie the number of ratings available to the number of thumbs-up each member has received for their own reviews, tips, etc. (e.g., three ratings allowed for each thumbs-up the member received for his/her own review or tip). That would provide incentive to create good reviews and also limit the abuse. We’d probably also need to get a few freebies every day on top of it to help people get started and to cover slow days when few reviews are given.

As for neutral, I’d not implement that. We should vote things up or down only. If I am ambivalent about something, then I should just let it be. There is no reason to rate every single piece of content.

If my available ratings are tied to thumbs-up’s on my own reviews, then the neutral condition is covered, i.e., there’s no reward for no ratings and a thumbs-down can still be helpful for showing a bad review.

Gamer Avatar
8
Professional Reviewer
BoardGaming.com Beta 1.0 Tester
Silver Supporter
Sentinels of the Multiverse fan

I like the idea of a neutral option.

I’d lean towards not a daily limit, but I think a monthly limit would be more preferable.

Implement a “review queue” for users where they can add a review to their personal queue for review later. That way when they hit the limit, they can go back once the next month is done. This could actually give people time to think about the review more if they hit the limit, and if they really want to rate it, they have it on queue for later.

A reward diminishing returns for the reviewer sounds good to me, and it can be based on grader level. This may sound odd, but I think the people who are serious about reviewing and the content will continue to review no matter what. The people who receive the grade still get the same benefit, and it doesn’t diminish on their side.

Requiring a reason for the tip is a good idea, because it might force people to think about their grade a little more then just blindly saying “yes” or “no”.

Gamer Avatar
7
Marquis / Marchioness
Advanced Reviewer
Professional Advisor
BoardGaming.com Beta 1.0 Tester

Eric, I’ve noticed this too, and I’d like it to be fixed, but it’s not an issue. Go ahead and click yes or no (I did it thinking maybe my review didn’t go through). If you’ve already submitted your response to a review, then it will not update and just say “Your response has been saved”. So there’s no taking advantage of that feature.

Gamer Avatar
9
USA
Platinum Supporter
Petroglyph
BoardGaming.com Beta 1.0 Tester

It pains me to say this, but I found a bug (I can’t believe your telling him this… Don’t listen to him, he’s DELUSIONAL! ~ Evil Voice Inside My Head). If you select “yes” or “no” on a review and then leave the page and come back… the option is there again. This could really be abused (*************). I’ve probably done this unknowingly (Keep telling yourself that pretty boy… whatever helps you sleep at night~EVIMH). Sigh.

Gamer Avatar
5
USA
Knight
BoardGaming.com Beta 1.0 Tester

Overall I think here are a few things that can be put in place.

1) A limit on the number of reviews possible daily, modified by your professor level. Potentially different limitations on each type of grading, from reviews to tips and strategies. this could also help make sure that tips are being paid attention to.

2) he ability to put a “Neutral” vote in. I really want this feature, I find myself wishing for this as I vote down, or ignore posts that say “his game is one of my favorites.” and nothing else.

3) I wan to see posts that are overall negative to be moved to the bottom or collapsed, so that I can get to the posts that are decidedly useful in relation to a game.

Gamer Avatar
7
Marquis / Marchioness
Advanced Reviewer
Professional Advisor
BoardGaming.com Beta 1.0 Tester

/Wishing I could edit my last post.

I do not want to see limiting the number of grades per day. There are so many other quests that depend on getting graded that it would be detrimental to the other quests if they were limited. It’s also not something you want to limit. It’s like any shopping or help website, you want to allow your users to say yay or nae to posts/products to help people recognize good from bad and quickly filter them out.

I think adding a “neutral” option is good, because it allows people to select something on everything, which I think most people will want to do if they’re getting rewarded for it, but prevents skewing based on random “Good” or “Bad” ratings.

Reading Okkay’s third option, I’m not sure if points are awarded per review or not. I don’t think they need to be, if they are. I think the quest reward itself is sufficient enough.

Gamer Avatar
9
USA
Platinum Supporter
Petroglyph
BoardGaming.com Beta 1.0 Tester

So far, I’ve read almost every review on the site. I have to admit that the XP is in the back of my mind, but it also got me to read some reviews I might have skipped (because I just wouldn’t have bothered). The reason I’ve read most of the reviews, is the game selection on here is mostly good games that have been around awhile AND you have mostly seasoned gamers giving reviews at this point (short or long). Eventually, we won’t bother reading newbie reviews… especially if the incentive is removed. ALSO: I’ve shifted my review style to a more descriptive one… because my initial simple reviews were being blasted. Without that feedback, I might have continued down that dark path. In short, I liked the incentive laden system, and think it will even out over time. A Neutral score might be good to add though. I’d give some of the weak reviews a Neutral rather than a BAD.

Gamer Avatar
7
Marquis / Marchioness
Advanced Reviewer
Professional Advisor
BoardGaming.com Beta 1.0 Tester

Okkay’s second point was the main one I was thinking of, and was actually a feature I would like to see, being able to comment one somebody’s tip and have to comment to be able to thumbs up or down.

Pro: I can explain why I support or don’t support somebody’s tip. This can also lead to discussion on tips/strategies/house rules.

Con: This can create a lot of discussion. You’ll need to add a “Hide/View” option next to each tip to hide or view all of the comments, and that could potentially be heavy on the server. But forcing people to comment will also discourage doing too much of it, so it will probably be manageable.

Gamer Avatar
2
Amateur Grader

A couple suggestions:
1. Add a “Neutral” option to whether the review was helpful or not. I like to XP grind as much as anyone, but I’ve read a lot of reviews I was so-so on, that I wasn’t comfortable giving either a “thumbs up” or a “thumbs down”. But since I wanted the XP, I picked one anyways. Giving people a neutral option would allow them to XP grind without affecting the score with a random rating.

2. In order to prevent people from mindlessly voting, maybe require them to put down a short reasoning behind their vote – something like “too short” or “great game explanation”, etc. This could make it easier for the moderators to find out the people who don’t actually read the reviews before voting. Or send these vote responses to the reviewer for them to read through – either they will get feedback for how to improve their reviews in the future, or they can report the responses that they feel are not valid (with some sort of check to ensure people don’t just report every single bad vote).

3. Maybe have some sort of diminishing returns system for gaining rewards from voting, instead of just a hard cap. That way, people can vote as much as they want, but for those who are only doing it for points, at some point it becomes not worth their time to be only gaining 1/100th of the normal point gain for each vote.

Gamer Avatar
6
Pick a Favorite LGS
Robots on the Line fan
Miniature Painter
I play blue

@ D_S I agree that the “wasn’t helpful” ratings are equally as important as the “was helpful” ratings. The data is being collected, and we hope to soon implement the ability to filter reviews by “most helpful,” to make sure the quality content users are adding stands out.

Gamer Avatar
6
Pick a Favorite LGS
Robots on the Line fan
Miniature Painter
I play blue

8/15/2011: “Grading” game reviews and game tips is currently only allowed from the Home page, until we can work out a way to avoid having these quests exploited.

Gamer Avatar
2
Amateur Grader

Is there a bug with being able to rate things as helpful? The only place where it gives me the button to rate something is on the front page. If I try to go to a game’s page and read a review, there is no option for me to rate that review.

Gamer Avatar
4
Viscount / Viscountess
Advocate
BoardGaming.com Beta 1.0 Tester

Limit the number of grades per day. There is still an incentive for brainless grading, but there’s less of a penalty for having some integrity.

There is of course the incentive to write poor, incomplete *reviews*, and heaven knows I’ve seen more than a few of *those*, but you didn’t ask about that. I might suggest, once the “not helpful” ratio is sufficiently high, those reviews get hidden (“Click here to see this review” sort of thing).

Add a Comment

You must be to add a comment.

× Visit Your Profile