Quest Discussion – Grading Reviews and Tips

Posted by admin24 | 15-Aug-11 | 74 comments

One of our prime directives for BoardGaming.com is to provide quality content and encourage quality content to be submitted by others.

Novice Grader In general, allowing users to “like” or give a “thumbs up” to another person’s review is a great way to encourage quality. On BoardGaming.com we call this action “grading” because that term fits nicely into the theme of the Professor skill class. You grade a review or tip by responding to the question “Was this review (or tip) helpful?”

We are concerned, though, that providing incentive for grading can lead to people not really thinking about the grade they are giving. We want to avoid having a person just click through all the reviews and give a random “yes” or “no” without thinking. As it is, the “grading” quest tends to encourage quantity only.

We definitely want to encourage people to grade reviews and tips, but how can we reward people for doing it thoughtfully, while preventing people from abusing the system by giving random “yes” or “no” responses?

We’re going to open this up for discussion and see if the community can collectively come up with a solution to this problem. For now, here are a few ideas to consider (none of which are ideal):

Solution #1: Daily limit – Like some other quests, we could place a limit on the number of grades you can give each day. We’ve used this technique for some quests, but for grading reviews and tips there are plenty of reasons why this would not be a good idea. We want to encourage people to read reviews, and the natural time to grade the review is immediately after reading it.

Solution #2: Less Valuable Rewards – At first, this might look like a good solution, but it could also backfire. Lower rewards could actually encourage exploitation (because it would tend to encourage quantity over quality).

Solution #3: Limited Rewards – Instead of continuing to give rewards for grading, we could cut off the rewards after you’ve graded X number of reviews or tips (such as cutting the quest series off at the “Amateur Grader” level). So far, this is the best idea we’ve been able to come up with to prevent the exploit … but it also means that we couldn’t reward the people who really DO put thought into their grading.

Solution #4: Pay to Grade – We could make it cost BG gold to grade a review. But, I don’t want to use that approach, because we’re already planning on implementing a different system for rewarding really good game reviews and tips … okay I’ll say it … paying BG gold to give a review or tip a “heart”.

Please remember to be polite to other users as you discuss ideas.

Update 8/16/2011

Here is a quick summary of some of the ideas from the comments …

  • Adding a Neutral option
  • Diminishing returns
  • Sort reviews by Yes/No (really a separate topic)
  • Including comments/feedback with the grades
  • Limit based on Professor level
  • Limit based on # of likes you’ve received yourself
  • Adding a time-delay before the Yes/No option appears
  • Only reward XP for a review after a threshold % or # of likes is received
  • Add quick impressions to the Dicussion page instead [Done!]
  • Issues with recent reviews getting bumped and never seen

Comments (74)

Gamer Avatar
8
Professional Reviewer
BoardGaming.com Beta 1.0 Tester
Silver Supporter
Sentinels of the Multiverse fan

I’m going to chime in on the front page limitation for grading as well. I’d say go ahead and get the daily limit in place and unlock the front page to allow the whole site to receive grades.

You can tweak it later with basing grading daily limits on levels, # of likes, and diminishing returns per level later. You can also tweak the filtering later too. I’ll go ahead say it: If the XP for grading gets tweaked, I don’t mind getting reset on levels.

It perturbs me, because I finished reviewing all the games I own on this site right before the lock down happened. So all my work doesn’t get graded anymore.

Be nice if I could go back an edit too, but I’d prefer the lock down be removed first.

Gamer Avatar
2
My First Heart

I think limiting the number of votes/day is a good option. I also agree with D_S about the front-page-only voting – it doesn’t make sense. I’d rather vote ON the game page rather than happen to see it on the front page. If I liked the review enough on the front page, I’ll seek it out on the game’s page to vote for it. I also agree that unhelpful reviews should sink below the top rated comments (happens with Amazon, too). But you have to strike a balance so that new reviews don’t get pushed down because they have 0% votes versus the “best review” which has, say, 99%. I think that’s the overall struggle – balance.

Gamer Avatar
3
Advanced Grader
BoardGaming.com Beta 1.0 Tester
Amateur Reviewer

I really agree with D_S about the front page restriction — especially as more and more people use the site and read the reviews not in the 15 minutes after they are posted but as they are reading and trying to learn more about the game. In fact, it seems that *these* types of users are, perhaps, better judges of what is and is not helpful!

Gamer Avatar
3
Critic - Level 2
Amateur Reviewer

I think the daily limit solution would work well. It’ll also encourage people to visit the site more often to do their “dailies.”

Several other sites have done this to great success, like Newgrounds, for example.

Gamer Avatar
4
Viscount / Viscountess
Advocate
BoardGaming.com Beta 1.0 Tester

Thanks for the bigger text boxes.

This is a great site and you guys are doing great work …

*but* I put a fair amount of work into composing, editing and posting a review this morning. It may not have been brilliant or anything but it did get a few +s in the fifteen or so minutes it was on the home page.

It was pushed off the front page by this one, the worthiness of which I leave for you, gentle reader, to evaluate for yourself:

***************
Great Fantasy game. Itโ€™s mix of strategy, empire building, conquest, quests, heroes, biddings. There are a lot of options to collect 6 runes and win this game. Sometimes Player with biggest army is not winner!
***************

I would *really* like people to be able to grade reviews after they’re off the front page. This front-page-only restriction is getting on my nerves.

Gamer Avatar
8
Professional Reviewer
BoardGaming.com Beta 1.0 Tester
Silver Supporter
Sentinels of the Multiverse fan

I got thinking again… uh oh.

If you institute a daily limit, which I’m okay with you doing, I would base the limit on the level of the Critic or Reviewer Level not the Professor! Why?

If you base it on Professor level, then you simple get more grades as you make more grades. Seems too easy to game still and not help improve content on the website.

However as a person is moving up in the Critic level (or Reviewer), people have judged you as a good Critic, and they think you make good reviews. Therefore, that should qualify you as someone who knows what a good review is based on your experience and everyone else’s judgement as well. So your daily limit for grading should increase to grade other people’s review. Almost democratic I’d say, and nice feedback loop for the reviewer and the boardgamming.com community.

Personally, if you tie the limit to the Reviewer level, it would make it more directly related to your reviews, but I don’t know if that’s possible or not since it is part of the Critic skill set.

Gamer Avatar
4
Viscount / Viscountess
Advocate
BoardGaming.com Beta 1.0 Tester

1. About the rate: My reviews/minute rate is going to be pretty high, particularly if I happen to be flipping through a few in a row where each is one or two sentences long and/or doesn’t have a rating. They aren’t helpful and I know that in seconds.

2. About a neutral option: If I feel neutral about one then I just don’t grade. Easy enough, right? But if some people insist on grinding, a daily limit, even a daily limit on the high side, should be enough.

3. I would *much* rather have a daily limit than have the current setup of only getting to vote on the front page reviews. This is, comparatively, much more annoying than that would be. I’d have more opportunity to give feedback, and more opportunity to get feedback on what I’ve already written. You could set the daily limit high. I’m also guessing there would be more meaningful information for *you guys* if you took the lid off the current front-page-only restriction.

Looking good, again, I love what you guys are doing here, but you need to be adding games faster than fast. Perhaps gather a pool of volunteers to help with that (no, I’m not stepping forward to be one of them).

Gamer Avatar
4
Novice Reviewer
Gamer - Level 4
Advanced Grader

Only reward XP for a review after a threshold % or # of likes is received

A possible issue with the threshold idea is that negative reviews will likely get less “likes” than positive ones. That doesn’t mean they are any less valuable.

Gamer Avatar
6
Platinum Supporter
Thunderstone Fan
I play blue

So, I took a little vacation to de-stress. Thanks for the animated gif, Eric. I like to be referred to as “the developer” by the way … it keeps things less personal. ๐Ÿ™‚

I edited the post to summarize a few of the points that have been made so far.

I would like to say that I am extremely impressed with the level of discussion so far. While I was reading through the comments I made two immediate changes:
(1) I added an “Impression” option to the Dicussion page and added a note in the game review form to suggest that people include their impressions in the Dicussion page. In fact, this was perfect for me today, because I played a game for the 2nd time, and wanted to give my impression, but didn’t feel qualified yet to write a review. Great suggestion.
(2) I increased the default size of the textarea in the review form, to provide a subtle hint that a review would tend to be longer than just a couple lines.

There was also some discussion about what makes a good review, so although that’s not really what this post is about, I just added a new post to address that very thing.

I will definitely end up providing filtering options for reviews and tips. I think that this will address a lot of concerns

To other developers it may be obvious why we went with WordPress as a foundation for a site like this … the main reason being it is so well supported and there are a crazy number of add-ins out there. Using add-ins allows us to implement things much faster than we would be able to do otherwise. With that said … currently, the add-in that we’re using for the rating system has some limitations. One of them is that there is no neutral. So, even though I like the idea of providing a neutral option, realistically it’s not going to happen any time soon. Another limitation of the plugin is the inability to change votes (game ratings and comment ratings). Now that we have a website to show the developer of the plugin, maybe I could convince him to add these features to his to do list. We’ll see. For now, we’ll just have to use a different solution.

I like the idea of not giving any rewards to reviews until they’ve received a threshold of likes … but that is more about the critic quest than the grading quest.

I like the idea of including comments or feedback with the grades … but that is oodles more complicated to develop. That doesn’t mean it would never happen … just won’t happen in this phase of development.

The daily limit is something that would be possible to implement because the votes are logged with dates. But, I just thing that would be too annoying to too many people.

I just have to comment on Andy’s idea for placing a delay on when the Yes/No buttons appear. Though I don’t think it will work for reviews (because they are such widely variable lengths) … I’d like to put out a teaser that some of the Explorer quests are probably going to involve something along those lines.

@marson … thanks for helping us come up with additional quest ideas. ๐Ÿ™‚ I’ve added these to our idea list. I like that you were thinking outside the box. Your SPEED idea made me think of some of the anti-spam tactics that search engines use. To prevent bots from scraping the search results, they check to see if you’re going through the search result pages faster than a person would normally go if they were reading or scanning the results.

So, I’m going to call it a night and dream about checking the rate at which people are clicking on the Yes/No. Meanwhile, would some of you please do me a favor … time yourselves as you read reviews and grade them and report back an approximate rate as: # reviews / minute.

I’m going to try to think of some witty things to say in the little message window when people break the speed limit. ๐Ÿ™‚

Gamer Avatar
7
Knight-errant
Old Bones
The Silver Heart

Something that I was looking to do was to read reviews of games I am interested in, and determining how helpful the review is on how much better informed I feel it makes me. Will the review sell me on my next favorite game, or prevent me from buying something I will regret? Then the review was helpful. Tying the ability to tag a review as helpful to something like a requirement to have the game on either an owned list or a wish list. Maybe throw in an “interested” category. That might be a good way to do it.

Gamer Avatar
3
Gamer - Level 3

At first, I was treating the reviews like the comment box on some other site. Not really noticing I had WAY more room to write. I regret a bunch of my entries now. Maybe some day you could add a feature to edit our own reviews/grades?

Gamer Avatar
9
My First Heart
My First Wish!
My First Favorite!
Gave My First Grade

Limiting the number per day seems definitely the way to go. It’s the closest you’re going to get to encouraging only reviewing something you’ve actually read.

Plus without the ability to grade things that aren’t on the front page almost no one is going to reach those 10 positive grades on their efforts because their stuff is on the front page so briefly unless they write a massively disproportionate amount of things to be graded – and then you’re likely to end up with a glut of not really paying attention to it writing instead of grading. That seems like a much worse outcome.

You could also reward those who have written a number of things that have garnered enough positive votes with more votes per day for themselves. This encourages good writing and more thoughtful grading since the people with the most votes have a better idea of the work that goes into a quality post.

Gamer Avatar
4
Gamer - Level 4
Junior
Novice Reviewer
Knight

I think you’re on to something marson.

This:
“…letโ€™s just change the quests so they donโ€™t reward quantity exclusively.”

Gamer Avatar
2
Rated 10 Games

This seems like two separate issues. (A) Preserving the value of the voting information and (B) preserving the fun of the vote-related quests and rewards.

As for (A) Vote spamming dilutes the votes, but should affect good and bad reviews equally. So this might be simply solved by only reporting the fraction of yes votes (30% positive, 80% positive, etc.) instead of the discrete vote totals (-1, +5, etc).

As for (B) let’s just change the quests so they don’t reward quantity exclusively.
The Professor grading quest could be modified to instead reward:
* Speed – Be one of the first X people to grade a new review or tip
or
* Accuracy – Vote with the majority (this would obviously require scores on new reviews to be hidden for a short time)

And the Critic quest could be changed to:
* Receive at least X% positive votes on X reviews (that have received at least X votes)

Gamer Avatar
4
BoardGaming.com Beta 1.0 Tester
Critic - Level 2
Gamer - Level 3

I think people who really care don’t need to be incentivized to do something so reducing the value of the awards doesn’t seem so bad.
However, I do like the idea of giving a heart to the reviewer that they can gain gold from. Keep in mind that if you do this, you should receive less gold than they give (e.g. -20G = +10G) so that you won’t have people abusing THAT.

Gamer Avatar
10
Critic - Level 5
Professional Advisor
Expert Reviewer
Marquis / Marchioness

Building on what ‘William McCarroll’ points out; would it work to have two columns on a game page for reviews/tips/etc, one that gives the highest rated reviews, and one that gives the “newest” reviews?

Or a box you can check to sort by newest, or sort by rating?

Gamer Avatar
10
Critic - Level 5
Professional Advisor
Expert Reviewer
Marquis / Marchioness

I wonder if ultimately, “short game impressions” as ‘ccmonter’ references could find a home under the Discussion tab. There is currently a spot of “opinion”, but I don’t know if, at the moment, people would find their way there to give these impressions.

In my mind, the quick impressions tend to correlate with the rating you give a game, but I could see not everyone using it that way. I would prefer if there were a specific place (hopefully separate from reviews) to put these quick impressions. It’d be even better if this could be incorporated without extra work by the site admins…

Gamer Avatar
3
Critic - Level 2

@Gabriel – I think there needs to be a balance between high rated reviews, and new reviews. Once the site takes off, if a new reviewer posts a review that sits on the front page for 10 minutes before it gets replaced by newer reviews, and then ends up at the bottom of the list for that game, it’s going to be hard for that review to get the exposure it needs to gain a positive response. So, perhaps an algorithm that calculates the position of a review based on the age of post (for the first week or so), along with it’s positive rating. That should float new posts towards the top for a while. Each day the “newness points” can decrease, moving it down the page, and if the review is a poor one, people will give negative feedback, which will will work against the “newness points” that the post has, and start pushing it down the ranks early.

Gamer Avatar
9
USA
Platinum Supporter
Petroglyph
BoardGaming.com Beta 1.0 Tester

@Jim I like those ideas a lot.
BTW Here’s a link to an animation for your developer:

http://www.millsworks.net/blog/wp-content/uploads/2009/04/writing_process.gif

This is how I feel about once a day. ๐Ÿ™‚

Gamer Avatar
6
Pick a Favorite LGS
Robots on the Line fan
Miniature Painter
I play blue

Does anyone here like the rotten tomatoes site, and the way they have critic reviews and user reviews? One thought we had, is that once we’ve got a good amount of well liked critics on our site, we could promote the top critics to a new status, then for each game you could see general user reviews, and how that stands up against the “top critics” reviews. We’ll pretend my developer didn’t just hear me spill the beans of yet more work to do!

Add a Comment

You must be to add a comment.

× Visit Your Profile