One of our prime directives for BoardGaming.com is to provide quality content and encourage quality content to be submitted by others.
In general, allowing users to “like” or give a “thumbs up” to another person’s review is a great way to encourage quality. On BoardGaming.com we call this action “grading” because that term fits nicely into the theme of the Professor skill class. You grade a review or tip by responding to the question “Was this review (or tip) helpful?”
We are concerned, though, that providing incentive for grading can lead to people not really thinking about the grade they are giving. We want to avoid having a person just click through all the reviews and give a random “yes” or “no” without thinking. As it is, the “grading” quest tends to encourage quantity only.
We definitely want to encourage people to grade reviews and tips, but how can we reward people for doing it thoughtfully, while preventing people from abusing the system by giving random “yes” or “no” responses?
We’re going to open this up for discussion and see if the community can collectively come up with a solution to this problem. For now, here are a few ideas to consider (none of which are ideal):
Solution #1: Daily limit – Like some other quests, we could place a limit on the number of grades you can give each day. We’ve used this technique for some quests, but for grading reviews and tips there are plenty of reasons why this would not be a good idea. We want to encourage people to read reviews, and the natural time to grade the review is immediately after reading it.
Solution #2: Less Valuable Rewards – At first, this might look like a good solution, but it could also backfire. Lower rewards could actually encourage exploitation (because it would tend to encourage quantity over quality).
Solution #3: Limited Rewards – Instead of continuing to give rewards for grading, we could cut off the rewards after you’ve graded X number of reviews or tips (such as cutting the quest series off at the “Amateur Grader” level). So far, this is the best idea we’ve been able to come up with to prevent the exploit … but it also means that we couldn’t reward the people who really DO put thought into their grading.
Solution #4: Pay to Grade – We could make it cost BG gold to grade a review. But, I don’t want to use that approach, because we’re already planning on implementing a different system for rewarding really good game reviews and tips … okay I’ll say it … paying BG gold to give a review or tip a “heart”.
Please remember to be polite to other users as you discuss ideas.
Update 8/16/2011
Here is a quick summary of some of the ideas from the comments …
- Adding a Neutral option
- Diminishing returns
- Sort reviews by Yes/No (really a separate topic)
- Including comments/feedback with the grades
- Limit based on Professor level
- Limit based on # of likes you’ve received yourself
- Adding a time-delay before the Yes/No option appears
- Only reward XP for a review after a threshold % or # of likes is received
- Add quick impressions to the Dicussion page instead [Done!]
- Issues with recent reviews getting bumped and never seen
Love seeing how this stuff shook out.
I think a degree of usefulness would be better than a simple yes/no. Make it a slider bar, that HAS to be moved before a RATE button can be clicked. Three or five shades of usefulness.
I guess that deleting an entry should also delete the Yes/No vote points (life is hard) but it should not lover the earned grader level. It should be left untouched even if the number of points falls below the grade threshold. Just the amount of points needed to reach the next level should grow as the result.
By the way in my opinion the grader quests suit better the explorer theme than the professor one.
Ran into a snag with allowing somebody to delete one of their reviews or game tips. When a comment is deleted, all of the votes for that comment are deleted as well (that’s the way the system is already set up). It does makes sense to cancel the positive and negative ratings for the user who wrote the review/tip, but …
What about the people who graded the review/tip? Should their total # of grades go down as well? I would think not. The graders shouldn’t be penalized.
However, this could create a loophole in the Grader quest that could be exploited by people working together. It would go like this … a person writes a review, then waits for the teammates to review it, then deletes it, then they repeat this process over and over.
Yes, I’m sure that some system could be put in place to monitor this behavior, but perhaps a simple incentive would help prevent this … and I’m thinking that having to pay X BG Gold to delete a review/tip would be that incentive.
If I don’t implement the ability to EDIT a review before enabling people to DELETE a review, then people would be able to delete a review for the purpose of editing. I’m thinking this would result in a lot of repeated reviews and tips showing up in the “most recent” lists that Graders would have already seen before (because they would have been deleted and resubmitted for the purpose of fixing typos). This would be annoying to Graders who were grading out of the interest of reading unique reviews, and exploited by Graders who were just XP grinding. However, having to pay X Gold would help act as a deterrent.
So, if you’re noticing that I’ve added the ability to delete News and Discussion posts, but not the Reviews and Game Tips, it’s because I’m still working on the ability to Edit reviews and tips.
What do you think? Is paying X Gold a sufficient deterrent to prevent the abuse of the “Delete Review/Tip” feature?
How long does it take to read a tip or a review? Let’s say – two minutes (average). So I propose introducing a time limit: a user can grade an entry no earlier than two minutes after their latest given grade.
@Jon – filter looks great. I like how you added gamer type into it as well. I think that feature will serve the site well.
Okay everybody … I finally got a chance to work some more on the filtering and sorting of reviews, based on the grade (e.g. “Most Helpful”). The only place I’ve implemented this so far is on the Gamer Reviews page. Go there and try out the drop-down boxes.
For those who what to submit feedback and bugs and other help, check out the note on the feedback page under “How Can I Help?”
Just in case you might think that it’s a simple next step to add the filtering/sorting to the game pages … it’s not. I’m starting to notice a significant speed issue on pages like Dominion that have a lot of reviews. So, I need to rewrite the comment loop and fix the issues that are causing the slowness. Also, the filters and sorting are going to use AJAX, to avoid having the entire page reload.
Eric makes a nice point…neutral score. Some reviews are simply weak, as he states. They are not outright awful.
I really like that reviews/tips are given a % now instead of a value. I also like that it is not displayed until after you rate it. I try to be as independent as I can, but I’ve noticed myself getting some sway from other votes.
What I don’t like is that even if I’ve already voted, I have to ‘revote’ (won’t be counted) to see the %. This is combined with the issue of no visual feedback when you’ve already rated something, which I assume will both be fixed together.
Also, is there anywhere anymore to post any feedback? I’ve started a list of things I don’t think have been mentioned yet. Unfortunately, it’s also at work and I won’t get to it again until Monday…
I don’t like any of the ideas that are based on the number of likes I’ve received. That means I have to write a lot of reviews just to be able to give my opinion on another review, I don’t see how those two things are tied together. There is no real good way to force people to give an honest answer. I think the best idea is what you’ve already done in not showing the current counts on the yes and no votes, this makes people think about it rather than just doing what the majority does. As far as those that will exploit the system, I don’t see why it really matters, this is supposed to be fun, so leave it fun and those that want to cheat can cheat.
Based on your feedback, the Yes/No grade results are now hidden until after you grade it, and the results now say “5 out of 10 said Yes (50%)” … no quest associated with guessing well, or being the first to rate, etc. but those kinds of hidden quests are possibilities.
I haven’t rolled this out across the entire site, but check out the Game Tips pages (not the home page or user profiles). You should see that after you grade a few tips and return to the page, your rating is displayed along with the results, such as “Did you like this tip? (You said Yes) 5 out of 7 said Yes (71%)”
I hope to have this rolled out to the rest of the site completely by the end of next week.
I like the idea of not displaying the Yes/No until after a person has voted. I also like the idea of displaying a percentage. These things are going to happen some way or the other, after I figure out how to do it.
Also on the agenda … we’ll end up displaying the % of positive votes in the main comment area so that you can see that “rating” of a review before clicking on the “view/hide” button.
After grading some more reviews – this is how I would envision a scale:
Is the review helpful?
Definitely
Mostly
Somewhat
A Little
Not at All
As for tips – I could go either way (keep it the same yes/no or use the scale above).
my idea is this:
-make the thumbs up/ down results of a review visible only after you voted
-give a limited daily/weekly amount of votes
-if your vote agrees with the majority get +1 vote in your limited vote-stack(so its actually a refund of the vote you spend)
anyway something need to be done. my review (a huge wall of text) got upvoted just seconds after i posted it
Here’s what we did for now …
We eliminated the XP gained for each time you grade (a lot of people probably didn’t even realize that was going on, anyway).
We also applied a diminishing return on the grading quest achievements for the higher levels. Now, if all you did was grade and nothing else, you’d only be able to get to the “Graduate” level in the Professor class, which kind of makes sense. You’re not going to become a Professor just by grading … you have to do some teaching and research as well, right?
I’ll probably still implement a daily limit or rate limit, but that’s all the time I had for today. 🙂
@Jon – I have no problem with a 5 face scale, so long as the scale is clearly defined. Which leads to a question: How do you define the scale?
1 – poor review: practically no worthwhile information about the game
2 – ?
3 – ?
4 – good review: concise information. Covers most details about the game.
5 – stellar review: gives plenty of information about gameplay, why it is fun, at least a con, what audience would like it.
Would XP for reviews or tips be credited on a average rating then like 3 out of 5 faces? With at least 5 grades?
Scale idea looks tough to implement, indeed:)
Would it be better just to use the scale for reviews and yes/no for tips?
I’m glad you noticed my comment. I forgot about grading tips. I personally like the feedback loop idea of people’s votes (or faith in your abilities) giving you more ability to grade other people’s reviews or tips. Maybe the ability to grade reviews should be separate from the ability to grade tips. Reviewers and Adviser levels affect the limits of their respective grading categories (reviews and tips). I think that would be great!
Of course you could make the whole thing a secret formula (gamer level + critic level + sq rt of followers / likes…blah blah), where the amount of grades you get daily is awarded to you ceremoniously each time it increases. Maybe the grade limit becomes a status symbol;)
@markmistr … Turns out that a numeric/star/face rating for reviews and tips is indeed possible, though it would require a significant amount of work to switch over to that system. But, I do like the idea, and I’m not going to dismiss it just because it would be hard. I’d like to hear what others have to say about using a “smiley” rating for reviews (probably a 5-face scale).
The restriction of grading to the home page was only ever meant to be temporary … but I went ahead and lifted the restriction … just realize that a reset is likely going to be necessary at some point, especially if we decided to change the rating scale.
Let me make a note about the big picture here. Right now, our user base is largely made up of experienced gamers. However, our target audience is much broader than that. At Gen Con, when we spoke to people about the site, we frequently referred to ourselves as a “gateway” … meaning that the site itself, while not 100% focused on “gateway games,” was designed to act as a gateway for people who like games but have no idea of the immensity of the board gaming world. Of course, we are trying to make the site appeal to experienced gamers as well … because they [you] are the ones who are most likely to provide the quality reviews and tips. But, if we fail to attract the family, social, and casual gamers to the site, we will have failed in our mission.
What does this have to do with Grading? It is this … when you are grading a review, are you voting on whether or not the review was helpful to YOU PERSONALLY? If so, then nobody’s opinion is any better than anyone else’s. It might make sense to based the daily limit on the overall gamer level, or possibly even the Explorer class level (because the Explorer class is going to be about discovery, and part of discovery is reading about games).
It wasn’t until Jay’s comment that I realized there could be a different thought process: When you are grading a review, are you voting on whether or not you think the review will be helpful to OTHERS? If so, then by Jay’s reasoning, it would make sense to base a daily limit on the critic, reviewer, advisor, and inventor experience.
I’m not sure this issue would even have been raised if we hadn’t called the process “grading”.
Removing the ability to grade is NOT permanent, we promise! We just ask your patience as grading reviews and tips will be key in helping the quality rise to the top, and with as many users as we have currently, it needs to be done right 🙂
I see that my idea probably won’t be implementable because you are limited by what the WordPress Plug-in can do. Or is there another plug-in out there that can do a grading scale?
I say do away with Yes/No which I don’t think really works here. I think a grading scale for 1 to 5 would work so much better. 5= Excellent review (Top of the line), 4= Good review (Not quite good enough for a 5, but a quality review, 3= Average review (of middling value), 2= Mediocre review (has a few points that might be of value, but overall not up to par), 1= Poor review (doesn’t really provide reader with anything substantial or is really poorly written.