I think it is fair to say that we WAY underestimated the speed at which people would end up writing reviews and tips. It’s also fair to say that we way underestimated people’s response to this site as a whole. We originally thought that moving up the ranks would at least take a gamer a few months, and a lot of the rewards and achievements were based on that growth rate.
Incentives are a very tricky beasts to control. We’re not psychologists or economists, but perhaps we need some on the team. We’re not going to get rid of incentives, because that’s what we think makes this site fun, but we do need to tweak them, and that’s why we’re in beta.
It should be pretty obvious that we’d like to encourage high quality reviews rather than quantity. In theory, if we end up with tens of thousands of users, we could end up having tens of thousands of reviews for each game (1 per person).
Tens of thousands of reviews for a single game is not better than say 50 really good ones. In fact, it could be annoying to have that many reviews. But, hundreds of great reviews for a single game might be useful, if we can end up subdividing those reviews based on gamer types and other factors that people find useful.
Oh, and before everyone launches into a dicussion about wanting reviews to be sorted by rating … I’m working on that … so let’s not talk about that here.
Instead, what are some ways to use quests and incentives to:
- encourage quality reviews
- help weed out the less helpful reviews
- ensure that high quality critics get the rewards they deserve
Regarding point #2 … we want to be very careful that we do that in a way that is actually positive rather than offensive or embarrassing or too judgmental.
One thought we had was that when you are logged in and looking at your own profile, perhaps we could highlight the really good reviews and the least successful reviews and give the option of either deleting or editing the reviews. This might be based on the % of “Yes” grades after some minimum total votes. We’d need to implement a system where the ratings were erased if you deleted or edited a review, because past ratings would no longer be accurate if you edited a review.
I suspect that automatically deleting reviews that were poorly rated could make people mad, so that might not be a good option. Maybe they could be hidden or just not listed anywhere but in a person’t profile?
Another thing we could do is create a minimum length for a review, but maybe we should wait and analyze the statistics before implementing that type of restriction. Who knows, maybe there are some extremely good tips and reviews that are just single sentences.
Anyway … that should be enough to start the discussion.
People need to stop worrying about others and just do their own review. Stuff the length and read the content or you are just a snob pushing your own sense of “morality” on others.
Breakdown the reviews percantage of yes`s after 10+ reviews then show the best ones. A review usually stays up on the home page for long enough to know where it is going to rank against other reviews.
The best, fairest way I can think of to accomplish this is to have it only display the top rated 5 reviews (ratio of read vs. rated) and 5 latest reviews of any given game on that game’s page, with an option to look at all of them, sorted in descending order so that people who submitted reviews but that got overlooked still get a shot at making it to the top.
Then, to close the loop on this equation, make it a point that a certain ratio of read to rated reviews grants a reward for people who rate them, like a quest. The more a person reads, the more they’d have to rate in order to keep that ratio up.
Or something like that.
I have studied writing. I have also been reading and studying video game reviews for a long time. Here is the real dirt as I see it (whether reviewing board or video games).
1) User reviews can be a great help and asset when a buyer needs a few more opinions, but are often not enough by themselves. A site solely of user reviews is not a useful site at all.
WHY?
Several reasons.
a) you do not want to spend time reading a review if it is off base or you just plain don’t agree. So, you have to depend on user Yay/Nay votes. However, these are often misleading.
– users often vote down (Nay) a review simply because it speaks poorly of a game they like, even if the review is accurate and well written.
– users may up vote (Yay) a review just because it gives a few sentences they agree with. The review may not say nearly enough to be useful, but they “Yay” it because they agree with the sentiments.
b) user reviews are often too short, too long, or not clear and, sadly, there isn’t any way to guarantee (or even necessarily get a high likelihood) of quality reviews.
c) even if the review is quality, it can be hard to determine if the game is for you. Maybe the reviewer is very harsh on the randomness in the game because they hate ANY randomness. Maybe you don’t mind randomness as much and it wouldn’t bother you so much.
SOLUTION:
* You need some regular, official reviewers that users can get to know. Example: When a GiantBomb.com editor reviews a game, I have read enough of their reviews that I know what they like and dislike. So, if they say they hate the inventory management, I know whether to take that “with a grain of salt” or whether to really take note because they don’t usually have a problem with such things. All those problems I mention above can be avoided via regular, official reviewers. You can be sure they write solid reviews because you have chosen them in response to good reviews they’ve written. The users know how to evaluate their reviews because they learn their likes/dislikes.
WE CAN’T PAY ANYONE. HOW CAN WE DO THIS?
I would suggest starting with scouring the site for users that regularly write good reviews (or even a few good reviews without any poor ones). Email them and ask if they would be willing to be a official, boardgaming.com reviewer. You could put a special stamp on their profile and reviews. You might request that they just review one game every one or two months. That isn’t too much to ask. Heck, I’d love to do that (yes, for free). If needed you could add incentive by giving them earlier access to new features or something.
I agree with @The Schaef, it now seems like the number of “Likes” needed to fulfill the later Professor Advisor, Strategy and House Rule Quests is enormous and way out of whack for the number of actual tips required.
If the idea is to accumulate “likes” for reviews over the long term, this doesn’t appear to be occurring. Very few of my reviews or tips seem to get any attention after a few days or so — without regard to how popular it was initially. This is probably due to the fact that there are now so many reviews and tips for a game that people get tired of reading them all and only check the latest ones on the top of the list.
Allowing people to sort by “most popular” will only reward the earliest popular reviewers who will float to the top of the list and benefit to the exclusion of new reviewers who may have superior things to say.
There may not be good ways to get around this, but making the quests require one to write dozens or hundreds of good tips (which seem to stop getting “Likes” or “Dislikes” after a few days), doesn’t seem to be the way to go.
The SOP for social sites these days seems to be to hide contributions that reach a certain threshhold, while allowing the user to click to show if they still want to read that ****** comment.
And I don’t know if this is the appropriate place for this, but I think it is tied to review quality: I’m not sure the ratio of reviews to thumbs is quite right for leveling up the Reviewer quest. It could just be a function of how many people are presently on the site but I am not sure a larger number would change my opinion much, because if anything it would only push reviews off the front page faster, meaning potentially fewer thumbs per review.
The current ratio starts out at about 13:1 to hit level 1, then to hit level 2 it goes up to 28:1. Level 3 only requires 8 reviews but I have to get up to 380 thumbs to move on. I only have 155 thumbs at this moment in time and I had to write 17 reviews to get them. The current ratio on the quest is almost 50:1 but I can barely pull 10 thumbs on my best-liked reviews. The final level only requires only 21 reviews but I have to average almost 110 thumbs per review to get that prize.
Again, I know you have to account for this being a beta and that there potentially will be a lot more users, but I also think that means the large amount of participation will dilute the thumbs available to people who write quality reviews. My early recommendation is that the thumb ratio be scaled down significantly, but also bump up the number of reviews to keep the higher levels pushed out. 2270 thumbs seems a billion miles away, but 3/5/8/13/21 is a cakewalk for number of reviews.
I would recommend avoiding some sort of size minimum for a review. Some people can share a lot of information very efficiently, whereas others can go on for hours without really saying a darn thing.
I would instead recommend some sort of rating system for reviews – something like Netflix. If it’s useful, let it float to the top. If it’s nothing more than “I like this game”, I’d guess it’ll sink out of sight with enough feedback.
Heck, even just having an example of “Things to Keep In Mind” when reviewing may help some people organize their thoughts before posting.
I also support the ability to update your own review in case your mind changes over time/experience.
I haven’t read all the comments, but I think the biggest thing that would encourage good reviews is to give awards as a review gets yes votes. So writing your first review might be a quest, but in order to get the xp and money you have to get say 3-5 yes votes. Further quests already do this, but if the xp and gold incentive did something similar people would be more likely to write a well thought out review. This would also be a good way to go further incentive to people who are writing good reviews, every 5 yes votes gets them more xp and gold so the review just keeps on giving, assuming people like it.
Good points regarding why you’d want to be able to edit reviews.
I’m finding that I’m spending all of my time reading people’s suggestions, and there is no time left to actually do anything. 🙂
I spent all my free time today setting up an account with FogBugz.com to allow people to volunteer their time and experience to helping improve this site. See the feedback page under “How Can I Help?” for more info on that.
I agree with Julian Leiberan-Titus that the ability to append further impressions of a game to your review without changing the original would be beneficial. Having only one, uneditable review available to you for each game can really make you think twice about reviewing a game you’ve only played a few times. Your opinion and experience may change (and quite probably will) the more you play a game.
On the “oh no, typos!” front, one possible solution (or at least a way to mitigate it somewhat) would be to allow people to preview the text of their review (or tip, or discussion post) before submitting the final version.
Another stat to track on quality of reviews & tips…
How about adding to the leader board (and profile) the users with the highest overall percentage of liked reviews and liked tips? This may give users another metric for “quality” versus “quantity” of a user they may want to follow, and adds another stat to brag about;-)
Hmmm… not sure if I like the idea of “paying” to delete the review/tip. I think that will backfire if your goal is quality content, and no one will get rid of a bad review/tip which in effect could detract from the quality of the content. I don’t mind losing XP or achievements for removing review or tip. That sounds more reasonable.
Is it possible to add the edit capability to fix reviews or tips for typos if they score high like 80%/90% or higher after 10 votes without a reset? I say that, because I would like to fix my typos. If you implement the time limit which is reasonable, hurts my hopes of ever correcting my submissions:-)
Slightly side topic: Is there a way to start your own discussion? I haven’t found it yet. I have some suggestions/issues (other than those below) but not sure where to post those.
Discussion
A couple of things about reviews that would be helpful:
1) When clicking Yes/No it shows you the current Yes/No results. Once you click it and come back, you cannot see what you voted for or the results from said vote so you must ‘vote’ again to see the results. Voting again will not change what you actually voted though. It would be nice if it ‘remembered’ that you voted and then just show the current results, plus highlight what YOU voted somehow. Also, I should be able to change my opinion of something.
2) The “x out of y said Yes” comment is annoying. It would be more useful to see “x Yes / y No” or something simpler and easy to read.
3) You cannot comment on a review and this would be nice to do – maybe the ‘comments’ can be a separate discussion somewhere so it doesn’t interfere with the whole ‘review’ overview and then provide a link if there is discussion.
4) Being able to edit a review for typos and such would be nice. I noticed the suggestion about a time limit. That would work. Another option would be to limit the number/amount of changes you can make per edit (percentage maybe). Granted someone could, over time, completely revamp a review (such as a bad one) but it might not be worth their effort. I don’t like the idea of having to pay to delete a review.
I have not really seen any short reviews that are useful and I can’t think of any way that they would be. Tips might be useful even in a shorter form. So maybe just start with reviews having a certain length requirement. If the site orders the reviews and tips by the number of yes votes to no votes, then deleting reviews and tips should be unnecessary. Also, I have noticed that some of the reporter incentives do not seem to update. I have invited some active users to the site and my invite points have not gone up any more at all. Also, I started following some other gamers and those did not show up. When I subscribed to the newsletter, it did not show that I had. I had to unsubsribe and resubscribe to get it to work.
Oh and I definitely am on board with letting people edit their reviews. One thing to consider is that personal opinion changes as you play a game more and more. If I write a review after playing a game 4 or 5 times, I might want to update that review (and it’s score) after playing a dozen more times.
I undestand the desire to just edit a bad review, but it might be nice if there was a way to provide an update to an existing review so that its tied to the review but doesn’t change the original.
Like if my first review says “This game is fantasic, I love it. You should run out and buy it.” But then later I can go add to the top “UPDATE: After playing the game for awhile more, I noticed faction A and B are really underpowered. This game may be fun for the first severaly play throughs, but it doesn’t have much staying power.”
I think what’s great about this site is how much it encourages participation. Don’t slow down the rewards too much. It’s better to let someone keep going to Level 500 critic with getting steady rewards for each set of reviews than to have people writing a dozen reviews before they hit level 2.
I wouldn’t worry too much about thousands of reviews. Just make organization of those reviews helpful. I might be more interested to see how my friends on the site are reviewing a game than I would be to check out some “expert” reviewer’s opinion even if my friends get low feedback from other people.
It might be nice if you could sort reviews based on info in the reviewer’s profile. For example, if a reviewer has many of the same favorite games as I do, I might trust his or her review more than someone who thinks highly of games I dislike.
Above all, don’t delete or punish reviews simply for getting lot’s of “No” votes. Let the cream rise to the top, but don’t remove a persons chance that someone will find their review and appreciate it.
This could work. Would we have the power to delete the review ourselves?
Also, when you question the “real” money equivalent, is this because there will be tangible items to buy at some point, or are you looking for a philosophic comparison?
So I’ve been looking into the review edit / review delete feature. I think that the best approach would be to place a time limit on the editing … so that the main reason to edit is just to fix typos. Allowing editing without a time limit could mess up the comment grading.
I think the best approach for fixing poorly graded reviews/tips would be to just delete them … and yes … you would be able to add a new one in its place. (Deleting would also be the simplest near-term solution). But, there would be penalties …
(a) You will lose all the grades associated with the review/tip
(b) The XP earned for writing the review/tip would be subtracted (but it would not cancel achievement completions)
(c) It will cost you 100G (or more) … if you didn’t have 100G, you could run a negative balance … but only because we’re kind of legally obligated to allow you to request your content to be removed.
(a) and (b) are really just cancellations of the rewards you got. The only real “penalty” would be the BG gold cost. We’re going to want the cost to be painful, so that it can help act as a deterrent.
Here’s a question … if you were to be charged actual $ to have your poor review/tip deleted, how much would you be willing to pay? $0.10 each? $0.20 each? $1.00 each? That is important, because it is related to how much BG Gold it would cost.
Check out your user profile … today I added the rating results “3 out of 5 gamers thought this was helpful” above the “View/Hide” button. If you’re logged in, you can see all of the results for your own reviews/tips, but when viewing other people’s, it will only show the results if they’ve received 5 or more grades. I also added the face to make it easy to scan through your reviews and see what people liked and didn’t like.
I’m still working on the edit/delete reviews and the filtering/sorting.
If the ultimate goal is good content on the site… allowing a user to edit or delete their own review would make perfect sense. They should get rewarded for trying again. Why not give someone a “clean” slate if they are working hard to get it right? Everyone will have the same opportunity, and the owners of the site get everyone’s best work.
If a review drops down to some abysmal percentage of “no” votes, then the owner should get a warning, and if he/she ignores it, then the review is removed by the system. I would still give them the ability to re-review that game in the future, as maybe they weren’t on the site for some valid reason…. vacation, sabbatical, prison.
I really like this site, and think it has loads of potential. I look forward to its evolution.