I think it is fair to say that we WAY underestimated the speed at which people would end up writing reviews and tips. It’s also fair to say that we way underestimated people’s response to this site as a whole. We originally thought that moving up the ranks would at least take a gamer a few months, and a lot of the rewards and achievements were based on that growth rate.
Incentives are a very tricky beasts to control. We’re not psychologists or economists, but perhaps we need some on the team. We’re not going to get rid of incentives, because that’s what we think makes this site fun, but we do need to tweak them, and that’s why we’re in beta.
It should be pretty obvious that we’d like to encourage high quality reviews rather than quantity. In theory, if we end up with tens of thousands of users, we could end up having tens of thousands of reviews for each game (1 per person).
Tens of thousands of reviews for a single game is not better than say 50 really good ones. In fact, it could be annoying to have that many reviews. But, hundreds of great reviews for a single game might be useful, if we can end up subdividing those reviews based on gamer types and other factors that people find useful.
Oh, and before everyone launches into a dicussion about wanting reviews to be sorted by rating … I’m working on that … so let’s not talk about that here.
Instead, what are some ways to use quests and incentives to:
- encourage quality reviews
- help weed out the less helpful reviews
- ensure that high quality critics get the rewards they deserve
Regarding point #2 … we want to be very careful that we do that in a way that is actually positive rather than offensive or embarrassing or too judgmental.
One thought we had was that when you are logged in and looking at your own profile, perhaps we could highlight the really good reviews and the least successful reviews and give the option of either deleting or editing the reviews. This might be based on the % of “Yes” grades after some minimum total votes. We’d need to implement a system where the ratings were erased if you deleted or edited a review, because past ratings would no longer be accurate if you edited a review.
I suspect that automatically deleting reviews that were poorly rated could make people mad, so that might not be a good option. Maybe they could be hidden or just not listed anywhere but in a person’t profile?
Another thing we could do is create a minimum length for a review, but maybe we should wait and analyze the statistics before implementing that type of restriction. Who knows, maybe there are some extremely good tips and reviews that are just single sentences.
Anyway … that should be enough to start the discussion.
As has been said may times, allowing revisions is a must, even if it has not met a certain threshold (of goodness) or fallen below another threshold. I think allowing users to delete reviews might also help clean things up, but then would they never be able to review that game again? Or not get points for doing bad reviews and deleting to redo them. (possible loophole)Just some things to consider with deleting reviews.
I agree with a lot of the above comments; there’s some great suggestions.
Editing!
Thumbs up/down rating system!
Filtering by gamer type!
I’d say that you should run your reviews like Youtube runs it’s comment system. Users upvote or downvote reviews. New reviews go on top for 18 hours or so, and then they are placed within the stack based upon what their score was for those 18 hours. I’d probably have a line of demarcation just so users can see where the new reviews end and where the top ranked reviews begin.
I think it’s okay to let people get upset if the system deletes their reviews. If you want to be a little less blunt with someone doing a bad job, just have it delete from the game’s page but leave it on their personal profile page.
Maybe make the number of reviews and tips that are shown on the front page a higher number, like ten instead of five, and require a that stuff to get an overall positive rating while it’s there for it to get put on a game’s page. And things can still get to the game’s page if they achieve the positive rating from folks rating things on the person’s page.
Then maybe a simple rule that if something gets voted onto a game’s page three times, the third time it’s voted off it can’t get back on.
Sorry for complaining about the work, everybody. Please don’t worry about whether an idea might be hard or not. Some things that might appear to be difficult sometimes turn out to be extremely easy. And some things that ought to be easy sometimes cause us to tear our hair out (mine’s already mostly gone, by the way). And yes, we’d ALL get tired of the green.
These discussions are extremely helpful, by the way.
Back to the topic of incentives … one area where we might be able to distribute some of the load (besides getting help adding games) would be finding somebody (or maybe multiple people) to help analyze the incentive/quest structure and rewards. It takes me a lot of time to think about, analyze, and adjust the rewards for quest achievements – time that I could be spending doing other development … or sleeping. I’m not sure how video game designers manage it, but we just have a spreadsheet. Contact Jim if you have an interest and think you could contribute along these lines, and realize that it would be purely volunteer assistance. I’m imagining something pretty simple … a shared Google Doc, a private forum discussion somewhere, etc.
I don’t think sorting reviews by score is a good idea. If there are a lot of reviews of something (say, multiple pages), that means you will glance at it and only see the good reviews and think “oh, people must love it!” and not see the bad ones later on. I think, if you could, it would be good to try to do it the way Amazon does it: show the highest-rated positive review and the highest-rated negative review on top, then the rest of them in either chronological or random order.
This is a very delicate question because most of the obvious solutions have obvious drawbacks, as you’ve said.
The highest rated reviews will be self-reinforcing, both specifically and by reviewer; requiring reviews to be a certain length will obviously encourage babblying for two paragraphs instead of one.
I’m not sure what the answer is, but I am sure most of what I’ve read so far isn’t it.
How about to limit bad reviews – you just limit reviews period (like start at 1 review per week and gain more as you level up your gamer level).
Another idea is to have word limits (to prohibit reviews that are either too short or too long).
At the least, the sight would not be yellow. There may be some black and white thrown in there.
BTW If Green Lantern were to help you develop the Web site, I’m pretty sure it would be entirely green. Jim might not like that.
@Jon My suggestion of “simple filter” was more a comparison to other high level design suggestions posed on the site… it was not meant to belittle the work of developers in any way. I run a web design firm, and used to be a developer myself (I know how hard it can be). That’s why I was suggesting a filter as an interim solution until you have time for more elaborate functionality. So, I guess the review system you are using doesn’t support sorting or is being created by you.
Adding a public repository for the code of your site and allowing people to submit patches would help accelerate feature implementations. In this way you can approve what you want for the site and individuals can add what they most want. I am not sure how much time I have to implement new features, but if given the opportunity I would certainly try.
Maybe you could have a separate tab from Reviews called “Thoughts” or something and like Twitter you could be encouraged to write short little blurbs. or maybe it could be a feed right on the side of each games profile. IDK. Just a thought. I don’t program so I don’t know how difficult things are to build.
I seriously need to be able to edit my reviews even without people’s input:-) Too many typos, because I’m the typo king, no matter how many times I reread my entries before submitting.
Anyway, I think you have a great idea Jon on the part of highlighting our personal reviews based on performance. I don’t mind an automatic deletion, so long as the user is notified by email that their reviews will be deleted soon if they don’t improve them. You know, something like “You have 30 days to fix this review before deletion or delete now if you desire”.
I agree with Andy about some sort of constructive criticism feedback mechanism to help a reviewer improve his or her review.
You mentioned sorting reviews by gamer type, and I agree that is what I would like. Being able to analyze a game based on the reviewer’s preferences helps a person better understand why this game might be a good or bad investment.
My only concern is that newer reviews will be ignored. Once a set of reviews have been establish say 20 or 30, then most people will ignore the newest reviews and go straight to the top rated and well established reviews. This makes it harder for new people to contribute. I don’t know how to work around that.
On single line or number of line quotas, I’m guessing that reviews should have more lines. I really can’t find much helpful in a single line review. However, I truly believe a good tip can be delivered in a single line. So if line quotas are going to be enforced, I would recommend that they only be done on reviews.
If you wanted to get strict on the quality reviews, then you could always move to a “peer” review system where new reviews are put in a queue for review under a different section of the site (say “reviews pending approval” under “Reviews”). They have to meet a certain criteria of say a number of likes or percentage of likes after a set number of likes before being published to their corresponding game page and the front page. If the review doesn’t meet approval in a certain time period, it is just deleted. The user could be given notices to improve it before deleting to keep in the approval process. People can still continue to vote on the reviews after they are published.
I love it when people say “a simple sort or filter” or “just a simple pull-down filter” … sure … no problem, but no matter how hard I try, I can’t will things into being. Where’s Green Lantern when you need him? I’d like to just borrow his ring for just a few seconds.
Another idea for the reviews … to solve the problem of having older reviews stop being found … how about listing at the top a single review selected at random from the collection of reviews that you have not rated yet … oh, and don’t hold your breath on that. Before that is possible, I have to figure out the whole selecting and sorting and filtering thing.
Please allow editing reviews. A simple sort or filter of good reviews would be helpful. Maybe extending that to sorting games that have many reviews that are marked helpful.
BTW The new incentive leap is pretty extensive. I’m only 10 away… augghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh! 🙂
Is it possible to get some more Quests added (especially for Explorer and Reporter)?
Have you considered just a simple pull-down filter for Most Liked Reviews? It could filter:
Most Liked/Least Liked/Newest/Oldest/Following
This would allow more control for the user, and default could be Newest… so new reviewers aren’t penalized. It would be the least amount of programming time as well.
Also, I can’t wait to edit my first reviews. It’s on my priority list!
I like the idea of having a person’s reviews in their profile be color coded (perhaps a green box for reviews with xx%+, a yellow box for moderate, and red box for lower), to see which reviews are well received, and which aren’t.
To really encourage great reviews, I think the site will need some sort of feedback form that graders can fill out with the reason for their rating. Perhaps you can give more “Professor” points to people who leave comments with why they gave the grade they did. I think people will learn best if someone points out WHY the rating was the way it was.
A person’s profile could link such comments to the review. To encourage these comments to be helpful, you could have the original reviewer select an option of whether the comment was helpful to them. This could be the basis of the extra Professor points for these comments.
Being able to give constructive criticism that helps someone improve their review is even tougher than writing a good review yourself. I think the users of the site could gain a lot by becoming better not only at writing reviews, but at giving this constructive feedback.