Full Games List
Featured Games
New Games
Reviews
Media
Images
Blogs
Game Updates
Images Feedback Summary
Game Design Highlights
Tutorials
Articles
Scripts
Utilities / Plugins
Resources
Engines
Engine Bulletins
What is Makerscore?
Past Events
Achievements
Misaos
Forums
Post History
Chat
New to RMN?
Full Store List
RMN Music Pack (free)
THE PROBLEM WITH REVIEWS (AND A NEW SUGGESTION!)
Posts
kentona
View playlists
CloseThere has been some discussion on reviews and reviewers of late...
...and I happened to read a related article on online reviews. I thought I'd share the key points of that article with you and maybe suggest some new ideas that could remedy some of the issues.
So here goes...
Inherent Bias
This one is a subtle issue, but makes sense once it is pointed out: People who write reviews have already made the decision to download and play the game, which usually means that they are predisposed to like the game (since it grabbed their interest), which means that the review and its score can be biased in its favor. This results in a higher rating.
Backlash
The higher ratings draw others to the game, people who have previously taken a pass on the game, and if they play it and hate it, their spite can lead to an overcorrections. This results in low ratings.
Merely Satisfactory
Compounding the issue of review bias is that people rarely review things that they find merely satisfactory. The ones most motivated to review a game are those who either really loved it or really hated it - a kind of evangelize/trash dichotomy. This results in an artificial judgment gap, wherein the "meh" voice is lost and only the extreme views are written, posted and heard.
These aren't specific to our site - they are common in any self-selected online rating system. But, what makes our situation more striking is the poorer ratio of reviews-to-products (ie- games). A perusal of Amazon will show that most products have reviews/ratings, with many of them having multiple reviews. Here, though, a lot of games have no reviews, or one review. Only a handful have more than one, and very few have more than 5. This just makes the inherent biases even more striking.
A few more points...
Super Reviewers
While we love that a few of us have taken the time to write dozens of reviews, just try to keep in mind that what is being reflected here is an increase in enthusiasm, and not necessarily wisdom. Take any reviews from our super reviewers with a grain of salt, and don't get all up in arms if one of them gives you a poor review.
Deliberate Manipulation
Since we give any member the freedom to submit reviews, our site is prone to direct manipulation of scores. Members can get friends to give their games glowing reviews, or others can even maliciously tank other game's ratings. Or people can even make fake accounts to write reviews. We do have standards and a submission queue, but we aren't going to catch everything. Keep that in mind.
Download Counts Act as a Recommendation
Download counts act as a recommendation to those viewing gamepages, and are about the only universally applied "rating" this site has. (all games with a download have a download count). When a potential player sees a gamepage and sees the download count, he will know that others have tried this game. If a lot of others have downloaded it, it will increase his likelyhood of downloading it himself. This aspect is also self-reinforcing (more downloads means more people will see the higher download count means that they download it and the count goes up, repeat). So, like it or not, Download Counts actually matter.*
So, what we have here is a bit of murkiness surrounding games, and ways that we can recommend games, or show that games are appealing an worth checking out. Because right now, in terms of collective ways to recommend games, we only have 1) Reviews and 2) Download Counts. So here is my list of recommendations and changes:
1) More reviews
This one is pretty obvious, but it needs to be said. More reviews mitigate the inherent bias, especially the "merely satisfactory" reviews.
2) More Reviewers
Another "duh" suggestion, but also needs to be said. More reviewers will help reduce the misperception of super reviewers as de facto experts.
3) More ways or types of Feedback
We could use more ways to recommend games, or give our impression of games than just the two ways we have now. (There is the voluntary "You Might Also Like..." movement I started, but that is creator-instigated. I am talking about from the player's perspective). I have some ideas that I will outline in greater detail below.
My idea:
Impression Score
Yeah, that's my big idea. Reviews are time consuming to write, especially for RPGs (which are long games, usually). This has resulted in a dearth of ratings for a lot of games. My suggestion is to implement a system wherein it is possible to give a quick impression of a game without having to invest the time to make a standards-ready and quality review.
So, each page will have an Impression tab, wherein you can see other people's impressions of the game and submit your own.
An Impression is a watered down review with a score, a short blurb and an indication of how much you've experienced the game.
I envision the form to be something like:
How far have you experienced this game?
() Viewed Gameprofile
() Played Briefly
() Played Demo
() Played Full Game
() Played Multiple Times
Write some feedback:
*big textblock here for you to write a blurb*
Impression Score
() 0.5
() 1.0
() 1.5
() 2.0
() 2.5
() 3.0
() 3.5
() 4.0
() 4.5
() 5.0
The Impression Score for a game will be shown below the Review Score, but will be a different color, and smaller. The score would be weighted average of all the Impression Scores, where Impression Scores for people that have played the full game would be worth more than people who have just viewed the profile/screenshots.
Why this is useful
-It gives another way for players to give feedback/recommendations, in an official capacity (which carries more weight)
-An Impression can be done quickly, by anyone
-With the ease of use, all games can get an Impression Score
-More feedback for developers
-People judge games by their screenshots all the time, but capturing that information could prove useful to both the developer, the site, and potential players.
My other ideas:
Give Members a Favorites List
Allow people to Favorite games on RMN, and then show that list on the member's profile for others to view.
EDIT:
Show Profile Views Count and Subscriber Count on public Gameprofile
This will act like Download Counts as a self-reinforcing way to recommend games. More information is always better.
Thumbs Up
Give a way to quickly give a game a thumbs up if you think it looks good. But becomes obsolete if Impressions are implemented and Subscriber counts are shown.
This Game Is Like...
Akin to the You Might Also Like... movement, This Game Is Like... would highlight the commercial game this amateur RPG/game is most similar to. So, This Game Is Like...Suikoden would allow us to group similar games and perhaps easier searching for something you might like to play. This idea is only half-baked, but I'd thought I'd throw it out there.
Download Counts
*My thought here is that it should matter less, but not by hiding Download Counts but by providing more ways to recommend or rate games.
That's all I got. Thoughts?
...and I happened to read a related article on online reviews. I thought I'd share the key points of that article with you and maybe suggest some new ideas that could remedy some of the issues.
So here goes...
Inherent Bias
This one is a subtle issue, but makes sense once it is pointed out: People who write reviews have already made the decision to download and play the game, which usually means that they are predisposed to like the game (since it grabbed their interest), which means that the review and its score can be biased in its favor. This results in a higher rating.
Backlash
The higher ratings draw others to the game, people who have previously taken a pass on the game, and if they play it and hate it, their spite can lead to an overcorrections. This results in low ratings.
Merely Satisfactory
Compounding the issue of review bias is that people rarely review things that they find merely satisfactory. The ones most motivated to review a game are those who either really loved it or really hated it - a kind of evangelize/trash dichotomy. This results in an artificial judgment gap, wherein the "meh" voice is lost and only the extreme views are written, posted and heard.
These aren't specific to our site - they are common in any self-selected online rating system. But, what makes our situation more striking is the poorer ratio of reviews-to-products (ie- games). A perusal of Amazon will show that most products have reviews/ratings, with many of them having multiple reviews. Here, though, a lot of games have no reviews, or one review. Only a handful have more than one, and very few have more than 5. This just makes the inherent biases even more striking.
A few more points...
Super Reviewers
While we love that a few of us have taken the time to write dozens of reviews, just try to keep in mind that what is being reflected here is an increase in enthusiasm, and not necessarily wisdom. Take any reviews from our super reviewers with a grain of salt, and don't get all up in arms if one of them gives you a poor review.
Deliberate Manipulation
Since we give any member the freedom to submit reviews, our site is prone to direct manipulation of scores. Members can get friends to give their games glowing reviews, or others can even maliciously tank other game's ratings. Or people can even make fake accounts to write reviews. We do have standards and a submission queue, but we aren't going to catch everything. Keep that in mind.
Download Counts Act as a Recommendation
Download counts act as a recommendation to those viewing gamepages, and are about the only universally applied "rating" this site has. (all games with a download have a download count). When a potential player sees a gamepage and sees the download count, he will know that others have tried this game. If a lot of others have downloaded it, it will increase his likelyhood of downloading it himself. This aspect is also self-reinforcing (more downloads means more people will see the higher download count means that they download it and the count goes up, repeat). So, like it or not, Download Counts actually matter.*
So, what we have here is a bit of murkiness surrounding games, and ways that we can recommend games, or show that games are appealing an worth checking out. Because right now, in terms of collective ways to recommend games, we only have 1) Reviews and 2) Download Counts. So here is my list of recommendations and changes:
1) More reviews
This one is pretty obvious, but it needs to be said. More reviews mitigate the inherent bias, especially the "merely satisfactory" reviews.
2) More Reviewers
Another "duh" suggestion, but also needs to be said. More reviewers will help reduce the misperception of super reviewers as de facto experts.
3) More ways or types of Feedback
We could use more ways to recommend games, or give our impression of games than just the two ways we have now. (There is the voluntary "You Might Also Like..." movement I started, but that is creator-instigated. I am talking about from the player's perspective). I have some ideas that I will outline in greater detail below.
My idea:
Impression Score
Yeah, that's my big idea. Reviews are time consuming to write, especially for RPGs (which are long games, usually). This has resulted in a dearth of ratings for a lot of games. My suggestion is to implement a system wherein it is possible to give a quick impression of a game without having to invest the time to make a standards-ready and quality review.
So, each page will have an Impression tab, wherein you can see other people's impressions of the game and submit your own.
An Impression is a watered down review with a score, a short blurb and an indication of how much you've experienced the game.
I envision the form to be something like:
How far have you experienced this game?
() Viewed Gameprofile
() Played Briefly
() Played Demo
() Played Full Game
() Played Multiple Times
Write some feedback:
*big textblock here for you to write a blurb*
Impression Score
() 0.5
() 1.0
() 1.5
() 2.0
() 2.5
() 3.0
() 3.5
() 4.0
() 4.5
() 5.0
The Impression Score for a game will be shown below the Review Score, but will be a different color, and smaller. The score would be weighted average of all the Impression Scores, where Impression Scores for people that have played the full game would be worth more than people who have just viewed the profile/screenshots.
Why this is useful
-It gives another way for players to give feedback/recommendations, in an official capacity (which carries more weight)
-An Impression can be done quickly, by anyone
-With the ease of use, all games can get an Impression Score
-More feedback for developers
-People judge games by their screenshots all the time, but capturing that information could prove useful to both the developer, the site, and potential players.
My other ideas:
Give Members a Favorites List
Allow people to Favorite games on RMN, and then show that list on the member's profile for others to view.
EDIT:
Show Profile Views Count and Subscriber Count on public Gameprofile
This will act like Download Counts as a self-reinforcing way to recommend games. More information is always better.
Thumbs Up
Give a way to quickly give a game a thumbs up if you think it looks good. But becomes obsolete if Impressions are implemented and Subscriber counts are shown.
This Game Is Like...
Akin to the You Might Also Like... movement, This Game Is Like... would highlight the commercial game this amateur RPG/game is most similar to. So, This Game Is Like...Suikoden would allow us to group similar games and perhaps easier searching for something you might like to play. This idea is only half-baked, but I'd thought I'd throw it out there.
Download Counts
*My thought here is that it should matter less, but not by hiding Download Counts but by providing more ways to recommend or rate games.
That's all I got. Thoughts?
MKID232
435
I like these ideas, specifically the impressions score. I also agree that Thumbs up seems obsolete if the impressions score is in effect.
Xanedil
490
Unfortunately I had suffered from inherent bias when I wrote my WDRPG review. I feel that while trying to point out flaws that I was purposely playing a bad game to simply beat a dead horse, but that was back when I was an upstart overambitious 15 year old.
Now that I'm an upstart overambitious 16 year old, I can reflect on my mistakes and I feel that my latest review is an improvement over my previous ones.
I feel that the Featured Games does a well enough job promoting promising games (even though it's only one at a time), however things like impressions would be an interesting idea.
Now that I'm an upstart overambitious 16 year old, I can reflect on my mistakes and I feel that my latest review is an improvement over my previous ones.
I feel that the Featured Games does a well enough job promoting promising games (even though it's only one at a time), however things like impressions would be an interesting idea.
ZPE
1562
Why not doing away with reviews all together and get impressions to replace it. Writing game reviews is a right old bother on RMN due to grievers, reviewers of reviews and a lack of a simple interface for the reviewer.
Xanedil
490
post=156015Because then Solitarye would be out of a job.
Why not doing away with reviews all together and get impressions to replace it. Writing game reviews is a right old bother on RMN due to grievers, reviewers of reviews and a lack of a simple interface for the reviewer.
I actually like being able to review because it gives the person a chance to analyze a game in ways he normally wouldn't if he were casually playing it. I find being able to analyze games is almost, if not just as important as being able to make your own because from you analysis you are able to draw things about what might hurt or help your own game. Of course reviews are not perfect, as you said there are grievers, reviewer reviewers and such, but I'd rather have that then some kind of lobotomized way of telling if a game is good or not.
Not that I'm knocking your suggestion Kentona...
I don't really like this impression score idea. It seems superfluous, because it's what reviews are already supposed to do anyway. It's approached in the same way, gives the same general effect (feedback and a score) and will probably cause the exact same grievances reviews already do.
Review reviewers are fine as long as "your score is too low" isn't the basis of their entire argument.
Review reviewers are fine as long as "your score is too low" isn't the basis of their entire argument.
I had a thought previously that is something akin to what kentona is suggesting. Basically, a faster way to leave feedback in a numerical format. Here's the problem I see:
I have a lot of thoughts and feelings on games that I can better represent in numbers rather than text. It takes a lot of effort to make a good review. The standards for a good review are high enough here that I would have to put a lot of effort to convey my thoughts that "the graphics are good but not great". I just summed it up right there, but there's no way it would pass through the submission queue.
However, I can give a numerical 1-5 score for it. It would be 3.5. There you go. Took very little time and I was still able to convey that the graphics are good but not great.
Now you (the royal you) might mention that it's possible for me to put said numerical scores in a review and that's true. But those numbers are not saved in any kind of format for me, the site developer, to use. The only thing I can use right now is the final review score and that can mean a lot of things, as we know.
Honestly, I don't feel like the system of "big box of text + out of 5 rating" works that great for conveying quality. I am definitely for alternative ideas.
I have a lot of thoughts and feelings on games that I can better represent in numbers rather than text. It takes a lot of effort to make a good review. The standards for a good review are high enough here that I would have to put a lot of effort to convey my thoughts that "the graphics are good but not great". I just summed it up right there, but there's no way it would pass through the submission queue.
However, I can give a numerical 1-5 score for it. It would be 3.5. There you go. Took very little time and I was still able to convey that the graphics are good but not great.
Now you (the royal you) might mention that it's possible for me to put said numerical scores in a review and that's true. But those numbers are not saved in any kind of format for me, the site developer, to use. The only thing I can use right now is the final review score and that can mean a lot of things, as we know.
Honestly, I don't feel like the system of "big box of text + out of 5 rating" works that great for conveying quality. I am definitely for alternative ideas.
A while ago I pitched a Critique idea, where you get a series of ratings for various common categories like Graphics, Sound, Story, Character, Fun, etc... and a text block for misc notes. Then you can submit that to the gameprofile/creator.
But we got hung up on the name "Critique" and it didn't go anywhere.
But we got hung up on the name "Critique" and it didn't go anywhere.
ariedonus
41
post=156131
I don't really like this impression score idea. It seems superfluous, because it's what reviews are already supposed to do anyway. It's approached in the same way, gives the same general effect (feedback and a score) and will probably cause the exact same grievances reviews already do.
Review reviewers are fine as long as "your score is too low" isn't the basis of their entire argument.
It's what reviews are supposed to do, not always what they do, which is Kentona's point. The main problem is that not every game gets a review, so whatever they're supposed to do can't even have a chance at happening. Even when a game does get the reviews, chances are it isn't getting enough of them and the score is more likely to be skewed. Because of the format of this "Impression" system, more are likely to come in, therefore giving developers more feedback and players more viewpoints/opinions to decide whether or not to play.
post=156133
Honestly, I don't feel like the system of "big box of text + out of 5 rating" works that great for conveying quality. I am definitely for alternative ideas.
I agree that giving a a numerical value is a lot easier for one to do that to write up a review about everything specific. GameFAQs has a system in place, as most gaming sites do, that takes all of these factors into consideration. It has a score for user reviews, a score for user ratings, and then a third score that shows what the pro reviewers think. With all three ratings, it gives players three distinct points of view on the game and helps a lot when deciding whether they should buy it or not.
To try and apply it to what we have here, we could have a score for user-ratings (just a quick opinion that registered users can leave in numerical form), quick review scores (Kentona's "Impressions"), and then full review scores (our tradition review system). This gives everyone three scores to look at that each hold a different weight, but are all important. Other sites have the option to leave quick reviews that are very much like Kentona's new system, because more feedback is always good. Yeah, this doesn't always ensure the utmost quality, but reviews don't always ensure that either (I have seen reviews that would be much better served in "Impression" form).
post=156138
A while ago I pitched a Critique idea, where you get a series of ratings for various common categories like Graphics, Sound, Story, Character, Fun, etc... and a text block for misc notes. Then you can submit that to the gameprofile/creator.
But we got hung up on the name "Critique" and it didn't go anywhere.
Leave it to RMN to get sidetracked on semantics, haha.
post=156131
I don't really like this impression score idea. It seems superfluous, because it's what reviews are already supposed to do anyway. It's approached in the same way, gives the same general effect (feedback and a score) and will probably cause the exact same grievances reviews already do.
Review reviewers are fine as long as "your score is too low" isn't the basis of their entire argument.
Odd. I figured you more than anyone would be in favor of it. =P
Those aren't empty arguments when people complain about you scoring low - you're legitimately fucking them over because you are able to drop their overall average down (or up) single-handedly. That isn't fair to anyone hosting their game here, and is a disincentive for people to use the system altogether since it's basically rigged, albeit completely unintentionally on all sides. The objective of this impression score is to get more people into the system so individual reviews don't carry so much weight like that.
If 1,000 people play a game and 800 like it and 200 dislike it, the average rating should be 80%. With 1,000 people in the total pool, one negative or positive review isn't going to massively swing it one direction or the other, therefore it's a more accurate average. Due to the overhead required to rate a game currently, most people don't. Since there is only 5 people in the sample pool, each review heavily swings the entire average of a game.
To me the only issue is the rating being tied to the review. Looking at YouTube, how many people would rate a video if you had to write an essay first? That's the actual problem. I think Kentona's proposal is pretty good but I'd even reduce it down further to a single click as I don't think the questions are really all that relevant.
Honestly... I don't like any of those ideas, and I wouldn't change a thing about the current system.
So, yeah, people have bias and stuff, but hey, we're humans, that's something we can expect.
Saying "there's a problem with reviews, let's write more reviews" obviously doesn't make any sense.
Quick numerical ratings would be far more open to bias, and results could be tragic.
I don't like dichotomical "like-dislike" ratings, because a game with over 50% dislike would automatically be viewed as "do not play".
Also, we have comments on gameprofiles. People can leave quicker feedback there. I don't see a point in stuffing too many ratings and pieces of info in the gameprofile. Confusing. It's fine the way it is, really.
So, yeah, people have bias and stuff, but hey, we're humans, that's something we can expect.
Saying "there's a problem with reviews, let's write more reviews" obviously doesn't make any sense.
Quick numerical ratings would be far more open to bias, and results could be tragic.
I don't like dichotomical "like-dislike" ratings, because a game with over 50% dislike would automatically be viewed as "do not play".
Also, we have comments on gameprofiles. People can leave quicker feedback there. I don't see a point in stuffing too many ratings and pieces of info in the gameprofile. Confusing. It's fine the way it is, really.
Well, the problem with few reviews is that they carry too much weight, so writing more reviews actually does make sense (since then any given review carries less weight). We have strong Sampling bias here with our review system, which is common with self-selection systems.
A larger sample size helps mitigate these issues.
A larger sample size helps mitigate these issues.
I'm not against more reviews, actually, I'm just saying those things you mentioned as "problems" aren't really "problems". They are part of the whole process. We can't expect any means to OBJECTIVELY rate games and OBJECTIVELY sample the games we play or review.
So, yeah, let's play more games, write more reviews and try to be constructive about them. That's the only real improvement we can expect.
So, yeah, let's play more games, write more reviews and try to be constructive about them. That's the only real improvement we can expect.
Well, this was partially directed to those that overreact to reviews too, since there ARE inherent biases that can't be avoided and to keep that in mind before flying off the handle.
I also just found the article and the points it makes interesting, and thought I'd share it.
I also just found the article and the points it makes interesting, and thought I'd share it.
ariedonus
41
post=156166
Honestly... I don't like any of those ideas, and I wouldn't change a thing about the current system.
So, yeah, people have bias and stuff, but hey, we're humans, that's something we can expect.
Saying "there's a problem with reviews, let's write more reviews" obviously doesn't make any sense.
Quick numerical ratings would be far more open to bias, and results could be tragic.
I don't like dichotomical "like-dislike" ratings, because a game with over 50% dislike would automatically be viewed as "do not play".
Also, we have comments on gameprofiles. People can leave quicker feedback there. I don't see a point in stuffing too many ratings and pieces of info in the gameprofile. Confusing. It's fine the way it is, really.
People do have bias, but if there are simple ways to lower the amount of bias occurring, why not? I find it hard to look at a problem, have all the resources to solve that problem, and leave it alone. It doesn't make much sense to me.
The problem isn't with the reviews, we still want reviews. Reviews are very helpful and should always be a part of the site. Kentona just explained, however, that we don't have enough of them, which is where the problem lies.
With the quick, numerical ratings, yeah some people are going to be ridiculous and vote in an extreme fashion, but reviewers do the exact same thing. The difference is that a lot more users are participating and the bias will be weighed out, because those who just feel like the game was "ok" will be more likely to input their feedback. I'm not exactly a fan of the like/dislike system, though, so we see eye-to-eye there.
You do have a good point about the commenting on profile pages, but the problem is that the feedback will likely go unnoticed by the players. We use comments on the profiles for many different things, such as asking questions, making silly jokes, etc. and the feedback is going to be harder to find. When players are looking over a game, chances are that they are going to look to the reviews, the stars, and whatever ratings are there.
catmitts
4862
I guess I like the impressions idea since it might make people more likely to say what they thought of a game without feeling the need for it to be exhaustive gameplay/music/chipset analysis. I don't care about the ratings and I get annoyed by like/dislike things with no justification involved. I would honestly either get no feedback at all or get a stack of onestar reviews than have to decipher some worthless and arbitrary THUMB UP / THUMB DOWN measurement with no comments attached, and I kind of feel this way about subscriber/download count too. It honestly seems pretty obnoxious to me that someone would just give a score without deigning to explain why! If you don't want to talk about it then don't talk about it.
Basically idk about getting more reviews or whatever but numerical measurements are always stupid and arbitrary at the best of times and I don't know why you'd want to play up their stupidity and arbitrariness while minimising what little level of accountability they ever had. Not saying that's what you're doing I am just a post *plucks guitar*.
Basically idk about getting more reviews or whatever but numerical measurements are always stupid and arbitrary at the best of times and I don't know why you'd want to play up their stupidity and arbitrariness while minimising what little level of accountability they ever had. Not saying that's what you're doing I am just a post *plucks guitar*.
