Full Games List
Featured Games
New Games
Reviews
Media
Images
Blogs
Game Updates
Images Feedback Summary
Game Design Highlights
Tutorials
Articles
Scripts
Utilities / Plugins
Resources
Engines
Engine Bulletins
What is Makerscore?
Past Events
Achievements
Misaos
Forums
Post History
Chat
New to RMN?
Full Store List
RMN Music Pack (free)
A QUICK LOOK AT RMN RATINGS
RMN Game Ratings: a Speculative Glance
edchuy- 05/04/2010 05:27 PM
- 15188 views
There were a couple of relatively recent articles that dealt with writing reviews and address within them the issue of ratings:
http://rpgmaker.net/articles/246/
http://rpgmaker.net/articles/249/
EDIT: Also, I found this MaxMcGee article that dealt with games rated 4.5 stars but that ended up having an interesting discussion about the issue:
http://rpgmaker.net/articles/252/
Yesterday (5/3/2010), I came up with the idea of doing some statistics and perhaps some charts regarding game ratings. There were several questions I wanted some guidance on, among them: Is the 3 stars standard for an average rated game one that is actually reflected on the reviews? Do games that have a certain status, for example "Completed" ones, receive much better ratings on average than those in other, for games in "Production"?
Keeping in mind, that the game rating shown for a game can be for as little as 1 review or an average of many of them (an issue not addressed here) and that some games can have only "unrated" reviews (so, that's why this is article is not about reviewed games per se, although you need to write a review to be able to rate it), I proceeded to the games page, to sort the games by minimum rating, status and proceeded to count. Then, I compiled the statistics from which I produced the graphs shown here, with some analysis following each. Before starting, I'll mention there were 858 total games in the RMN database, 289 of them rated (0.5 stars or above).
Drum roll please!
This graph shows the distribution of game ratings percentage-wise for all rated games independent of status as well as sorted by status.
- The rating distribution for Complete games mimics that of all rated games, which is not surprising, since this is their most common status (more on that later).
- For Cancelled games, the sample size was smaller than that of any status, which explains why some of the rating values don't show for it.
- What was said for Cancelled could also be said of those on Hiatus, although it should be noted that very few lower rating values appear for the latter.
- For games in Production, you'll notice that there are slightly less 4 and 4.5 ratings than for all the other categories.
Some of this observations made here are corroborated or furthered by the following graph.
This graph shows the average rating for all rated games independent of status as well as sorted by status.
- The average for all rated games is very close to rating of 3 that has been proposed, which might be a surprise or not. I personally didn't expect this to be the case, given that different reviewers have differing standards. It appears that while there might be some reviewers that tend to rate their games high and other low, it all evens out in the larger picture.
- The average for Completed games was, as expected, about the same as for all rated games.
- For the Cancelled games the average was a little more than 2% higher than for all rated games which is probably statistically insignificant.
- Curiously, for games on Hiatus, the average was a little more than 10% higher than for all rated games. I took a quick look at the rated games that have this status and suspect that the developers who got their demos rated and that chose to put the game in this status, are not your average Joes. In the next point you'll find the answer as to why I think this is the case.
- The average rating of games in Production is just a little under 4% of the average of all rated games. I would have guessed that the difference would be much larger, given that demos, in addition to be incomplete, often can be not as polished as complete games. Not to imply there aren't some demos that are much better than some complete games, since I'm talking on average here.
The next two graphs will be not about the ratings themselves, but about how often the RMN members rate games as a community.
In this graph we see the distribution by status percentage-wise of all games and all rated games.
- Completed games are just under 2/5 of all games, but just under 2/3 of all rated games.
- Cancelled games are close to 10% of all games and all rated games. I would expect they are rated in a similar proportion to all games (more on this later).
- Games on Hiatus are about 8% of all games and 4% of all rated games.
- Games in Production are about 45% of all games and 22% of all rated games.
This final graph completes the quick glance I wanted to give:
This graph shows the percentage of the games by status that are rated:
- A little over 1/3 of all games regardless of status are rated.
- A little under 60% Completed games are rated, just a little under twice the overall proportion. I wonder what WIP will think of this particular statistic. Personally, I would like to help the % in this category improve.
- Just over 30% of Cancelled games are rated. I wonder if their status somehow influences players in not reviewing, therefore not rating, the game. Since these games can be thought (sadly) as completed games, this is another category that I would like to help improve.
- Games in Hiatus and Production are rated in roughly the same proportion: 1/6. This is about a half of the overall proportion.
Please let me know about the contents of this article, especially if you think there's some other way of interpreting these graphs. Any other type of comments are welcome as well.
http://rpgmaker.net/articles/246/
http://rpgmaker.net/articles/249/
EDIT: Also, I found this MaxMcGee article that dealt with games rated 4.5 stars but that ended up having an interesting discussion about the issue:
http://rpgmaker.net/articles/252/
Yesterday (5/3/2010), I came up with the idea of doing some statistics and perhaps some charts regarding game ratings. There were several questions I wanted some guidance on, among them: Is the 3 stars standard for an average rated game one that is actually reflected on the reviews? Do games that have a certain status, for example "Completed" ones, receive much better ratings on average than those in other, for games in "Production"?
Keeping in mind, that the game rating shown for a game can be for as little as 1 review or an average of many of them (an issue not addressed here) and that some games can have only "unrated" reviews (so, that's why this is article is not about reviewed games per se, although you need to write a review to be able to rate it), I proceeded to the games page, to sort the games by minimum rating, status and proceeded to count. Then, I compiled the statistics from which I produced the graphs shown here, with some analysis following each. Before starting, I'll mention there were 858 total games in the RMN database, 289 of them rated (0.5 stars or above).
Drum roll please!
This graph shows the distribution of game ratings percentage-wise for all rated games independent of status as well as sorted by status.
- The rating distribution for Complete games mimics that of all rated games, which is not surprising, since this is their most common status (more on that later).
- For Cancelled games, the sample size was smaller than that of any status, which explains why some of the rating values don't show for it.
- What was said for Cancelled could also be said of those on Hiatus, although it should be noted that very few lower rating values appear for the latter.
- For games in Production, you'll notice that there are slightly less 4 and 4.5 ratings than for all the other categories.
Some of this observations made here are corroborated or furthered by the following graph.
This graph shows the average rating for all rated games independent of status as well as sorted by status.
- The average for all rated games is very close to rating of 3 that has been proposed, which might be a surprise or not. I personally didn't expect this to be the case, given that different reviewers have differing standards. It appears that while there might be some reviewers that tend to rate their games high and other low, it all evens out in the larger picture.
- The average for Completed games was, as expected, about the same as for all rated games.
- For the Cancelled games the average was a little more than 2% higher than for all rated games which is probably statistically insignificant.
- Curiously, for games on Hiatus, the average was a little more than 10% higher than for all rated games. I took a quick look at the rated games that have this status and suspect that the developers who got their demos rated and that chose to put the game in this status, are not your average Joes. In the next point you'll find the answer as to why I think this is the case.
- The average rating of games in Production is just a little under 4% of the average of all rated games. I would have guessed that the difference would be much larger, given that demos, in addition to be incomplete, often can be not as polished as complete games. Not to imply there aren't some demos that are much better than some complete games, since I'm talking on average here.
The next two graphs will be not about the ratings themselves, but about how often the RMN members rate games as a community.
In this graph we see the distribution by status percentage-wise of all games and all rated games.
- Completed games are just under 2/5 of all games, but just under 2/3 of all rated games.
- Cancelled games are close to 10% of all games and all rated games. I would expect they are rated in a similar proportion to all games (more on this later).
- Games on Hiatus are about 8% of all games and 4% of all rated games.
- Games in Production are about 45% of all games and 22% of all rated games.
This final graph completes the quick glance I wanted to give:
This graph shows the percentage of the games by status that are rated:
- A little over 1/3 of all games regardless of status are rated.
- A little under 60% Completed games are rated, just a little under twice the overall proportion. I wonder what WIP will think of this particular statistic. Personally, I would like to help the % in this category improve.
- Just over 30% of Cancelled games are rated. I wonder if their status somehow influences players in not reviewing, therefore not rating, the game. Since these games can be thought (sadly) as completed games, this is another category that I would like to help improve.
- Games in Hiatus and Production are rated in roughly the same proportion: 1/6. This is about a half of the overall proportion.
Please let me know about the contents of this article, especially if you think there's some other way of interpreting these graphs. Any other type of comments are welcome as well.
Posts 
View playlists
CloseWe had a system like that for a while, but it was hugely underused. Like, a game would have like 4 Thumbs Up and 2 Thumbs Down out of the hundreds of people that visit the page. Maybe it was a visibility problem, or maybe it was just an apathetic userbase, but either way it didn't work out exactly in that ideal fashion, DE.
I would like to see some more ways of rating games, or anonymous rating, or something. Right now, reviews just aren't enough.
I would like to see some more ways of rating games, or anonymous rating, or something. Right now, reviews just aren't enough.
People would look at the 1 star review vs the look of a game. If the game look > review look, people play it and review it.
People that frequent the site probably would, but if I'm viewing the site for the first time ever then the first thing I do is to launch up the gamebase and sort the games by ratings, then try some of the top-rated games. A game rated 1 star I do not care about. If it has just one star then it's probably shitty, after all it has been reviewed by people that know what they're talking about.
A voting system must be anonymous and preferably a scale system (say 1-5 stars, not just thumbs up and thumbs down - what if I like a game, but I still feel it's only so-so compared to some other games I had downloaded off RMN?). I also think it should be open to everyone, not just the members. It would seem open to abuse, but we would just have to wait and see how it works out.
A voting system must be anonymous and preferably a scale system (say 1-5 stars, not just thumbs up and thumbs down - what if I like a game, but I still feel it's only so-so compared to some other games I had downloaded off RMN?). I also think it should be open to everyone, not just the members. It would seem open to abuse, but we would just have to wait and see how it works out.
comment=29273
Getting rid of review scores and replacing them with scores based on a voting system would be a good idea. Right now if someone wants to fuck up another developer, they can whip up a quick review and give the game one star and there's not that high of a chance that someone else will come over and submit a 5-star review to balance the score somewhat. And even then due to one spiteful shithead the game has 3 stars only. How is that fair? With popular vote a good game will have a good score, and a bad game will have a bad score, because your average player doesn't care about RMN drama, internal cliques, and circle jerking. He cares about the games he plays. Oh, and that would mean that probably more than 90% of the games would have an actual score. Yay!
In principle that would work, but certain things would have to be in place, among them I would say:
- Voting should be voluntary. (we are talking about democratizing ratings here!)
- Only registered users should be able to vote (to reduce spamming votes).
- Nobody should be able to vote for a game more than once (see above).
- Only games with a download should be eligible for voting (otherwise you get what they have in RRR top 10 games, with games that are just on paper).
- No developers should be able to vote for their own games (for obvious reasons).
- New voting should be in place when a new version (bugfixes don't apply and new version has to have new content or significant changes) of a game comes out.
- The people who rate actually have to had played the game (it would be a scout's honor system; at least in a review you have to convice us you at least tried to play the game and somehow justify your rating).
- I would advocate for a system with a rating, with some sort of obligatory feedback (few basic questions: what they liked, disliked, what they would see improved).
- All voting and related feedback of a game should be visible to all registered users. (for transparency's sake).
- Any registered user should be able to see any others registered user's voting record. (see above).
- If anybody's caught cheating somehow they're banned from RMN. (I remember there was some Misao voting issue ...)
Currently there's no easy way of knowing how many of all RMN games actually have some sort of download. I would guess it's in the 65-85% range of all games listed.
And yeah, I agree that, unfortunately rated reviews can be used for obscure purposes.
EDIT: It seems that my concept of the system is quite different than what you have in mind, DE!
Reviews are required to give a rating so that there is some attempt at quality control, preventing "My bro made this game 5/5!." With a review, the person at least has to give some explanation as to why they rated something the way they did (some go into a lot more detail than others), and any obvious rating padding/sabotage is generally caught by the administrative staff who approves submissions. Occasionally something bad slips through, if you ever see a review that is obviously and flagrantly nonsense, feel free to point it out to the staff.
Personally I don't think score or rating is all that important, it doesn't really mean anything, as I feel the content of the review is far more important. Saying a game is "Two stars," without giving any explanation as to why you think that, is almost useless, both to the players and the developer. There is also a near ubiquitous obsession with prettiness on this site and I can guarantee you that letting anyone vote for a game would lead to many, many people voting entirely based on screenshots. Scouts honor doesn't work in a community like this, it hasn't in the past and it is unlikely to in the future. By writing a review, you at least have to convince everyone you actually played the game. This is why I would not really like a system that allowed people to score games without writing a review.
I know enough about statistics to know more or less what you're talking about in this article, edchuy, and I appreciate the effort you have gone through for this. I am not sure off the top of my head how significant any of it is, but it is an interesting read nonetheless. However, I don't like too much emphasis being put on ratings around here. The only person who can really decide if you will like a game is you.
Personally I don't think score or rating is all that important, it doesn't really mean anything, as I feel the content of the review is far more important. Saying a game is "Two stars," without giving any explanation as to why you think that, is almost useless, both to the players and the developer. There is also a near ubiquitous obsession with prettiness on this site and I can guarantee you that letting anyone vote for a game would lead to many, many people voting entirely based on screenshots. Scouts honor doesn't work in a community like this, it hasn't in the past and it is unlikely to in the future. By writing a review, you at least have to convince everyone you actually played the game. This is why I would not really like a system that allowed people to score games without writing a review.
I know enough about statistics to know more or less what you're talking about in this article, edchuy, and I appreciate the effort you have gone through for this. I am not sure off the top of my head how significant any of it is, but it is an interesting read nonetheless. However, I don't like too much emphasis being put on ratings around here. The only person who can really decide if you will like a game is you.
Go to the Youtube ratings...
Like or dislike. Its very easy. I've been to sites where the rating is 1-10... everyone rates it 10!!! It becomes pointless seeing every video file is 10. Instead of 890 people LIKE this or 12 people DISLIKE this.
Heck...
graphics = like or dislike
gameplay = like or dislike
story = like or dislike
grammar = like or dislike
design = like or dislike
Like or dislike. Its very easy. I've been to sites where the rating is 1-10... everyone rates it 10!!! It becomes pointless seeing every video file is 10. Instead of 890 people LIKE this or 12 people DISLIKE this.
Heck...
graphics = like or dislike
gameplay = like or dislike
story = like or dislike
grammar = like or dislike
design = like or dislike
I agree with Solitayre at least on a few points he made:
- They go through administrative approval (although I still think nobody should be able to approve their own review, that it has to be a peer, at least).
- At least the developer has a chance to interact with the reviewer through the review itself and can also point it out to the staff.
- There's no honor among thieves.
- I know the issue in RRR is that a lot of the voting they have is based on screenshots.
- In the end the player him/her is the one that ought to decide how much and what they like (or not) about a game.
- The content of the review should tell more than the overall rating.
I'm sorry that you don't think ratings are important. Unfortunately, developers as well as potential players make certain decisions based on them, regardless of whether we like it or not. Perhaps in an ideal world they shouldn't be relevant, but the fact is they are.
- They go through administrative approval (although I still think nobody should be able to approve their own review, that it has to be a peer, at least).
- At least the developer has a chance to interact with the reviewer through the review itself and can also point it out to the staff.
- There's no honor among thieves.
- I know the issue in RRR is that a lot of the voting they have is based on screenshots.
- In the end the player him/her is the one that ought to decide how much and what they like (or not) about a game.
- The content of the review should tell more than the overall rating.
I'm sorry that you don't think ratings are important. Unfortunately, developers as well as potential players make certain decisions based on them, regardless of whether we like it or not. Perhaps in an ideal world they shouldn't be relevant, but the fact is they are.
Well it's not that I think ratings are useless or anything, I just think too much emphasis is put on them, and for the wrong reasons. If you get a bad review but it is filled with useful information to improve your game, isn't that the most important thing?
I've noted this before, but the more popular a game is, the more reviews it gets, and the more "average" its rating turns out to be. This leads to many exceptional games being rated fairly "averagely." So it's not exactly a useful way to really tell what games are good and which aren't. Though, note that if a game has 10 reviews but still has an average rating above four stars (like Hero's Realm), it's probably something pretty awesome.
I've noted this before, but the more popular a game is, the more reviews it gets, and the more "average" its rating turns out to be. This leads to many exceptional games being rated fairly "averagely." So it's not exactly a useful way to really tell what games are good and which aren't. Though, note that if a game has 10 reviews but still has an average rating above four stars (like Hero's Realm), it's probably something pretty awesome.
I've reviewed 1,000s of things on Amazon... I think I'm up to 1,114, and I think people read interviews to feel the same as you do. So fact is more important than opinion.
Some game sites won't even bother to display a score until a game has more than 2 reviews (or a staff review). Now, I'd love to explore that kind of option here, but the reality is we do not have a chance of having that kind of review volume (we currently sit at roughly 30% of the games having reviews now?).
EDIT:
My point is that as it stands now, a single review and its score carries a significant amount of weight. The obvious solution would be to dilute it with a bevy of reviews. However, that is just hopeful thinking.
EDIT:
My point is that as it stands now, a single review and its score carries a significant amount of weight. The obvious solution would be to dilute it with a bevy of reviews. However, that is just hopeful thinking.
comment=29311
Well it's not that I think ratings are useless or anything, I just think too much emphasis is put on them, and for the wrong reasons. If you get a bad review but it is filled with useful information to improve your game, isn't that the most important thing?
I've noted this before, but the more popular a game is, the more reviews it gets, and the more "average" its rating turns out to be. This leads to many exceptional games being rated fairly "averagely." So its not exactly a useful way to really tell what games are good and which aren't. Though, note that if a game has 10 reviews but still has an average rating above four stars (like Hero's Realm), it's probably something pretty awesome.
OK, I agree, it's only a tool, but the only one we have other than word of mouth. About somebody getting a bad review, it might be helpful to the developer not necessarily a potential player.
What you define as an exceptional game, isn't necessarily isn't necessarily the same as others. Actually, a game having many rated reviews I actually see as a good thing, since it presents a more balanced view of the game. Games that have a high rating but only one rated review, on other extreme, I think as being less reliable, unless I actually have a pretty decent idea of who the reviewer is and how he/she views games and reviews, in general. To give you an example, I trust Roger Ebert for movie reviews, because most of the time his assessment is close to how I would rated the movie. In spite of that, there are times I completely disagree and that's OK, I have come to accept that possibility. It's not as if he were some LA or Hollywood unknown that some really crappy movies seem to quote in their advertisements use to make it appear everybody is given them rave reviews.
I agree that review scores don't really help much at all. I think that the reason reviews are so helpful (and necessary!) is because they provide the developer with valuable feedback on how to improve their game, or how to improve their future projects.
That being said, the reason most people make games is because they want people to play them. Regardless of what we as developers think of the review scores, they're the single biggest factor determine whether Joe Random Lurker will download and play a game or not. We can keep saying how "oh, they shouldn't worry about the review score!" all we want, but it's not going to change the reality that the vast majority of the people that play our games judge which ones to play based on the score. When a review is submitted for a game, whether intentionally or not, it is directly affecting whether or not that game will be played.
That being said, the reason most people make games is because they want people to play them. Regardless of what we as developers think of the review scores, they're the single biggest factor determine whether Joe Random Lurker will download and play a game or not. We can keep saying how "oh, they shouldn't worry about the review score!" all we want, but it's not going to change the reality that the vast majority of the people that play our games judge which ones to play based on the score. When a review is submitted for a game, whether intentionally or not, it is directly affecting whether or not that game will be played.
