A QUICK LOOK AT RMN RATINGS

RMN Game Ratings: a Speculative Glance

  • edchuy
  • 05/04/2010 05:27 PM
  • 14701 views
There were a couple of relatively recent articles that dealt with writing reviews and address within them the issue of ratings:

http://rpgmaker.net/articles/246/
http://rpgmaker.net/articles/249/

EDIT: Also, I found this MaxMcGee article that dealt with games rated 4.5 stars but that ended up having an interesting discussion about the issue:

http://rpgmaker.net/articles/252/

Yesterday (5/3/2010), I came up with the idea of doing some statistics and perhaps some charts regarding game ratings. There were several questions I wanted some guidance on, among them: Is the 3 stars standard for an average rated game one that is actually reflected on the reviews? Do games that have a certain status, for example "Completed" ones, receive much better ratings on average than those in other, for games in "Production"?

Keeping in mind, that the game rating shown for a game can be for as little as 1 review or an average of many of them (an issue not addressed here) and that some games can have only "unrated" reviews (so, that's why this is article is not about reviewed games per se, although you need to write a review to be able to rate it), I proceeded to the games page, to sort the games by minimum rating, status and proceeded to count. Then, I compiled the statistics from which I produced the graphs shown here, with some analysis following each. Before starting, I'll mention there were 858 total games in the RMN database, 289 of them rated (0.5 stars or above).

Drum roll please!



This graph shows the distribution of game ratings percentage-wise for all rated games independent of status as well as sorted by status.

- The rating distribution for Complete games mimics that of all rated games, which is not surprising, since this is their most common status (more on that later).
- For Cancelled games, the sample size was smaller than that of any status, which explains why some of the rating values don't show for it.
- What was said for Cancelled could also be said of those on Hiatus, although it should be noted that very few lower rating values appear for the latter.
- For games in Production, you'll notice that there are slightly less 4 and 4.5 ratings than for all the other categories.

Some of this observations made here are corroborated or furthered by the following graph.



This graph shows the average rating for all rated games independent of status as well as sorted by status.

- The average for all rated games is very close to rating of 3 that has been proposed, which might be a surprise or not. I personally didn't expect this to be the case, given that different reviewers have differing standards. It appears that while there might be some reviewers that tend to rate their games high and other low, it all evens out in the larger picture.
- The average for Completed games was, as expected, about the same as for all rated games.
- For the Cancelled games the average was a little more than 2% higher than for all rated games which is probably statistically insignificant.
- Curiously, for games on Hiatus, the average was a little more than 10% higher than for all rated games. I took a quick look at the rated games that have this status and suspect that the developers who got their demos rated and that chose to put the game in this status, are not your average Joes. In the next point you'll find the answer as to why I think this is the case.
- The average rating of games in Production is just a little under 4% of the average of all rated games. I would have guessed that the difference would be much larger, given that demos, in addition to be incomplete, often can be not as polished as complete games. Not to imply there aren't some demos that are much better than some complete games, since I'm talking on average here.

The next two graphs will be not about the ratings themselves, but about how often the RMN members rate games as a community.



In this graph we see the distribution by status percentage-wise of all games and all rated games.

- Completed games are just under 2/5 of all games, but just under 2/3 of all rated games.
- Cancelled games are close to 10% of all games and all rated games. I would expect they are rated in a similar proportion to all games (more on this later).
- Games on Hiatus are about 8% of all games and 4% of all rated games.
- Games in Production are about 45% of all games and 22% of all rated games.

This final graph completes the quick glance I wanted to give:



This graph shows the percentage of the games by status that are rated:

- A little over 1/3 of all games regardless of status are rated.
- A little under 60% Completed games are rated, just a little under twice the overall proportion. I wonder what WIP will think of this particular statistic. Personally, I would like to help the % in this category improve.
- Just over 30% of Cancelled games are rated. I wonder if their status somehow influences players in not reviewing, therefore not rating, the game. Since these games can be thought (sadly) as completed games, this is another category that I would like to help improve.
- Games in Hiatus and Production are rated in roughly the same proportion: 1/6. This is about a half of the overall proportion.



Please let me know about the contents of this article, especially if you think there's some other way of interpreting these graphs. Any other type of comments are welcome as well.


Posts

Pages: first 12 next last
edchuy
You the practice of self-promotion
1624
comment=29143
This is a lot of effort and it is informative, yet all I am taking away from this is "my games are better than average!".

Let's see, you have 2 completed which show 3 stars each, which is the definition of RMN average rating. Now if you were to calculate an average for your rated games: Frog, The Collector, 3.25 (2 reviews); Sore Losers, 3.41 (6 reviews). How RMN rounds to those to 3 is beyond me and precisely one of the points I mentioned in my article didn't address. Just to let you know, as a reviewer you have made 45 reviews so far, given them an average 3.32 rating. That doesn't mean you're necessarily biased (after all rating is a subjective matter). Perhaps you have been lucky to choose overall better than average games to rate or a slightly nicer than average reviewer without knowing it (almost 58% of your reviews were 3.5 or higher).

Then again, I did use the words "Quick" and "Speculative".
Hahaha, man, you're such an engineer :P.
edchuy
You the practice of self-promotion
1624
comment=29158
Hahaha, man, you're such an engineer :P.


Guilty as charged!
This is rather in-depth analysis! I will have to give it a thorough read tomorrow when I have 3 hours to spare ;D
edchuy
You the practice of self-promotion
1624
comment=29160
This is rather in-depth analysis! I will have to give it a thorough read tomorrow when I have 3 hours to spare ;D


A good half an hour should be sufficient unless math and statistics aren't scare the bejesus out of you.
This was all way over my head and I read it a few times.
edchuy
You the practice of self-promotion
1624
comment=29161
A good half an hour should be sufficient unless math and statistics aren't scare the bejesus out of you.

comment=29186
This was all way over my head and I read it a few times.

Case in point. Don't worry SS, I don't expect everybody to understand what I am showing here. Thanks for trying, though!
This very interesting!

I find it very sad that only about 60% of completed games have reviews. As one of the 40%(Not that I've had my game up very long) I know how disheartening it can be to not get any feedback, and I can't imagine what it would be like not to get any feedback on a game - ever. After Games Gale I'm going to start fighting this startling statistic!

Also you are a hero for arranging this.
edchuy
You the practice of self-promotion
1624
comment=29193
This very interesting!

I find it very sad that only about 60% of completed games have reviews. As one of the 40%(Not that I've had my game up very long) I know how disheartening it can be to not get any feedback, and I can't imagine what it would be like not to get any feedback on a game - ever. After Games Gale I'm going to start fighting this startling statistic!

Also you are a hero for arranging this.

Thanks a lot. There's a little but regarding that, which is when developers have posted that status, even if they haven't necessarily even submitted the game to RMN.

I see your completed game has only been posted for a week and sadly reports 10 downloads. That's another thing that my analysis didn't go into. Also, being of the puzzle adventure genre, probably doesn't help your case either (in my case, getting stuck in a puzzle really frustrates me!) in terms of getting feedback, regardless of whether it's just a comment or detailed as in a rated review. Looking more in depth into those types of issues would definitely be a follow-up, but will require me to dedicate more time and be really inspired to come up with a good compilation of data and corresponding analysis.

Also, I wanted to comment, your Avatar seems to look like it came main image of your game with a bit of a Miro twist.
Now that its morning, I understand.

About 60% of completed games having reviews, some of the games were completed in 2002, then this site came around and people reuploaded them, so there was a flood of games. Too much for people to review.

Plus some completed games look like they suck or they do suck, preventing anyone from having the desire to review them.
edchuy
You the practice of self-promotion
1624
comment=29196
That doesn't mean you're necessarily biased (after all rating is a subjective matter). Perhaps you have been lucky to choose overall better than average games to rate or a slightly nicer than average reviewer without knowing it (almost 58% of your reviews were 3.5 or higher).
Getting less than 40% on a degree course means you fail so getting less than 4/10 from me is difficult since that sort of... scoring system has been drilled into me. As a result, I consider 6.5 (ish) to be average rather than 5 so all my rating actually are biased as they are skewed upwards. I've been meaning to re-rate my reviews because of this, I just haven't gotten around to it yet!

Frog, The Collector, 3.25 (2 reviews); Sore Losers, 3.41 (6 reviews). How RMN rounds to those to 3


Tell me about it...

I long got used to the fact that everything is rated down, but the Sore Losers thing still bugs me everytime!


My intent here was to deal with the ratings RMN shows in the gamepages rather than the actual individual ratings of the reviews posted in RMN. Having people reconsider how they rated games was by no means the reason I did this, but if you were planning to reevaluate, that's your prerogative.

I suspect that given that RMN seems to round down the ratings averages, the result was lower averages in graph 2. I wouldn't surprised if the actual averages were from .1 to .2 stars higher if we considered individual reviews. Perhaps, I can sit down and do this particular bit later.
edchuy
You the practice of self-promotion
1624
comment=29225
Now that its morning, I understand.

About 60% of completed games having reviews, some of the games were completed in 2002, then this site came around and people reuploaded them, so there was a flood of games. Too much for people to review.

Plus some completed games look like they suck or they do suck, preventing anyone from having the desire to review them.

Sorry if I kept you up at night, busting your brain!

Yeah, you're probably right about both things, especially the latter one. I have finished playing some of the completed games as well as demos here, but didn't get around to doing reviews until RS!X, which is another way some of that occurs. I was probably not alone in that regard. However, now I have the chance to make up for that. Which is what's important for me.
No you didn't keep me up.
I, too, am surprised that the average game rating is just below 3! I get the impression that some believed that the average ratings were too high, when in fact it's probably where it should be. For games on Hiatus or Cancelled, it's almost as if the promise of the game inflates the scores a bit. Like, this is what it could have been. That being said, 60% of completed games being rated is higher than I was expecting - I was pleasantly surprised. Of course, it could and should be better, but it's still more than I thought.

This is just a personal anecdote, but I found that my Cancelled games get less downloads due to their status, which probably affects how likely they are to be reviewed.

EDIT: and RMN simply truncates decimals. The rating system is on a 10 point scale, so if you have a game average rating of 3.41, the database stores (or calculates) the average as a 6, which translates to 3 on the 5 star scale (since 3.41 x 2 = 6.82 => 6 after the decimals are dropped => 3/5 stars).

Until May 1st, Generica had an average review score of 3.9 stars, which was truncated down to a 3.5/5.
That's what happens when you use an integer to store a floating point value :D
edchuy
You the practice of self-promotion
1624
comment=29230
I, too, am surprised that the average game rating is just below 3! I get the impression that some believed that the average ratings were too high, when in fact it's probably where it should be. For games on Hiatus or Cancelled, it's almost as if the promise of the game inflates the scores a bit. Like, this is what it could have been. That being said, 60% of completed games being rated is higher than I was expecting - I was pleasantly surprised. Of course, it could and should be better, but it's still more than I thought.

This is just a personal anecdote, but I found that my Cancelled games get less downloads due to their status, which probably affects how likely they are to be reviewed.

EDIT: and RMN simply truncates decimals. The rating system is on a 10 point scale, so if you have a game average rating of 3.41, the database stores (or calculates) the average as a 6, which translates to 3 on the 5 star scale (since 3.41 x 2 = 6.82 => 6 after the decimals are dropped => 3/5 stars).

Until May 1st, Generica had an average review score of 3.9 stars, which was truncated down to a 3.5/5.

As I said, that average is based on what RMN shows. I think I'll go ahead and try to start calculating later today the average based on individual reviews (among other things) and do an addendum to what I wrote here, with some new graphs to get a clearer picture.

Regarding games that have been rated and are in hiatus, I will look into this further to see if I can figure out something else. Cancelled games would probably not be affected by the promise since they're DOA projects, but you're probably right about the download issue. I would probably guess that the fact that both of this status are shown in Red in the "Games" list, probably doesn't help them with getting downloads either. Although, I would guess some contrarians would still go ahead and take a look at these games, if not download them, out of morbid curiosity.

EDIT: Currently, Generica has 4 stars rating. Thanks to the latest review, it actually averages a 4.0 rating on the nose.
DE
*click to edit*
1313
Getting rid of review scores and replacing them with scores based on a voting system would be a good idea. Right now if someone wants to fuck up another developer, they can whip up a quick review and give the game one star and there's not that high of a chance that someone else will come over and submit a 5-star review to balance the score somewhat. And even then due to one spiteful shithead the game has 3 stars only. How is that fair? With popular vote a good game will have a good score, and a bad game will have a bad score, because your average player doesn't care about RMN drama, internal cliques, and circle jerking. He cares about the games he plays. Oh, and that would mean that probably more than 90% of the games would have an actual score. Yay!
Pages: first 12 next last