SPOT CHECKING THE RATING SYSTEM VOLUME I: FEBRUARY-MARCH 2010

I review (not as in critique, as in, look over) the top 13 games and the available data on them.

Spot Checking the Rating System Volume I: February-March 2010
I review (not as in play and critique, as in, look over) the top 13 games and the available data on them.

Common sense tells us that RMN site users (not necessarily forum users) will most likely use a rating filter to see which are the "best" games available on the site, and play those games. So the numerical ratings that games receive (based on reviews which have no standardized criteria and which everyone can submit) are super important.

I apologize if the following encroaches upon EPIC MATH FAIL:

The way that reviews work here, each game is given a review with a rating from 0.5 to 5 Stars. I am not a statistician (although I have worked with them a little bit) but this seems to me that games with more reviews will have more "average" scores than games with less reviews. In other words, games with only one review are more likely to have high or low scores because there is not a process of aggregation. A game with one review might have 5 Stars or 1 Star because it got a 5 Star or 1 Star review. A game with two reviews which received a 1 Star review and a 5 Star review will have a three star review.

For this article I looked at all of the games on RMN with a rating of 4.5 or higher (putting myself in the shoes of someone just coming to this site to play computer games, with no prior knowledge of the community to influence my decisions, who was using rating to look for the BEST games). For each game I will be looking at several categories to determine why the game is on the top twenty list, if it seems to make sense that it is there, and what review data is available for it. From this data we should be able to get a picture of how (and how well) RMN's "Star" rating system is working and look for any interesting hiccups or inconsistencies in the data set. This evaluation was not written with any particular conclusion in mind and I look forward to seeing what I can learn. So let's dive in.

I will be asking the following questions about each of RMN's top games.

What is the game called?
What engine was this game made in?
Who made this game?
When was this game submitted to RMN? How many downloads does it have?
What do I know about this game going in?
Have I played this game?
How many reviews does this game have?
How was this game reviewed?

In this way I will be trying to correlate several things. Rating and download count. Number of reviews and rating. number of reviews and download count. Etc. and so on. Now, let us BEGIN.

#1
What is the game called? Cast Aside
What engine was this game made in? RMXP
Who made this game? Anamei Productions
When was this game submitted to RMN? How many downloads does it have?
Added to RMN on June 20th, 2009, Cast Aside has 1,380 downloads.
What do I know about this game going in? I have heard of this game but never played it. It was made by Anamei Productions who are known for very very very high quality, polished final products with original art and complex and unique custom systems. I played their SkyE last year and was utterly blown away by it. This game is described as an Episodic Active Strategy game with puzzles and an involved plot. Its tone appears to be somewhat light to me and it seems to have something of an anime style. This is an episodic game.
Have I played this game? No, and hence I am not qualified to discuss its actual quality.
How many reviews does this game have? This game has been reviewed once by Solitayre.
How was this game reviewed? I may repeat this a LOT in future sections, but Solitayre grades on five categories. Balance, level design, characters, story, and music and sound. In this case, Cast Aside received a Balance Score of 3/5, a Level Design Score of 4.5/5, a Characters score of 5/5, a storyline score of 4/5, and a music and sound score of 3/5. Solitayre says: "Any fan of tactical RPGs could do far worse than to give this game a try. Or any fan of puzzle games. Or fans of well-written characters. Hell, the only real complaint I have with this game is that it’s just not long enough! There must be more. I look forward immensely to seeing where this project goes."

#2:
What is the game called? Tina of the Stars
What engine was this game made in? RMXP
Who made this game? Ratty524, who it is worth noting, I have NEVER heard of.
When was this game submitted to RMN? How many downloads does it have?
July 22nd, 2009 and 70, respectively. This is our first anomalous statistic as the downloads have been around for some time. However there are several explanations, such as technical difficulties, for why the download count does not match the rating and time the game has been available. Of course if we rule that out we might attribute it to the game's genre or tone.
What do I know about this game going in? All I know is that the game has bright, cheery graphics and implores me to "Grab some stars and save your fairy friends!". As far as I can tell this is a complete game.
Have I played this game? No, and hence I am not qualified to comment on its quality.
How many reviews does this game have? One, by Fallen-Griever, a well-known reviewer.
How was this game reviewed? This game was reviewed shortly after it came out. Fallen-Griever grades in terms of Gameplay, Characters/Dialogue/Storyline, Graphics/Mapping, and Music/Sound. Each aggregated category is given a score out of 10. Fallen-Griever gave this game an NA in the Charactes/Dialogue/Storyline aggregate as it is an arcade game. Other categories got a 9/10 except for music/sound which got a 10. Fallen-Griever said of the game: "For what this game is trying to be it is really good. It obviously isn't one for those people who are into games with lots of depth, purpose, storyline and characterisation, but as far as simple, arcade action goes this game is solid. Like Pacman? Play this."

#3:
What is the game called? Legends of Illarion
What engine was this game made in? RPG Maker 2003
Who made this game? Dyluck
When was this game submitted to RMN? How many downloads does it have?
5/18/2009 :: 2410
What do I know about this game going in? "Legends Of Illarion is an RM2K3 game based on Illarion, the online roleplaying game. Illarion is a free, graphical fantasy game that focuses on true roleplaying rather than hack & slash." It appears to use some RTP graphics. As far as I can tell this is a complete game.
Have I played this game? Nope. See above.
How many reviews does this game have? One review, from SilverSoul, who has written one other review.
How was this game reviewed? SilverSoul does not use numerical categories. He had this to say: "All in all, I had a lot of fun with this game. I stopped right at the end trying to fight the dragons, but I found the size of that sidequest very satisfying. The sheer number of characters, and having to switch between them, keeps gameplay from becoming monotonous, and the harvesting system gives you something to look forward to. There are also plenty of amusing scenes and supporting characters. I give this game a very high rating for people who like traditional RPGs."

#4:
What is the game called? Arian Wild
What engine was this game made in? RMVX
Who made this game? Craze & Karsuman
When was this game submitted to RMN? How many downloads does it have?
January 2nd, 2010; 280 Downloads
What do I know about this game going in? "Arian Wild is a game made by the creators of Visions & Voices. It is a linear RPG in the vein of traditional role-playing classics, except for the many parts where it isn't." It looks "dark". As far as I can tell this is a battle demo.
Have I played this game? Not yet. I did play Visions & Voices, but still see the disclaimers above!
How many reviews does this game have? One, by Forever an Ill Fate
How was this game reviewed? FaIF's review reviewed only the battles, and seemed more like a beta test report than a true review. He did not assign numerical categories, nor should he have, since he just played a battle demo. He compared the game to an "Etrian Odyssey and WoW love child".

#5
What is the game called? Geondun
What engine was this game made in? RMVX
Who made this game? Craze
When was this game submitted to RMN? How many downloads does it have?
March 2009. 1177 Downloads.
What do I know about this game going in? This game HAVE STATS. "Six races, six classes, three faiths - 432 party combinations." This is a demo.
Have I played this game? Yes, finally, and I wish I had had time to play it more!
How many reviews does this game have? One (noticing a pattern yet?), from Fallen-Griever.
How was this game reviewed? Gameplay 9/10, Characters Classes 10/10, Graphics/Mapping and Music and Sound both 6/10. Storyline was (again) dismissed as NA. I largely agree with Fallen-Griever's review...my scores would have been within one standard deviation different. Griever's final word:
"The aesthetic factors aren’t even slightly important in this game. It could have the worst music and graphics ever and, so long as everything was still functional, I would’ve probably still enjoyed this game. Maybe not quite as much, but I would've enjoyed it. I'm sure Craze is sick of hearing the D-word by now, but this game surpasses Demon Tower by mile; if you are one of the many fans of Demon Tower I really suggest you download this instead as it is vastly superior."

#6
What is the game called? RMN Christmas Card 2009. Awww...
What engine was this game made in? RM2k3
Who made this game? YDS and AznChipmunk, possibly with help from others.
When was this game submitted to RMN? How many downloads does it have?
I'm going to guess Christmas of 2009? 222 Downloads.
What do I know about this game going in? It is a community game of some kind?
Have I played this game? Nope.
How many reviews does this game have? FIVE! Wow, what a way to break the trend!
How was this game reviewed? All of them are very glowing! As a community game being reviewed well by the community and the specific people who made it, this doesn't warrant much more serious analysis.

#7
What is the game called? The Longing Ribbon
What engine was this game made in? RPG Maker 2000. A first on our list.
Who made this game? Gibmaker.
When was this game submitted to RMN? How many downloads does it have?
February 6th, 2008, making it by far the oldest game looked at so far. For that, it has a whopping 2600+ downloads. This is a complete game.
What do I know about this game going in? Horror RPG, winner of the 2005 Misao for Best Atmosphere.
Have I played this game? I played this game in 2005 and was impressed by its atmosphere. The battle system bothered me a lot, however, as I never got very far in it. I filed it off as a forgotten classic like barrel the apocalypse Guile Machination.
How many reviews does this game have? Four.
How was this game reviewed? Well. Honestly, the reviews here are by so many people and are all so glowingly positive that out of all the games I've looked at so far this is the only one whose place in the top twenty seems unassailably earned regardless of my personal feelings on it (and I didn't think it was a BAD game by any means).

#8
What is the game called? Wilfred the Goddamned Hero
What engine was this game made in? RPG Maker 2003 as of this release.
Who made this game? The Incomparable Teo Mathlein and the Unflappable Brandon Abley
When was this game submitted to RMN? How many downloads does it have?
July 23rd 2007. 1300+ Downloads.
What do I know about this game going in? Innovative RPG with 100% custom-assets, including graphics, music, and gameplay systems. I believe this is a demo.
Have I played this game? Yes, and I am a pretty big fan.
How many reviews does this game have? One, by VideoWizard.
How was this game reviewed? VideoWizard breaks it down like this:

Graphics (10) They couldn't be any better than they are now!
Sound (10) Among the best in an RPG Maker game.
Balance (6) It's a little off (nearly getting killed in the first battles), but still alright.
Atmosphere (10) Couldn't ask for anything better in RM.
Overall (9) Again, one of the top RM games.

#9
What is the game called? Dragon Fantasy
What engine was this game made in? RPG Maker 2000
Who made this game? Ephiam
When was this game submitted to RMN? How many downloads does it have?
June 13th, 2007. 727 Downloads.
What do I know about this game going in? I thought this game was a Dragon Quest clone made in RM2k. Its creator says "Traditional NES-type RPG made in RPG Maker 2000. Help the chosen hero (you) save the world from the Entity Of Sin in which wishes to be reborn. Protect the crystals and restore light to the world." So I wasn't far off. I believe this is a complete game.
Have I played this game? Only briefly. I don't feel qualified to personally comment on it.
How many reviews does this game have? Four. Reviews from Enker, Myersguy, and Maia cement this game's place on the top twenty.
How was this game reviewed? A possible error is a less-than-glowing review from Aurabolt, which describes the game as "Old School at Its Pretty Good" did not carry a numerical rating. If it did, one imagines it might have knocked this game out of the top twenty although not by much.

#10
What is the game called? Iffermoon
What engine was this game made in? Our first Game Maker game! And also (spoiler) our last.
Who made this game? animdude
When was this game submitted to RMN? How many downloads does it have?
June of 2008; 2285.
What do I know about this game going in? Almost literally nothing. Amazing 3D Graphics and cartoon...ducks? This is a complete game?
Have I played this game? I have never even HEARD of this game.
How many reviews does this game have? One.
How was this game reviewed? The game's lone review is by Jabbo, and it is also Jabbo's lone review. What does Jabbo have to say?

Pros
* The graphics are great
The sound and music fit in very well
The atmosphere of the world flows
The battle system is fun to explore and doesn't get old
There are a lot of things to collect and the arcade minigames are fun

Cons
* Long loading times
Some lack of battle balance
Some of the in-game tasks are repetitive

Summary
Iffermoon is a pretty short game that stays fun all the way through. It portrays the side scrolling RPG genre well and never falls flat of its aspirations. If you can get over the loading times, then you'll find yourself with a vibrant, interesting world to explore and a rewarding gameplay experience. It may not be the best game of all time, but Iffermoon is NOT "more of the same" in the RPG world, and you'd really be missing out not to play it.

#11
What is the game called? Castle Quest
What engine was this game made in? RPG Maker 2003
Who made this game? Blitzen
When was this game submitted to RMN? How many downloads does it have?
April 2009. 545 Downloads.
What do I know about this game going in? "Basically Castle Quest is an old-tyme dungeon adventure game." Its most conspicuous feature is B&W graphics. I believe this is a demo?
Have I played this game? I had never heard of this game.
How many reviews does this game have? Two.
How was this game reviewed? There is a very simple review from Maia with no categories which concludes "Plain and simple, this game is fun and different than any other RPG Maker game. It is a demo that will take you 30-60 minutes to play through and well worth your time". Another review by thunderdraco "A good game that could use some revisions" includes CATEGORIES WHICH TOTAL TO AROUND 3-3.5 (Gameplay 3, Graphics/Mapping 4, Atmosphere and Setting 3, Story 3) BUT HAS NO FINAL SCORE. So although both reviews are meant to have scores, only one score is being counted to determine this game's rating, Maia's 4.5 review. We have a review with a final score and no categories and a review with categories and no final review.

#12
b]What is the game called? The Frozen World
What engine was this game made in? RM2k
Who made this game? Lysander86
When was this game submitted to RMN? How many downloads does it have?
June 2007. 3111. Wow.
What do I know about this game going in? This is a very well-received game by Lys86.
Have I played this game? Not enough to be qualified to give an opinion on it.
How many reviews does this game have? Two.
How was this game reviewed? Well, as in glowing praise and basically unassailable reviews (Mr. Y and Solitayre). This one definitely deserves its place here.

#13
What is the game called? The Blue Contestant
What engine was this game made in? RM2k
Who made this game? Lysander86
When was this game submitted to RMN? How many downloads does it have?
June, 2007 but only 1115 downloads. : (
What do I know about this game going in? This is one of my favorite RM games.
Have I played this game? I have beaten it more than once. I LOVE it.
How many reviews does this game have? There is A review. From Mr. Y.
How was this game reviewed? But it is only 4 lines long. Maybe I should review this game.

WHAT HAVE WE LEARNED

*For the most apart, all of the games with a Rating of 4.5 or better have a good amount of downloads. NOT shocking. Tina of the Stars is the glaring exception.

*GM appears once, RM2k and RMXP are well represented, as are RM2k3 and RMVX.

*Out of thirteen games rated 4.5 or higher, EIGHT of them had only ONE review. Four had two or more reviews. One had two reviews, but only one of them had a score.

*Most of the highest rating games are COMPLETE games which sits well with me.

Final Thoughts::
I was relieved not to find many glaringly horrible games amidst the Top 13. However it does seem strange to me that eight of the thirteen games RMN (as a site) puts forth as "The Best" have only one review. While I admit is certainly possible that "One" is the AVERAGE number of reviews for all games on the site, regardless of rating, it still seems to me that some of these games warrant a "second opinion" to determine whether or not they make the cut. Ultimately more research will have to be done to determine if multiple reviews over time do indeed tend to "standardize" a game's rating, or not. For certain there were a few games that received many reviews, ALL of which were high, like The Longing Ribbon and Dragon Fantasy.

In any case, I hope this information- a snapshot of RMN's top 13 games at the time of this writing, 2/27/2010- is useful to the people responsible for the ongoing of the Real Men Network.

Posts

Pages: first 12 next last
i accepted this but i dunno what i am doing my brain hurts nooooo
To be more serious I have no idea what you are trying to do here and it would be far more suited to a forum topic.
Solitayre
Circumstance penalty for being the bard.
18257
I find this idea of doing a "Top 10" or "Best of RMN" a pretty repugnant undertaking.
Craze
why would i heal when i could equip a morningstar
15150
This list was altered because of Max's latest polar review.
Man... Where are the diagrams? I would not have minded some sort of download count/score diagram and of course a bell curve of the "amount of reviews" vs "score".

This is like a datadump. And with all the first person narrative it's not very statistically interesting at that.
Sat Feb 27 | 11:56 am | <Kaiterra> You know I think the point he brings up about scores being skewed due to less reviews doesn't really amount to jack. If anyone ends up playing a shitty but high rated game, they can submit a review for it.
Sat Feb 27 | 11:56 am | <Kaiterra> But a lot of time people won't not just because they're too lazy
Sat Feb 27 | 11:57 am | <Kaiterra> But because they basically agree with the reviews that are up there already.
I have read this entire article and have come to the conclusion that replacing every instance of the word "I" with "dongs" makes it much more enjoyable.
Alas, the true list of quality games is tucked away in the forums instead of on the front page where it ought to be.

http://rpgmaker.net/forums/topics/2630/
Note that comment doesn't really mean that it's a non-issue, just that it mostly is in relation to your "top 13" point. If some game REALLY doesn't deserve to be on the top of the list, it'll get taken down by someone sooner or later, even if it's just a point or two.
Hey there's a good idea, a Featured Game archive linked from the Featured Game.
Review sample size is totally a known issue. I prefer to talk about possible solutions.
LOL WRITE MORE REVIEWS!!111111!!!11
I did an insanely time-consuming stats collection just because it's the stuff I occasionally obsess over.

Mostly because of a statement in the beginning of the article that didn't have any backing up:
I am not a statistician (although I have worked with them a little bit) but this seems to me that games with more reviews will have more "average" scores than games with less reviews. In other words, games with only one review are more likely to have high or low scores because there is not a process of aggregation. A game with one review might have 5 Stars or 1 Star because it got a 5 Star or 1 Star review. A game with two reviews which received a 1 Star review and a 5 Star review will have a three star review.

So some stats on that.
Stats for games with more than one review:
4,5 = 4 games
4 = 30 games
3,5 = 33 games
3= 32 games
2,5 = 13 games
2 = 12 games
1,5 = 2 games
1 = 5 games
0,5 = 2 games

This seems weighted toward games with 3 or more in review scores. Apparently people mostly review games they like. Let's see the stats for games with more than 2 reviews (3+ that is)
4,5 = 3 games
4 = 22 games
3,5 = 22 games
3 = 19 games
2,5 = 5 games
2 = 6 games
1,5 = 1 game
1 = 1 game
0,5 = 1 game

Doesn't change the trend a whole lot but it's interesting that 3 and 3,5 were the most popular when 2+ reviews were included while 3,5 and 4 were the most popular with 3+ reviews. (though the difference wasn't very many)

One fault of this is that I didn't count how many games in each category had one review to perhaps also see what percentage of games in a certain category have more than one.
Max McGee
with sorrow down past the fence
9159
Karsuman:
This is explicitly about the site, I thought it would be silly to post it on the forums. I know it is kind of a weird topic.

Solitayre:
I think you missed the point. I am just looking at the top RATED games on RMN. In short, RMN already has a Top 10, I'm just looking at it.

Shinan:
Yeah this is kind of a dry article, I agree! Diagrams would really have helped. Maybe someone else shall make them. I am not very good with diagrams.

DFalcon:
I think the first step is a detailed understanding of the problem. Besides, based on Darken's next post, the solution is a known issue to. : )

Shinan:
Excellent! I'm glad someone was willing to take this a step further.

Stats for games with more than one review:
4,5 = 4 games
4 = 30 games
3,5 = 33 games
3= 32 games
2,5 = 13 games
2 = 12 games
1,5 = 2 games
1 = 5 games
0,5 = 2 games


Very, very interesting. There is kind of a skewed bell-curve going on here, with the "fattest" portion of reviews centering around games with 3.5 Stars. If you move away in either direction the reviews really thin out. Which kind of goes towards my thesis: games with "outlier" scores and 1 Review might deserve a second opinion. Anyway, thanks for these stats. I would like to see those percentages.


It's not a very good solution.
he does make a good point that the highest and lowest rated games deserve a second review.

Yes but another point is, what if the second review has the same rating. Then the point of the second review becomes moot. What if people already agree with said review and have nothing else to add. It's the same as leaving the 1 review alone. People like Kait are arguing that if someone strongly disagreed with said review, they would have written one by now. Of course there is the well known fact: People are lazy. This is obvious and "WRITE MORE REVIEWS" hasn't been working because people are lazy (even me).

DFalcon mentioned in IRC the possibility of "mini one line reviews". Perhaps we could have two types of reviews. Detailed ones, and simple ones. The detailed ones are worth makerscore, have more of an average rating impact. But simple ones have a paragraph limit that briefly summarizes the poster's overall thoughts. This is not a very good solution IMO. However at least it will most likely generate a little more input. Something to work with. Something better than PLAY SOMETHING!! or REVIEW CHALLENGE!!

It reminds me back when you could THUMBS UP or THUMBS DOWN and it would affect the rating, but it didn't work because a) you didn't even have to play the game b) author's friends would thumbs up without hesitation. But "mini reviews" if detailed properly could be a better version of that.
Max McGee
with sorrow down past the fence
9159
Yes but another point is, what if the second review has the same rating. Then the point of the second review becomes moot. What if people already agree with said review and have nothing else to ad


I disagree with the bolded part. If a game has two reviews that have the same score (especially if those reviews were created "in a vacuum from each other") then that validates the first score.

The mini-reviews idea sounds interesting. I agree with Darken's opinion of it. A better but still far-from-perfect version of the thumbs up/thumbs down system.

I am glad that this article (which was itself a response to an ongoing discourse about reviews) has prompted further discussion.
Solitayre
Circumstance penalty for being the bard.
18257
Solitayre:
I think you missed the point. I am just looking at the top RATED games on RMN. In short, RMN already has a Top 10, I'm just looking at it.


I disagree with your wording then. No one thinks of these games as RMN's top twenty or whatever. That is definitely not what the star rating system is for. It's for a very simple reason, anyone from anywhere can post a five star review of any game and it would be at the top of your "Top Twenty" regardless of its actual quality.

The games aren't ranked, aren't meant to be ranked, and I object to your attempts to rank them based on their score. As I and others have reiterated at least eight thousand times this last week, reviews are much more about the content than the rating. It is the author's thoughts on the game and why it should or should not be played.
guests and new users filter thier searches based on the stars and click on game pages, again, based on the stars.

Even though the people who have been here long enough to know that they aren't really that important compared to the content of the review, they still have a huge impact if you are talking about how many people will give your game a play.

More reviews for those highest rated games certainly couldn't hurt. The more people review them, the more aware we are of whether these really are amon the site's best games or not.

I agree with you, of course, but I think what max has found through this study is that the games with the higher scores get the most downloads regardless of what is actually written in the reviews. The stars are what people see before they click on the game page; the review's content is what people see after they have already clicked on the game page, clicked on the game's reviews page, clicked on the review, and then read the review. This difference is important I think.

The review content SHOULD be what matters, but for alot of people its the score. That is why game creators get so bent out of shape when a mostly postive review gives a low score. They know people will think the game is worse then even the reviewer thought it was. The bottom line is we need more reviews, but we already knew that for awhile now.
Pages: first 12 next last