SPOT CHECKING THE RATING SYSTEM VOLUME I: FEBRUARY-MARCH 2010

I review (not as in critique, as in, look over) the top 13 games and the available data on them.

Spot Checking the Rating System Volume I: February-March 2010
I review (not as in play and critique, as in, look over) the top 13 games and the available data on them.

Common sense tells us that RMN site users (not necessarily forum users) will most likely use a rating filter to see which are the "best" games available on the site, and play those games. So the numerical ratings that games receive (based on reviews which have no standardized criteria and which everyone can submit) are super important.

I apologize if the following encroaches upon EPIC MATH FAIL:

The way that reviews work here, each game is given a review with a rating from 0.5 to 5 Stars. I am not a statistician (although I have worked with them a little bit) but this seems to me that games with more reviews will have more "average" scores than games with less reviews. In other words, games with only one review are more likely to have high or low scores because there is not a process of aggregation. A game with one review might have 5 Stars or 1 Star because it got a 5 Star or 1 Star review. A game with two reviews which received a 1 Star review and a 5 Star review will have a three star review.

For this article I looked at all of the games on RMN with a rating of 4.5 or higher (putting myself in the shoes of someone just coming to this site to play computer games, with no prior knowledge of the community to influence my decisions, who was using rating to look for the BEST games). For each game I will be looking at several categories to determine why the game is on the top twenty list, if it seems to make sense that it is there, and what review data is available for it. From this data we should be able to get a picture of how (and how well) RMN's "Star" rating system is working and look for any interesting hiccups or inconsistencies in the data set. This evaluation was not written with any particular conclusion in mind and I look forward to seeing what I can learn. So let's dive in.

I will be asking the following questions about each of RMN's top games.

What is the game called?
What engine was this game made in?
Who made this game?
When was this game submitted to RMN? How many downloads does it have?
What do I know about this game going in?
Have I played this game?
How many reviews does this game have?
How was this game reviewed?

In this way I will be trying to correlate several things. Rating and download count. Number of reviews and rating. number of reviews and download count. Etc. and so on. Now, let us BEGIN.

#1
What is the game called? Cast Aside
What engine was this game made in? RMXP
Who made this game? Anamei Productions
When was this game submitted to RMN? How many downloads does it have?
Added to RMN on June 20th, 2009, Cast Aside has 1,380 downloads.
What do I know about this game going in? I have heard of this game but never played it. It was made by Anamei Productions who are known for very very very high quality, polished final products with original art and complex and unique custom systems. I played their SkyE last year and was utterly blown away by it. This game is described as an Episodic Active Strategy game with puzzles and an involved plot. Its tone appears to be somewhat light to me and it seems to have something of an anime style. This is an episodic game.
Have I played this game? No, and hence I am not qualified to discuss its actual quality.
How many reviews does this game have? This game has been reviewed once by Solitayre.
How was this game reviewed? I may repeat this a LOT in future sections, but Solitayre grades on five categories. Balance, level design, characters, story, and music and sound. In this case, Cast Aside received a Balance Score of 3/5, a Level Design Score of 4.5/5, a Characters score of 5/5, a storyline score of 4/5, and a music and sound score of 3/5. Solitayre says: "Any fan of tactical RPGs could do far worse than to give this game a try. Or any fan of puzzle games. Or fans of well-written characters. Hell, the only real complaint I have with this game is that it’s just not long enough! There must be more. I look forward immensely to seeing where this project goes."

#2:
What is the game called? Tina of the Stars
What engine was this game made in? RMXP
Who made this game? Ratty524, who it is worth noting, I have NEVER heard of.
When was this game submitted to RMN? How many downloads does it have?
July 22nd, 2009 and 70, respectively. This is our first anomalous statistic as the downloads have been around for some time. However there are several explanations, such as technical difficulties, for why the download count does not match the rating and time the game has been available. Of course if we rule that out we might attribute it to the game's genre or tone.
What do I know about this game going in? All I know is that the game has bright, cheery graphics and implores me to "Grab some stars and save your fairy friends!". As far as I can tell this is a complete game.
Have I played this game? No, and hence I am not qualified to comment on its quality.
How many reviews does this game have? One, by Fallen-Griever, a well-known reviewer.
How was this game reviewed? This game was reviewed shortly after it came out. Fallen-Griever grades in terms of Gameplay, Characters/Dialogue/Storyline, Graphics/Mapping, and Music/Sound. Each aggregated category is given a score out of 10. Fallen-Griever gave this game an NA in the Charactes/Dialogue/Storyline aggregate as it is an arcade game. Other categories got a 9/10 except for music/sound which got a 10. Fallen-Griever said of the game: "For what this game is trying to be it is really good. It obviously isn't one for those people who are into games with lots of depth, purpose, storyline and characterisation, but as far as simple, arcade action goes this game is solid. Like Pacman? Play this."

#3:
What is the game called? Legends of Illarion
What engine was this game made in? RPG Maker 2003
Who made this game? Dyluck
When was this game submitted to RMN? How many downloads does it have?
5/18/2009 :: 2410
What do I know about this game going in? "Legends Of Illarion is an RM2K3 game based on Illarion, the online roleplaying game. Illarion is a free, graphical fantasy game that focuses on true roleplaying rather than hack & slash." It appears to use some RTP graphics. As far as I can tell this is a complete game.
Have I played this game? Nope. See above.
How many reviews does this game have? One review, from SilverSoul, who has written one other review.
How was this game reviewed? SilverSoul does not use numerical categories. He had this to say: "All in all, I had a lot of fun with this game. I stopped right at the end trying to fight the dragons, but I found the size of that sidequest very satisfying. The sheer number of characters, and having to switch between them, keeps gameplay from becoming monotonous, and the harvesting system gives you something to look forward to. There are also plenty of amusing scenes and supporting characters. I give this game a very high rating for people who like traditional RPGs."

#4:
What is the game called? Arian Wild
What engine was this game made in? RMVX
Who made this game? Craze & Karsuman
When was this game submitted to RMN? How many downloads does it have?
January 2nd, 2010; 280 Downloads
What do I know about this game going in? "Arian Wild is a game made by the creators of Visions & Voices. It is a linear RPG in the vein of traditional role-playing classics, except for the many parts where it isn't." It looks "dark". As far as I can tell this is a battle demo.
Have I played this game? Not yet. I did play Visions & Voices, but still see the disclaimers above!
How many reviews does this game have? One, by Forever an Ill Fate
How was this game reviewed? FaIF's review reviewed only the battles, and seemed more like a beta test report than a true review. He did not assign numerical categories, nor should he have, since he just played a battle demo. He compared the game to an "Etrian Odyssey and WoW love child".

#5
What is the game called? Geondun
What engine was this game made in? RMVX
Who made this game? Craze
When was this game submitted to RMN? How many downloads does it have?
March 2009. 1177 Downloads.
What do I know about this game going in? This game HAVE STATS. "Six races, six classes, three faiths - 432 party combinations." This is a demo.
Have I played this game? Yes, finally, and I wish I had had time to play it more!
How many reviews does this game have? One (noticing a pattern yet?), from Fallen-Griever.
How was this game reviewed? Gameplay 9/10, Characters Classes 10/10, Graphics/Mapping and Music and Sound both 6/10. Storyline was (again) dismissed as NA. I largely agree with Fallen-Griever's review...my scores would have been within one standard deviation different. Griever's final word:
"The aesthetic factors aren’t even slightly important in this game. It could have the worst music and graphics ever and, so long as everything was still functional, I would’ve probably still enjoyed this game. Maybe not quite as much, but I would've enjoyed it. I'm sure Craze is sick of hearing the D-word by now, but this game surpasses Demon Tower by mile; if you are one of the many fans of Demon Tower I really suggest you download this instead as it is vastly superior."

#6
What is the game called? RMN Christmas Card 2009. Awww...
What engine was this game made in? RM2k3
Who made this game? YDS and AznChipmunk, possibly with help from others.
When was this game submitted to RMN? How many downloads does it have?
I'm going to guess Christmas of 2009? 222 Downloads.
What do I know about this game going in? It is a community game of some kind?
Have I played this game? Nope.
How many reviews does this game have? FIVE! Wow, what a way to break the trend!
How was this game reviewed? All of them are very glowing! As a community game being reviewed well by the community and the specific people who made it, this doesn't warrant much more serious analysis.

#7
What is the game called? The Longing Ribbon
What engine was this game made in? RPG Maker 2000. A first on our list.
Who made this game? Gibmaker.
When was this game submitted to RMN? How many downloads does it have?
February 6th, 2008, making it by far the oldest game looked at so far. For that, it has a whopping 2600+ downloads. This is a complete game.
What do I know about this game going in? Horror RPG, winner of the 2005 Misao for Best Atmosphere.
Have I played this game? I played this game in 2005 and was impressed by its atmosphere. The battle system bothered me a lot, however, as I never got very far in it. I filed it off as a forgotten classic like barrel the apocalypse Guile Machination.
How many reviews does this game have? Four.
How was this game reviewed? Well. Honestly, the reviews here are by so many people and are all so glowingly positive that out of all the games I've looked at so far this is the only one whose place in the top twenty seems unassailably earned regardless of my personal feelings on it (and I didn't think it was a BAD game by any means).

#8
What is the game called? Wilfred the Goddamned Hero
What engine was this game made in? RPG Maker 2003 as of this release.
Who made this game? The Incomparable Teo Mathlein and the Unflappable Brandon Abley
When was this game submitted to RMN? How many downloads does it have?
July 23rd 2007. 1300+ Downloads.
What do I know about this game going in? Innovative RPG with 100% custom-assets, including graphics, music, and gameplay systems. I believe this is a demo.
Have I played this game? Yes, and I am a pretty big fan.
How many reviews does this game have? One, by VideoWizard.
How was this game reviewed? VideoWizard breaks it down like this:

Graphics (10) They couldn't be any better than they are now!
Sound (10) Among the best in an RPG Maker game.
Balance (6) It's a little off (nearly getting killed in the first battles), but still alright.
Atmosphere (10) Couldn't ask for anything better in RM.
Overall (9) Again, one of the top RM games.

#9
What is the game called? Dragon Fantasy
What engine was this game made in? RPG Maker 2000
Who made this game? Ephiam
When was this game submitted to RMN? How many downloads does it have?
June 13th, 2007. 727 Downloads.
What do I know about this game going in? I thought this game was a Dragon Quest clone made in RM2k. Its creator says "Traditional NES-type RPG made in RPG Maker 2000. Help the chosen hero (you) save the world from the Entity Of Sin in which wishes to be reborn. Protect the crystals and restore light to the world." So I wasn't far off. I believe this is a complete game.
Have I played this game? Only briefly. I don't feel qualified to personally comment on it.
How many reviews does this game have? Four. Reviews from Enker, Myersguy, and Maia cement this game's place on the top twenty.
How was this game reviewed? A possible error is a less-than-glowing review from Aurabolt, which describes the game as "Old School at Its Pretty Good" did not carry a numerical rating. If it did, one imagines it might have knocked this game out of the top twenty although not by much.

#10
What is the game called? Iffermoon
What engine was this game made in? Our first Game Maker game! And also (spoiler) our last.
Who made this game? animdude
When was this game submitted to RMN? How many downloads does it have?
June of 2008; 2285.
What do I know about this game going in? Almost literally nothing. Amazing 3D Graphics and cartoon...ducks? This is a complete game?
Have I played this game? I have never even HEARD of this game.
How many reviews does this game have? One.
How was this game reviewed? The game's lone review is by Jabbo, and it is also Jabbo's lone review. What does Jabbo have to say?

Pros
* The graphics are great
The sound and music fit in very well
The atmosphere of the world flows
The battle system is fun to explore and doesn't get old
There are a lot of things to collect and the arcade minigames are fun

Cons
* Long loading times
Some lack of battle balance
Some of the in-game tasks are repetitive

Summary
Iffermoon is a pretty short game that stays fun all the way through. It portrays the side scrolling RPG genre well and never falls flat of its aspirations. If you can get over the loading times, then you'll find yourself with a vibrant, interesting world to explore and a rewarding gameplay experience. It may not be the best game of all time, but Iffermoon is NOT "more of the same" in the RPG world, and you'd really be missing out not to play it.

#11
What is the game called? Castle Quest
What engine was this game made in? RPG Maker 2003
Who made this game? Blitzen
When was this game submitted to RMN? How many downloads does it have?
April 2009. 545 Downloads.
What do I know about this game going in? "Basically Castle Quest is an old-tyme dungeon adventure game." Its most conspicuous feature is B&W graphics. I believe this is a demo?
Have I played this game? I had never heard of this game.
How many reviews does this game have? Two.
How was this game reviewed? There is a very simple review from Maia with no categories which concludes "Plain and simple, this game is fun and different than any other RPG Maker game. It is a demo that will take you 30-60 minutes to play through and well worth your time". Another review by thunderdraco "A good game that could use some revisions" includes CATEGORIES WHICH TOTAL TO AROUND 3-3.5 (Gameplay 3, Graphics/Mapping 4, Atmosphere and Setting 3, Story 3) BUT HAS NO FINAL SCORE. So although both reviews are meant to have scores, only one score is being counted to determine this game's rating, Maia's 4.5 review. We have a review with a final score and no categories and a review with categories and no final review.

#12
b]What is the game called? The Frozen World
What engine was this game made in? RM2k
Who made this game? Lysander86
When was this game submitted to RMN? How many downloads does it have?
June 2007. 3111. Wow.
What do I know about this game going in? This is a very well-received game by Lys86.
Have I played this game? Not enough to be qualified to give an opinion on it.
How many reviews does this game have? Two.
How was this game reviewed? Well, as in glowing praise and basically unassailable reviews (Mr. Y and Solitayre). This one definitely deserves its place here.

#13
What is the game called? The Blue Contestant
What engine was this game made in? RM2k
Who made this game? Lysander86
When was this game submitted to RMN? How many downloads does it have?
June, 2007 but only 1115 downloads. : (
What do I know about this game going in? This is one of my favorite RM games.
Have I played this game? I have beaten it more than once. I LOVE it.
How many reviews does this game have? There is A review. From Mr. Y.
How was this game reviewed? But it is only 4 lines long. Maybe I should review this game.

WHAT HAVE WE LEARNED

*For the most apart, all of the games with a Rating of 4.5 or better have a good amount of downloads. NOT shocking. Tina of the Stars is the glaring exception.

*GM appears once, RM2k and RMXP are well represented, as are RM2k3 and RMVX.

*Out of thirteen games rated 4.5 or higher, EIGHT of them had only ONE review. Four had two or more reviews. One had two reviews, but only one of them had a score.

*Most of the highest rating games are COMPLETE games which sits well with me.

Final Thoughts::
I was relieved not to find many glaringly horrible games amidst the Top 13. However it does seem strange to me that eight of the thirteen games RMN (as a site) puts forth as "The Best" have only one review. While I admit is certainly possible that "One" is the AVERAGE number of reviews for all games on the site, regardless of rating, it still seems to me that some of these games warrant a "second opinion" to determine whether or not they make the cut. Ultimately more research will have to be done to determine if multiple reviews over time do indeed tend to "standardize" a game's rating, or not. For certain there were a few games that received many reviews, ALL of which were high, like The Longing Ribbon and Dragon Fantasy.

In any case, I hope this information- a snapshot of RMN's top 13 games at the time of this writing, 2/27/2010- is useful to the people responsible for the ongoing of the Real Men Network.

Posts

Pages: first prev 12 last
Solitayre
Circumstance penalty for being the bard.
17987
One of the biggest problems I have with the community is you can't actually find many good games that way. The ultimate irony is that the better/more popular a game is, the more acclaim/attention it receives, and the more reviews it gets, dragging its score towards the middle. Most of the games I would consider to be truly exception have much more "average" ratings. Which again, is a big reason why going by stars alone is a very bad idea. It is very easy for a game with one review to have a high rating. It is much harder for a game with many reviews.

The best advice I can give is find a reviewer you consistently agree with and see what they say about a game.
I'm beginning to see Soli's point.
Craze
i bet she's a diva with a potion popping problem
13234
I am saddened that with three post-overhaul four-star reviews for V&V are outweighed by two pre-overhaul three-point-five-star reviews due to MATH.
Craze
i bet she's a diva with a potion popping problem
13234
(This is a real-life example of Soli's post.)
"The best advice I can give is find a reviewer you consistently agree with and see what they say about a game."

Admittedly useful, but this option doesn't always work, if only because there are more games than reviews total, let alone by any one person.

For me, Solitayre, -you- are one such person. I've enjoyed your review style, and have tended to agree with you where I've actually looked for myself at a game you've reviewed. I'm also fortunate that you're the top-ranked reviewer by quantity - thereby giving me a greater number of possible options. And you're not the only reviewer for whom I hold this. However, in response, I've also shied away from some games on the grounds that without a review from one of those in my 'I trust these people' slots it's too big a risk; I don't have that much free time, and I don't really want to gamble that against something, no matter how good it actually is, if there's a chance it might not be worth the investment. I'm sure I've missed some wonderful games because of this, but that's my loss.

And I have to agree with the general point of the original article, as well. You seem to be coming at this from a different tack than the original article's investigation. This article isn't about this community; it's about those on the outside of the community who are looking in to find something to play. So long as we have any kind of ranked measurement whatsoever on the games begin hosted here, those individuals will use that, more than any other option, to pick what they want to look into. Others have made the point that people are lazy - as an explanation to why those who agree with reviews aren't likely to push one of their own up as well. But that laziness also affects those just dropping by to pick up something to waste their time for a while - the 'easy' choice is to go by star-rank, rather than following reviews, because it requires less mental effort.

Going by star rank requires 3 steps. 1)Oh, this has five stars! 2)Head to game download page. 3)Download.

Going with reviews requires several more. At the least, we've got W) Go to review page, X)Hmm, which of these reviews should I look at? Y)Actually reading the review, and Z)Well, do I really trust this guy? Does he like or appreciate the same things I would? And remember, that's -in addition to- steps 1-3 above.

And don't forget, even though it can be very useful in making the decision, time spent reading one or more reviews is time not spent actually playing the game. That's also going to count against looking at them for the kind of player I feel this article was written regarding.
Max McGee
with sorrow down past the fence
9219
It's for a very simple reason, anyone from anywhere can post a five star review of any game and it would be at the top of your "Top Twenty" regardless of its actual quality.


Totally one of the reasons for this article being made, man. : )
Max McGee
with sorrow down past the fence
9219
guests and new users filter thier searches based on the stars and click on game pages, again, based on the stars.


Someone give this man a gold star.

(Although I'm not accusing anyone of having failed to apprehend this fact. : ) )

One of the biggest problems I have with the community is you can't actually find many good games that way


Yet another one of the reasons that I wrote this article. Sorry, I missed that there was a second page when I last commented.
Max McGee
with sorrow down past the fence
9219
Sorry inability to edit comments is REALLY killing me. But I think EnderX has struck on exactly what I was getting at.
kentona
One of RMN's Top 10 Admins of all-time
20443
Aggregate scores of all the reviews a game has would be interesting... *has 37.5 star game*
kentona
One of RMN's Top 10 Admins of all-time
20443
actually, simply listing more information on the game list would help visitors make more informed decisions. Number of Reviews/Ratings would help identify the outlier games ("5 stars, but only 1 rating...hrmmm..."), Download counts could be hugely influential ("wow! 5437 people can't be wrong!"), date released/completed ("I want new games!"), name of the lead maker ("wow, this kentona guy makes good games!") etc..etc..

If the issue is uninformed consumers of the site, the most straightforward way to fix that would be to make them more informed.
kentona
One of RMN's Top 10 Admins of all-time
20443
I wonder what game has the most reviews? It'd be interesting to know what we've peeked at so far.
Solitayre
Circumstance penalty for being the bard.
17987
Right now I think Forever's End and Hero's Realm have the most by a long shot.
All of them are very glowing! As a community game being reviewed well by the community and the specific people who made it, this doesn't warrant much more serious analysis.

I am kind of curious what you mean by "the specific people who made it"..."doesn't warrant much more serious analysis."

And it was only YDS and I who made the game, everyone else just submitted dialogue or whatnot.
Max McGee
with sorrow down past the fence
9219
Oh really? That was unclear from your gamepage! I thought the people on the "thanks" page had all made significant contributions.

Anyway, all I meant is that I wasn't shocked that a community game was well reviewed by the people who contributed to it. : )
Sated
puking up frothing vitriolic sarcastic spittle
4064
If there were enough reviews on games I would say that the highest and lowest reviews be ignored when calculating the average score - you know, like they do with diving and ice-dancing and other such crap at the Olympics/Winter Olympics?

Alas, there are not enough reviews for this to be feasible, but it would eliminate the "one review and skew the general opinion" problem.

(And it would move Sore Loser's rating up. Huzzah!)
Max McGee
with sorrow down past the fence
9219
Welcome back F-G where have you been? (Ignore the fact that i myself has been gone.)
Pages: first prev 12 last