Full Games List
Featured Games
New Games
Reviews
Media
Images
Blogs
Game Updates
Images Feedback Summary
Game Design Highlights
Tutorials
Articles
Scripts
Utilities / Plugins
Resources
Engines
Engine Bulletins
What is Makerscore?
Past Events
Achievements
Misaos
Forums
Post History
Chat
New to RMN?
Full Store List
RMN Music Pack (free)
REVIEW SCORING: STANDARDIZATION, PROFESSIONALISM, ETC.
Posts
Decky
View playlists
CloseThis is a major issue that seems to affect much of the community whenever a harsh review is posted. Each time it comes up, the issue is either dropped or smothered. The issue is not necessarily with the content of reviews, but with the actual scoring. In other words, the score is either ridiculously low or ridiculously high relative to (1) the content of the review and (2) the standard throughout the site.
I've been discussing this issue with a lot of people over the last few months. There is such a wide range of opinions on what score signifies "average" or "subpar". Some people want RMN critics to review games on a strictly professional scale, in which a 2.5 out of 5 would be a standard attained by average pro games, let alone average RPG Maker games. Others feel that we should adapt the "game magazine" scoring system while still holding games to the amateur standard, in which case a 7.5/10 would likely be a middle-of-the-road score on the site.
Scoring is a direct reflection of the critic's philosophy. Some users who score low tend to hold these RPG Maker games to higher standards, encoraging the amateur designers to push themselves and their hobby so that they become the best. Other, more casual reviewers, are more considerate of the effort and the fact that it is a hobby with a steep learning curve. And there's a whole spectrum in between.
Scoring is the "bottom line", and we live in a bottom line society. We tend to measure everything by metrics and numbers, as they can be measured, standardized, and interpreted easily. The content of the review is most important to the developer, but the casual visitor seems to focus on the scores/averages (as well as the gamepage) when determining what to try - especially if they don't feel like sifting through a block of reviews. That is why this is an important issue that must be discussed.
Here is a place where we can discuss this tricky and multifaceted issue.
I've been discussing this issue with a lot of people over the last few months. There is such a wide range of opinions on what score signifies "average" or "subpar". Some people want RMN critics to review games on a strictly professional scale, in which a 2.5 out of 5 would be a standard attained by average pro games, let alone average RPG Maker games. Others feel that we should adapt the "game magazine" scoring system while still holding games to the amateur standard, in which case a 7.5/10 would likely be a middle-of-the-road score on the site.
Scoring is a direct reflection of the critic's philosophy. Some users who score low tend to hold these RPG Maker games to higher standards, encoraging the amateur designers to push themselves and their hobby so that they become the best. Other, more casual reviewers, are more considerate of the effort and the fact that it is a hobby with a steep learning curve. And there's a whole spectrum in between.
Scoring is the "bottom line", and we live in a bottom line society. We tend to measure everything by metrics and numbers, as they can be measured, standardized, and interpreted easily. The content of the review is most important to the developer, but the casual visitor seems to focus on the scores/averages (as well as the gamepage) when determining what to try - especially if they don't feel like sifting through a block of reviews. That is why this is an important issue that must be discussed.
Here is a place where we can discuss this tricky and multifaceted issue.
Perihelion
665
I think adopting a game magazine-like score system is silly when you consider that the reason professional review scores are so high is pressure from game companies. We have no such restriction here. Limiting the score to only the upper end severely undermines the entire review system, and this applies to professional game reviews as well, because you eliminate the obvious benchmarks and institute invisible, arbitrary ones instead. This is not only confusing for the reader but makes it harder for review scores to be consistent between different writers. A 2.5/5 should be an average game regardless of who made it or what engine it was made in.
As for RM games specifically, in my opinion, it's intellectually dishonest and encourages complacency to have lower standards for RM games than other games, regardless of whether they're indie or mainstream. If I play an RM game, I play it to have fun, and if I read reviews of it, I want those reviews to compare it to other ways I could spend my time instead.
As for RM games specifically, in my opinion, it's intellectually dishonest and encourages complacency to have lower standards for RM games than other games, regardless of whether they're indie or mainstream. If I play an RM game, I play it to have fun, and if I read reviews of it, I want those reviews to compare it to other ways I could spend my time instead.
author=Fallen-Griever
If we want to be taken seriously then we should definitely compare our games to professional/indie games when applicable,
In that case, wouldn't most of the reviews on this site be .5/5 and 1/5? I highly doubt many games on this sites are are good as the average professional RPG. Hero's Realm to me was an excellent RPG Maker game - just as enjoyable as an average professional RPG. My scoring would be whatever "average" constitutes (2.5/5 or 3.5/5...but not 4.5/5).
Edit: but if we rate on the same scale as professional magazines, where 7.5/5 (or 3.75/5) is average, than even our bad games would only be a 1/5. I see where you're coming from FG.
I've been of the opinion for the long time that we should eliminate stars and other numerical scoring metrics. S.F. LaValle said it well in the review comments:
Casual visitors who stroll through the site are likely to base their judgment of whether or not to play a game based entirely on the review score. This puts reviewers in a difficult position where they might be dissuaded against posting a low-score review because it would hurt the game's visibility. We should be able to post critiques and suggestions, but without a visible scoring metric that can be misinterpreted by outsiders and hurt a game's potential to be recognized.
That aside, I'm personally of the belief that we should make better games that deserve a higher score instead of adjusting our scores to make our games look better. Regardless of your scoring metric, you're going to either clump together a bunch of bad games or a bunch of good games under the same score. Whether you try to lift up the bad games or celebrate the good games is a matter of personal philosophy and I don't think there's a consensus on which should be done here.
While I'd personally like this not to be the case, these easily-influences star scores for games affect the game's visibility. To us, not by a great deal, but we can't ignore what implications a star score gives to site visitors who aren't scene-savvy.
Casual visitors who stroll through the site are likely to base their judgment of whether or not to play a game based entirely on the review score. This puts reviewers in a difficult position where they might be dissuaded against posting a low-score review because it would hurt the game's visibility. We should be able to post critiques and suggestions, but without a visible scoring metric that can be misinterpreted by outsiders and hurt a game's potential to be recognized.
That aside, I'm personally of the belief that we should make better games that deserve a higher score instead of adjusting our scores to make our games look better. Regardless of your scoring metric, you're going to either clump together a bunch of bad games or a bunch of good games under the same score. Whether you try to lift up the bad games or celebrate the good games is a matter of personal philosophy and I don't think there's a consensus on which should be done here.
Well, I tried to tackle this before...
http://rpgmaker.net/articles/246/
http://rpgmaker.net/forums/topics/6897/
...it didn't really go anywhere.
http://rpgmaker.net/articles/246/
http://rpgmaker.net/forums/topics/6897/
...it didn't really go anywhere.
Casual visitors who stroll through the site are likely to base their judgment of whether or not to play a game based entirely on the review score.Also, this is a false assumption. The presentation of the game page is actually the defining factor.
New rule: whenever a discussion gets sidetracked, we should all agree to discuss Hero's Realm instead.
Perihelion
665
author=Fallen-GrieverIf I play an RM game, I play it to have fun, and if I read reviews of it, I want those reviews to compare it to other ways I could spend my time instead.In which case, you'll want to compare it to professional games... which you can't do if we don't use the same fucking scale.
I object to the rating scale as it exists, but I don't necessarily disagree with that for the reason you said. I was unclear about that in my post, however. I'm somewhat ambivalent on the subject.
I see nothing against the "game magazine"/professional model of 7.5/10 being average. After all, nobody wants a low rating on their game, we don't want an entire site full of what looks like crap. In school, 5/10 is a failing grade. Even RESTAURANTS are out of 4 stars, and a 2/4 is barely acceptable. Hell, even Hotels. Do you ever hear anyone bragging that their review got a solid 2 stars?
The problem is the relativity of peoples' opinions. Frequently, people seem to want to respond to bad reviews or specific criticisms with "but it's better than so many other RM* games" or "well everyone else does the same thing" which is really the same as saying it is the most fragrant turd in the punchbowl. Let's stop pointing to "other RM* games" as an acceptable level of the bar, because any idiot can and has picked up an RM* engine and puked out gaem.
This seems pretty okay to me. As compared to a professional RPG, I would have to give Hero's Realm 3.5. Compared to "other RM fare" it's a solid 4-5 stars, because that is using the frame of mind that other people have mentioned about scoring based on intent. I understand what the game is going for, it entertains me, it makes sense... but if it was compared to a modern, professional RPG that wasn't 'going for' the nostalgic design? That score isn't going to hold up. I do think it's a little unfair to hold every RM* game to a FULLY professional standard. We're not professionals. Games like Hero's Realm DO deserve recognition for design intent, so it's hard for me to come down clearly on one side of the standard or not.
(can I also add that using 2k3 automatically takes you out of the running to be truly compared to anything professional? if you want to get down to the nitty-gritty, the engine, resolution etc are all going to hold you back on a 100% professional scoring scale)
Whether you try to lift up the bad games or celebrate the good games is a matter of personal philosophy and I don't think there's a consensus on which should be done here.
The problem is the relativity of peoples' opinions. Frequently, people seem to want to respond to bad reviews or specific criticisms with "but it's better than so many other RM* games" or "well everyone else does the same thing" which is really the same as saying it is the most fragrant turd in the punchbowl. Let's stop pointing to "other RM* games" as an acceptable level of the bar, because any idiot can and has picked up an RM* engine and puked out gaem.
author=Sailerius
we should make better games that deserve a higher score instead of adjusting our scores to make our games look better.
author=Deckiller
Hero's Realm to me was an excellent RPG Maker game - just as enjoyable as an average professional RPG. My scoring would be whatever "average" constitutes (2.5/5 or 3.5/5...but not 4.5/5)
This seems pretty okay to me. As compared to a professional RPG, I would have to give Hero's Realm 3.5. Compared to "other RM fare" it's a solid 4-5 stars, because that is using the frame of mind that other people have mentioned about scoring based on intent. I understand what the game is going for, it entertains me, it makes sense... but if it was compared to a modern, professional RPG that wasn't 'going for' the nostalgic design? That score isn't going to hold up. I do think it's a little unfair to hold every RM* game to a FULLY professional standard. We're not professionals. Games like Hero's Realm DO deserve recognition for design intent, so it's hard for me to come down clearly on one side of the standard or not.
(can I also add that using 2k3 automatically takes you out of the running to be truly compared to anything professional? if you want to get down to the nitty-gritty, the engine, resolution etc are all going to hold you back on a 100% professional scoring scale)
I wish I still had the review statistics.
The average was about a 3.15/5. In other words, the average RMN game received somewhere between 3 and 3.5 stars, or between a 6 and 7 out of 10. That puts it slightly below the average for your typical game magazine review, but probably not far enough below average when you factor in the true differences of quality. If I'm even making any sense.
To me, the average rmn game is about equivalent to a playable, but poor professional RPG. A 2.5/5. My two favorite RPG Maker games - Hero's Realm and A Blurred Line - are about as enjoyable as games that I've given a high 7 or low 8/10 on GameFAQs - a 3.5 or 4/5. Slightly above average on that "game magazine" scale.
The average was about a 3.15/5. In other words, the average RMN game received somewhere between 3 and 3.5 stars, or between a 6 and 7 out of 10. That puts it slightly below the average for your typical game magazine review, but probably not far enough below average when you factor in the true differences of quality. If I'm even making any sense.
To me, the average rmn game is about equivalent to a playable, but poor professional RPG. A 2.5/5. My two favorite RPG Maker games - Hero's Realm and A Blurred Line - are about as enjoyable as games that I've given a high 7 or low 8/10 on GameFAQs - a 3.5 or 4/5. Slightly above average on that "game magazine" scale.
author=Deckiller
The average was about a 3.15/5. In other words, the average RMN game received somewhere between 3 and 3.5 stars, or between a 6 and 7 out of 10.
this seems pretty much correct for "average" games out of a score of 5 where 5 is amazing
author=Fallen-GrieverThat aside, I'm personally of the belief that we should make better games that deserve a higher score instead of adjusting our scores to make our games look better.If you want to use professional games as the metre stick, you need to use the scoring system used to measure those games. Anything else is silly.
As soon as I start being bribed to give games higher scores than they deserve, I'll start doing it.
author=Fallen-GrieverThat aside, I'm personally of the belief that we should make better games that deserve a higher score instead of adjusting our scores to make our games look better.It's not adjusting the scores to make games look better, it's correcting the fact that we currently lower scores to make them look worse.
Or, in some cases, giving games higher scores than they deserve if we compare to professional games. It's a bit of both.
The bottom line is those magazine scores are the ones being issued, bribes or not. People see those scores and compare using those scores.
Racheal
7238
Perhaps instead of a numerical scoring system, or as well as, there could be a word system. One person might view 2.5/5 as "average" whereas another might view it as "just barely playable," but if you label it as average, then everyone would view that rating similarly. It wouldn't functionally be that different, but it might make a rating more universal from one person to another.
