REVIEW SCORING: STANDARDIZATION, PROFESSIONALISM, ETC.

Posts

Ciel
an aristocrat of rpgmaker culture
367
mediocrity is not really a problem because like i said it all comes down to whether you are willing to recommend that others play the game or not. they will either play it or not play it. your reputation as a reviewer is at stake! the point of a review is ultimately to tell people if they should invest (time, money) in the game or not. you can't really say 'well i kind of think you should play it but then again you shouldn't 5/10', a reviewer who does that has rendered himself completely useless to everyone. you will eventually have to bite the bullet and tell people if you can recommend it or not. caveats can be expressed in the written portion (which people should be reading) and were never effectively communicated by any form of numerical system anyway.
halibabica
RMN's Official Reviewmonger
16633
Hmm, looking at what's been said in favor of thumbs up/down so far, I'm starting to think it would be a good idea. It wouldn't have to replace the current system; it could just be an additional thing. I remember a while back when RMN had a system like that, but in those days it determined the game's star score via average. So 1 up vote meant a 5-star game. o_o

Having it detached from the game's rating is obviously necessary. But even if only a handful of people used it for a particular game, like Ciel said, you definitely know that number of people cared enough to vote up or down.

If we do end up implementing it, I'd just like to see two things:

1. only one vote per person (per game)

2. ability to change your vote (yay flip-flopping)
Max McGee
with sorrow down past the fence
9159
I think thumbs up/thumbs down is a pretty bad idea. It seems very easy to abuse, and I don't see how it fixes any of the problems of starred reviews. A lot of other suggestions that have been made in this topic I've liked a lot more.

2. ability to change your vote (yay flip-flopping)

What purpose does this serve, except bullshit like downvoting someone's game because they were mean to you, or because they downvoted yours? In all seriousness, your review score of a game might change over the years but why would whether or not you liked it ever do a complete 180?

If we do HAVE to have thumbs up/thumbs down it goes without saying (hopefully) that it should only be possible for games with downloads. Games are already judged enough on the basis of their screenshots.
halibabica
RMN's Official Reviewmonger
16633
I should think it actually far more difficult to abuse. Say you're a starving developer and your game is in a funk. You say to your buddies, "hey guys go hype up my game through RMN's thumbs system!" So they do it, and you get about five more thumbs up than you should.

...so what? That's it? If the game is really awful, then won't the people who play it later vote it back down?

Also, the main reason I would like changeable votes is mostly because a game you did not like before could be improved to the point where you would like it. Suppose a demo came out that was really terrible and needed vast, significant improvements. If you down voted it before, then later came back to find it was fixed up a great deal, wouldn't you want to change your vote?
sounds really bad, but i'm not even sure what the argument is for changing the system anymore.

the unavoidability of people deciding to be extreme and rate a game many think is unfair will only be somewhat alleviated with a thumbs up or thumbs down system and that's at the expense of giving less information in the end.
plus, i expect the average game doesn't actually get many reviews, so most games would just be like 1 thumb up. 1 thumb down. 3 thumbs up, 1 thumb down (if you're lucky enough to get 4 reviews), which gives far less information than a score.

author=Ciel
you can't really say 'well i kind of think you should play it but then again you shouldn't 5/10', a reviewer who does that has rendered himself completely useless to everyone.
well, no, not really (i mean, the speech is obviously useless, but we both know nobody would actually say that haha).
in fact knowing he's rated it 5/10 is very not useless and i don't see how it would be. if they're not totally inept, the readers should then use their own head to decide whether that's good enough for them to give it a go based on the good/bad factors presented in the review. by changing what he would've given 5/10 to 'thumbs up,' it's simplifying it way too much and leaving less in the hands for the gamer to decide whether it's worth his/her time.

saying that, a good thing i can see coming of it is people reading the reviews a whole lot more seeing as one thumb up could mean anything from 'this is fantastic' to 'yeah it's passable.'

author=halibabica
So they do it, and you get about five more thumbs up than you should.
tbh, if the thumbs come along with reviews, then people will only get their mates to rate their games as much as they do now. the same amount of effort would require to be put in, only maybe a bit less information needed to warrant a thumbs up/down. i doubt you would ever see 5 reviews biased towards the developer and if so it would be painfully obvious what was going on.
Max McGee
with sorrow down past the fence
9159
saying that, a good thing i can see coming of it is people reading the reviews a whole lot more seeing as one thumb up could mean anything from 'this is fantastic' to 'yeah it's passable.'


Agree. Loss of granularity is bad. A less granular system does nothing to augment a more granular one when added on.
halibabica
RMN's Official Reviewmonger
16633
Um...I think something got mixed up somewhere in there.

Who said thumbs up/down would only come with reviews? Or was that what Ciel was proposing?

Because I don't think that would do any good. A regular up/down system to go along with our current one could be helpful, but replacing review scores with up/down would not.

I think we may be overthinking this issue!
Starscream
Conquest is made from the ashes of one's enemies.
6110
author=halibabica
I think we may be overthinking this issue!


winner winner, chicken dinner
author=Max McGee
saying that, a good thing i can see coming of it is people reading the reviews a whole lot more seeing as one thumb up could mean anything from 'this is fantastic' to 'yeah it's passable.'
Agree. Loss of granularity is bad. A less granular system does nothing to augment a more granular one when added on.


Though obviously we're all talking up different ideas at the same time, I figured I may as well be clear in terms of how I've been proposing a change: the thumbs up/thumbs down system would replace the star review system. At least in terms of recommending games to players, it's simply "I would" or "I wouldn't." I think a loss of granularity is precisely what is needed this case. This is true because of the problem you mention above: abuse. It's a bit harder to abuse the current system because a full review is required, but when you clear that hurdle, the damage is greater (as we have witnessed). One ill-intended "thumbs down" is hardly a dealbreaker, though, and it should be among a greater sampling (in a perfect world, much greater sampling) of honest thumbing.

author=boobledeeboo
well, no, not really (i mean, the speech is obviously useless, but we both know nobody would actually say that haha).
in fact knowing he's rated it 5/10 is very not useless and i don't see how it would be. if they're not totally inept, the readers should then use their own head to decide whether that's good enough for them to give it a go based on the good/bad factors presented in the review. by changing what he would've given 5/10 to 'thumbs up,' it's simplifying it way too much and leaving less in the hands for the gamer to decide whether it's worth his/her time.


This may sound misleading, but we do have a role that's different than, say, IGN or Joystiq. While they are providing objective (lol) coverage and reviews for commercial games they have no stake in (lol again), we are actually supposed to be helping indie game makers peddle their games to the people. When you think about that, it actually sounds quite silly that we would host reviews talking down the very games we're offering to the public. Meaning, while being a tad more deceptive as you say by not discussing that middle ground of mediocrity, we are compelling more people to download games they might not if it had a 2.5 instead of 5 up/5 down (or, like you say, they may just read full reviews to investigate further).

It seems ultimately beneficial to the developers, removing that granularity. I also like the idea of being able to change your vote. People might be more receptive and responsive to criticism if they know someone's thumb vote hangs in the balance. It seems petty, but minor as it is, it puts more control in the hands of the developer. Even now as they clamor for star points, what will they be willing to do to gain that one or few extra thumbs up? Have the game proofread to remove spelling and grammar errors? Shorten the intro to 30 seconds? Change the hero chipset to something other than a Cloud re-color? Increase text and/or walking speed? Invest time into better battle balancing? Maybe a bunch of things that a creator won't find worth it; whatever, it's up to them. Instead of feeling the weight of a 0.5 review slamming down on you with no chance of shrugging it off, can a series of nudges with many little thumbs inspire you to make your game just a little bit better? Where the line is drawn is up to you.
Sailerius
did someone say angels
3214
author=S. F. LaValle
author=Max McGee
saying that, a good thing i can see coming of it is people reading the reviews a whole lot more seeing as one thumb up could mean anything from 'this is fantastic' to 'yeah it's passable.'
Agree. Loss of granularity is bad. A less granular system does nothing to augment a more granular one when added on.
Though obviously we're all talking up different ideas at the same time, I figured I may as well be clear in terms of how I've been proposing a change: the thumbs up/thumbs down system would replace the star review system. At least in terms of recommending games to players, it's simply "I would" or "I wouldn't." I think a loss of granularity is precisely what is needed this case. This is true because of the problem you mention above: abuse. It's a bit harder to abuse the current system because a full review is required, but when you clear that hurdle, the damage is greater (as we have witnessed). One ill-intended "thumbs down" is hardly a dealbreaker, though, and it should be among a greater sampling (in a perfect world, much greater sampling) of honest thumbing.

author=boobledeeboo
well, no, not really (i mean, the speech is obviously useless, but we both know nobody would actually say that haha).
in fact knowing he's rated it 5/10 is very not useless and i don't see how it would be. if they're not totally inept, the readers should then use their own head to decide whether that's good enough for them to give it a go based on the good/bad factors presented in the review. by changing what he would've given 5/10 to 'thumbs up,' it's simplifying it way too much and leaving less in the hands for the gamer to decide whether it's worth his/her time.


This may sound misleading, but we do have a role that's different than, say, IGN or Joystiq. While they are providing objective (lol) coverage and reviews for commercial games they have no stake in (lol again), we are actually supposed to be helping indie game makers peddle their games to the people. When you think about that, it actually sounds quite silly that we would host reviews talking down the very games we're offering to the public. Meaning, while being a tad more deceptive as you say by not discussing that middle ground of mediocrity, we are compelling more people to download games they might not if it had a 2.5 instead of 5 up/5 down (or, like you say, they may just read full reviews to investigate further).

It seems ultimately beneficial to the developers, removing that granularity. I also like the idea of being able to change your vote. People might be more receptive and responsive to criticism if they know someone's thumb vote hangs in the balance. It seems petty, but minor as it is, it puts more control in the hands of the developer. Even now as they clamor for star points, what will they be willing to do to gain that one or few extra thumbs up? Have the game proofread to remove spelling and grammar errors? Shorten the intro to 30 seconds? Change the hero chipset to something other than a Cloud re-color? Increase text and/or walking speed? Invest time into better battle balancing? Maybe a bunch of things that a creator won't find worth it; whatever, it's up to them. Instead of feeling the weight of a 0.5 review slamming down on you with no chance of shrugging it off, can a series of nudges with many little thumbs inspire you to make your game just a little bit better? Where the line is drawn is up to you.

I agree with this system 100% and pretty much for the same reasons.
author=S. F. LaValle
This may sound misleading, but we do have a role that's different than, say, IGN or Joystiq. While they are providing objective (lol) coverage and reviews for commercial games they have no stake in (lol again), we are actually supposed to be helping indie game makers peddle their games to the people. When you think about that, it actually sounds quite silly that we would host reviews talking down the very games we're offering to the public. Meaning, while being a tad more deceptive as you say by not discussing that middle ground of mediocrity, we are compelling more people to download games they might not if it had a 2.5 instead of 5 up/5 down (or, like you say, they may just read full reviews to investigate further).
this is very true actually, i hadn't really thought of it that way.
Ciel
an aristocrat of rpgmaker culture
367
author=halibabica
Who said thumbs up/down would only come with reviews? Or was that what Ciel was proposing?

i did say that. when you write a review and it is accepted by the staff as being a properly written review with an appropriate amount of effort and impartiality, you enter a recommend/do not recommend verdict instead of a star score. you have to write a thoughtful review (proving that you actually played and analysed the game) in order to up/down it. this seems fair and logical to me.

as for changing your mind, if 1 year later or something a substantially improved version of a game you previously did not recommend is released i am sure the staff would allow you to submit a new review discussing why it is good now and give it a recommendation.
Max McGee
with sorrow down past the fence
9159
i did say that. when you write a review and it is accepted by the staff as being a properly written review with an appropriate amount of effort and impartiality, you enter a recommend/do not recommend verdict instead of a star score. you have to write a thoughtful review (proving that you actually played and analysed the game) in order to up/down it. this seems fair and logical to me.

Oh. I actually have much less problem with this. I don't know if this would fix the problems of the current system, but I don't see how it would make it worse either. And the textual portions of reviews could provide the difference between 'bad' and 'terrible' and between 'good' and 'great'.

Though obviously we're all talking up different ideas at the same time, I figured I may as well be clear in terms of how I've been proposing a change: the thumbs up/thumbs down system would replace the star review system. At least in terms of recommending games to players, it's simply "I would" or "I wouldn't." I think a loss of granularity is precisely what is needed this case. This is true because of the problem you mention above: abuse. It's a bit harder to abuse the current system because a full review is required, but when you clear that hurdle, the damage is greater (as we have witnessed). One ill-intended "thumbs down" is hardly a dealbreaker, though, and it should be among a greater sampling (in a perfect world, much greater sampling) of honest thumbing.

SFL, hypothetically, would reviewers still be allowed to assign scores and subscores within the text of the review? I know the site wouldn't average or display them. As much as the site has problems with how it handles numerical scores and visibility, I am still rather attached to them as an evaluative tool, and I'd like if this system were implemented to be able to continue using them informally in my reviews, rather than there beings some kind of a fatwa against Likert scales.
I've been a frequent visitor of this site, but have had no need to introduce myself or really take part in any of the discussions (i.e. most of them are silly anyway) so I'm just going to make this comment, and then disappear entirely. This is entirely for your benefit, I have no stake in any one game or developer on here.

If you want to be taken seriously as a community, then stop downgrading your own games, and put an end to the "it's only a hobby" crap that has kept your community from being formally recognized for so long.

I'm not going to say much about myself, only that at where I work your community is usually laughed at because not a single game listed in your database would ever be accepted by an IGN staff member for digital distribution if any of your developers wanted to sell their products.

It's not because the games are crap, they're not. It's not because your community is crap, your community is definitely not that.

Here's the #1 secret to why not ONE game in your database would ever be considered "fit for commercialization"...you're not promoting your games to the general public correctly.

Here's a brief list of a few RM/Indie (all turn-based RPG) games that are doing very well, are commercialized, and that certain staffers would and have taken notice of:

Laxius Force Series
Aveyond Series
Winter Voices Series
Chronicles of a Dark Lord Series
King's Bounty Series
Disciples Series
Spellforce Series

You may loathe every game on that list, you may despise the developers, or you may like them or even love some of those games, but one thing they all have in common is that they have all been professionally reviewed by internet/tv/radio shows like G4TV, Just Press Start, IGN, and so on. Not just once either, but multiple times, and their games have stood up to professional scrutiny.

Not only have they have been professionally reviewed, they were generally (consensus-wise) well-liked by a majority sampling of customer data and sales figures. If you want to be taken seriously outside of the RM community, then start holding yourselves up to the par those developers have set, and exceed them.

Those professional reviews go a long way with businesses and distributors, and that is why they are successful while your games...unfortunately...are not so much. If your only audience is the "RM maker audience" then don't expect to be taken seriously outside of that audience. It's a cold way of thinking yes, but it's the absolute truth, and it does a disservice to those on your site who are thinking or who want to get in on the indie gaming scene as actual game developers.

They were promoted to the public correctly, and now they're all generating a buzz because of the way they were promoted. If you all truly want to be taken seriously as a community, to get your games out there, free or otherwise, then the day you stop seeing yourselves as "this is only a hobby" is the day you'll be taken seriously as game developers. Until then, don't expect to be. Or do what your predecessors have done, and don't be part of this community and distance yourself from it.

Put it this way, there are some great games on this site that could make some good money on the market if they were promoted correctly. Here are a few of the games on this site that could do quite well if promoted correctly in the 'real' business world:

Legionwood: Tale of the Two Swords

Alter A.I.L.A. Genesis

Leo & Leah: A Love Story

Obviously, if you are one of those developers then any and all rips from past games would have to be re-designed with standard/edited RM or original graphics, though games like those would likely do very well if they were put out on the market correctly.

You want your game to be "fit for commercialization"? Then get your game ready with all RM/edited RM graphics/original graphics, RM/edited RM music/royalty-free music/original music and such, then go to an entity like IGN, Game Informer, Just Press Start, and try to have your game reviewed (usually a demo works fine). Then go to any number of digital distributors (i.e. Gamersgate, GoG, Steam), submit your game, and hope it gets accepted. Distributors can be very picky, so even getting on just one distributor is a big deal, it means that they can expect sales from your game because it meets a certain quality of standard. Remember that.

If you have a comment to make, don't make it to me on here, for as soon as I click "logout" you'll never hear from me again more than likely, though as I said in the beginning this is for you all to know. Take this advice, don't take it, up to you.

That diatribe misses a very important point: For some (most?) of us, this is just a hobby. Personally, I couldn't give a shit if IGN would laugh at my games or if they wouldn't be distributed on another site. It's nice when people recognise that my work is good (that sounds like I am saying it is; I don't think it is, but it's nice when other people think that it is), I'm not going to deny that, but that's not why I'm making games. I'm making games because I find that making games is fun.

Still, good advice for those people who do want other people to recognise their work more or who do want to break out of the RM* community (I guess Craze would be the most obvious example of someone with this mindset), in that you have to focus on using open-source or RTP-based resources if people are going to take you "seriously" (what does that even mean?) This goes double if you ever wanted to sell your game since it would be pretty illegal to use anything from a commercial game if you wanted to do that. We already knew that, though, didn't we?

Bah, whatever.
IGN acknowledges us? Awesome!

Also, for many of us (unfortunately?) this is just a hobby, but I suppose of all the RM sites out there this one probably has the best codebase to really showcase the best RM games. And people do hate wasted potential, and I do see some potential here for moving on to bigger and better things.

I get the sense that some people would love to push us in that direction, while others enjoy the relaxed informal setting.

I am quite impressed that a few RM-created games managed to break into the mainstream, and good for them! It is nice to see people with the motivation, ambition and dedication to pull that off.
NB: I was going to edit this in, but I think I will put it as a seperate post instead.

Fundamentally, I have to disagree with that diatribe. It isn't "entirely for our benefit" to try and make all our games at a standard that would be acceptable to websites like IGN. It is not "entirely for our benefit" to avoid using rips of music and graphics from commercial games when we are putting together our projects. And it is definitely not "entirely for our benefit" that we reject games that use rips. We play games because we want to have fun, the only thing that is "entirely for our benefit" is supporting developers in putting together fun experiences, however abstract a concept that may seem to be.

Just think of how many good gaming experiences we have all had when playing games that use rips. The names of these games aren't going to be the same for all of us, but I am fairly sure that we have all played at least one game that contains commerical rips that we thought was an excellent game. Anyone here who hasn't isn't playing enough of the games on the site because it shouldn't take long for people to find a game with rips that they find fun. The obvious example is the one we (or at least I) use in every, single thread that touches on an issue like this, but that doesn't make it any less relevant: Hero's Realm simply wouldn't exist if we took this kind of elitist standpoint and I will always maintain that HR is a better game than most in the RPG world (and I don't just mean that in comparison to non-commerical games, I mean that in an all encompassing sense). More recently, games that I have absolutely loved, like Beautiful Escape: Dungeoneer and Forever's End, simply wouldn't have been hosten by this site if we were to try and push out developers who don't meet this silly "original/open source" standard. So, I have to ask myself, why would I want such a standard if it would rule out some of the most fun games that this community has to offer?

More important than the games we've already come to love, though, is this general concept. A concept that is undeniably true:

Original graphics, music and <whatever else> do not equal better games.

Fun is not determined by the amount of hours a developer put into spriting their characters. It is not determined by the number of non-SquareEnix tracks that are present in their game's music folder. Fun is a much more complex concept than that. Why should we try and deny it? You could argue that any game that is currently good could only be made better with original graphics and music, but I disagree. It wouldn't be a different game, it'd be the same game with slightly different graphics. More eye-candy, but not any more fun...

(This is all completely off-topic, though)
@DemonofElru
RMN doesn't even host commercial games, what makes you think this should be a place for money-making? if people wanted to sell their games this is obviously not the place to do it. i mean what the hell...just because certain people make money off their game doesn't mean we should be laughed at because we don't. as if people call RMN a joke just because it's a hobby to the majority of us. commercial shite is what that is.

i don't have a problem with RMN games going commercial despite it probably looking like i do haha, this just clearly isn't the place for it (right now at least).

but yeah i guess good advice for people who want to break into the industry via RM.
i think you were mislead by the title of this topic though haha.
author=somedudeoutofnowhere
If you want to be taken seriously as a community, then stop downgrading your own games, and put an end to the "it's only a hobby" crap that has kept your community from being formally recognized for so long.


but this is just a hobby
I am going to be reviewing games on here.

I intend to have them reviewed in a very simple way: Did I like it?

If yes, they will be scored well based on how much I enjoyed em! If not, then I will score them low and explain WHY I didn't enjoy em.

See how simple that is? Straight, honest, to the point!