Full Games List
Featured Games
New Games
Reviews
Media
Images
Blogs
Game Updates
Images Feedback Summary
Game Design Highlights
Tutorials
Articles
Scripts
Utilities / Plugins
Resources
Engines
Engine Bulletins
What is Makerscore?
Past Events
Achievements
Misaos
Forums
Post History
Chat
New to RMN?
Full Store List
RMN Music Pack (free)
OPINION: STOP RATING DEMOS, IT'S UNFAIR...TO COMPLETED GAMES
Max McGeePosts 
Corfaisus
"It's frustrating because - as much as Corf is otherwise an irredeemable person - his 2k/3 mapping is on point." ~ psy_wombats
7874
View playlists
CloseBut what about cancelled games?
Avee
5794
While I understand the concern behind this article, I disagree with the point of view.
If a game is 50% completed and has a download, that means that the second half of the game is vaporware. There is no guarantee that the game will actually be completed and that a new download will be released.
Therefore I must consider the available download as being 100% of the game so far. It is 100% of what's available, what exists.
This is still true if you release a tech demo that features 5 minutes of gameplay and nothing else is ever added to it. That tech demo is a finished, completed project.
Now if both a demo and the completed game are released and available, I believe that the demo should not receive any review or rating because it's an incomplete product at this point.
I must also add that I am not fully in favor of ratings because of how subjective they are to both the reviewer and the reader. The review itself, what you write in it, has infinitely more informative and objective potential than the rating.
I would rather encourage better reviews for any kind of download.
If a game is 50% completed and has a download, that means that the second half of the game is vaporware. There is no guarantee that the game will actually be completed and that a new download will be released.
Therefore I must consider the available download as being 100% of the game so far. It is 100% of what's available, what exists.
This is still true if you release a tech demo that features 5 minutes of gameplay and nothing else is ever added to it. That tech demo is a finished, completed project.
Now if both a demo and the completed game are released and available, I believe that the demo should not receive any review or rating because it's an incomplete product at this point.
I must also add that I am not fully in favor of ratings because of how subjective they are to both the reviewer and the reader. The review itself, what you write in it, has infinitely more informative and objective potential than the rating.
I would rather encourage better reviews for any kind of download.
Does this mean that shorter games will get better reviews than longer ones? Maybe it's been like that in your case, but often a shorter game does not automatically get better reviews. I don't know, but I'm willing to overlook a few flaws if the overall package is good. I don't rate a game based on a benchmark score minus its flaws, I rather rate it on how good I thought it was. Often I'll ignore flaws in a long game when scoring games, just because the overall experience was good. In fact someone who has created a longer game shows me that they've put more effort into making it good, usually.
My beliefs on this subject are, quite frankly, this.
Honestly, there are quite a few game demos who are 5, 10 times longer than, well, quite a few completed games!
I absolutely second everything that Avee and Cash said, as well. But I don't dismiss what you said, it's actually something important to think about. But just proper judgement from the reviewers is usually more than enough.
But being a little more restrictive about what reviews are accepted and what are not could be interesting. Emphasis on the could.
I absolutely second everything that Avee and Cash said, as well. But I don't dismiss what you said, it's actually something important to think about. But just proper judgement from the reviewers is usually more than enough.
But being a little more restrictive about what reviews are accepted and what are not could be interesting. Emphasis on the could.
Addit
6394
So does this mean now that you’re going to go back to all your previous demo reviews, Maxy, and change them all to the “not rated” status after writing all this? Hmm…
Anyways, opinion noted. I guess it does seem a bit silly to some people on why some reviewers, like yours truly, gives scores to demo versions of games in sort of the same vein as actual completed games, but I only do it because, one, that person gets at least some makerscore for their cause, even if they end up never completing the game or cancelling it in the end, and, B, because I still believe personally that giving a score to demos is still a viable rating scale based on whether or not I’d be stoked to play the full version of that game sometime in the future if it ever gets completed. Every reviewer does it a bit of a different way and what not when it comes to this.
And in any sense, I usually try to stay away from reviewing demos and focus more on completed versions of games unless that person wants any feedback, is for a contest, or hasn’t had a major update in years and has only that one demo build to speak of, or it got cancelled.
Anyways, opinion noted. I guess it does seem a bit silly to some people on why some reviewers, like yours truly, gives scores to demo versions of games in sort of the same vein as actual completed games, but I only do it because, one, that person gets at least some makerscore for their cause, even if they end up never completing the game or cancelling it in the end, and, B, because I still believe personally that giving a score to demos is still a viable rating scale based on whether or not I’d be stoked to play the full version of that game sometime in the future if it ever gets completed. Every reviewer does it a bit of a different way and what not when it comes to this.
And in any sense, I usually try to stay away from reviewing demos and focus more on completed versions of games unless that person wants any feedback, is for a contest, or hasn’t had a major update in years and has only that one demo build to speak of, or it got cancelled.
So does this mean now that you’re going to go back to all your previous demo reviews, Maxy, and change them all to the “not rated” status after writing all this?
It honestly depends how much traction this idea gets.
At present, I'm not planning on it.
author=Max McGeeOops, I hope I haven't started a landslide.
It honestly depends how much traction this idea gets.
At present, I'm not planning on it.
As far as revising previous reviews. I'd only do so if you go back and find that a given game has a "more definitive" version than what you played. If the version you reviewed is still the version being downloaded, no edits are necessary. You're explaining whether or not that demo is worth the player's time.
I think my irritation here had less to do with the principle of reviewing demos than it did with the idea that I hadn't updated the download fast enough to excise the faults mentioned before it was reviewed.
It wasn't Essence Enforcer specifically. It's actually not ANYTHING specifically. I really was just going back and thinking about all of the demos and completed games I have ever played and reviewed in the entire like eternity I've been around.
MercynMetal
132
Isn't it possible to create a review without giving any stars? (I've never done a review myself, so I'm not certain, but...) I think I've read reviews that didn't award stars, without bringing down the game's rating. Just seemed to "skip" that option. So if it's not already in existence, maybe it would be fair to allow reviewers to "skip" the star rating.
Yes, that is presently possible. To clarify, I am currently suggesting it be mandatory for incomplete games.
Addit
6394
You know, thinking about it some more, maybe we should all put this to a community vote in the future and see whether or not to decide to change the rules or keep them currently in place. (Maybe you can talk to Kentona about possibly doing that?) I really don’t mind either way; I just don’t want to suddenly start changing demo scores to games that I’ve already reviewed in the past and have those said people suddenly come out of the woodwork and complain about “how come my makerscore is suddenly so low” type of thing. I mean, I don’t want to be THAT evil.
You know, thinking about it some more, maybe we should all put this to a community vote in the future and see whether or not to decide to change the rules or keep them currently in place.
This seems like the logical result of this topic, if this topic gets enough traction to seem like the vote would have any chance of passing. Which is a big if. We're definitely not there yet.
(As kentona can totally see this topic, I don't think I specifically need to call it to his attention.)
VideoWizard
2842
Idea: Keep it the 0.5/1.0/1.5/2.0 etc out of 5 stars but only the creator of the game and maybe co-creators or testers can actually see the score. Everyone else gets Not Rated/5 on their screen. The game still gets full makerscore and no one loses makerscore as a result.
Would that work?
Would that work?
Hasvers
1298
One question: how am I supposed to proceed if I want to find a good demo to play, then?
I am kind of glad that, say, A Blurred Line is brought to the general attention through its high rating, even though it isn't complete and never will be.
I think if this is ever implemented as a rule, it should be left as an option for the author of the game. But more generally, any rating system that is supposed to hold over very heterogeneous items (different genres, wildly different lengths and team sizes) is going to be unfair and there is very little we can do about it.
(I have to say though that your reasoning is especially original on one point: so far, most of the people I've seen ask not to get ratings on their demos did so because they expect the complete game would have a higher rating. Someone once gave this excuse to remove a game from this site entirely after their demo got reviewed.)
I am kind of glad that, say, A Blurred Line is brought to the general attention through its high rating, even though it isn't complete and never will be.
I think if this is ever implemented as a rule, it should be left as an option for the author of the game. But more generally, any rating system that is supposed to hold over very heterogeneous items (different genres, wildly different lengths and team sizes) is going to be unfair and there is very little we can do about it.
(I have to say though that your reasoning is especially original on one point: so far, most of the people I've seen ask not to get ratings on their demos did so because they expect the complete game would have a higher rating. Someone once gave this excuse to remove a game from this site entirely after their demo got reviewed.)
One question: how am I supposed to proceed if I want to find a good demo to play, then?
I did actually invest literally hundreds of words into answering this question in the original article. Short version: use thine eyes.
Hasvers
1298
Well not really, no. On the contrary, you argue that incomplete games do not require your first function of reviews, which is to be recommended to people, and I am unconvinced, since I may want to find good games and play them even though they are not complete.
Taking this to a not-too-serious extreme, no one should ever give a rating to the Chronicles of Amber because Zelazny died before he finished the series :P
(De-edited)
Taking this to a not-too-serious extreme, no one should ever give a rating to the Chronicles of Amber because Zelazny died before he finished the series :P
(De-edited)
On the contrary, you argue that incomplete games do not require your first function of reviews, which is to be recommended to people, and I am unconvinced, since I may want to find good games and play them even though they are not complete.
But doesn't looking at screenshots and reading game descriptions and reading comments and reading reviews (SANS SCORES) help you find (incomplete) games you might be interested in?
To push this to a not-roo-serious extreme, no one should ever give a rating to the Chronicles of Amber because Zelazny died before he finished the series.
That's only true if you use like an insanely narrow and almost impossible to achieve definition of 'finished'. Of course the Chronicles of Amber should be reviewed. It's comprised of ten finished novels!
I agree with you that A Blurred Line is an interesting edge case. And honestly it sort of spills into a larger issue which while a semantic issue is nonetheless really important...we use 'demo' in a completely different meaning than the entire greater gaming industry. The 3-5 hour long (IIRC) latest release of ABL really would not be described as a "demo" anywhere outside the microcosm of this community.
Hasvers
1298
Disclaimer: Sorry, I have the nasty habit of editing my post right after posting it, because I tend to write things too fast for my own good.
I have reverted my post above and added the details here.
I would say mainly two things:
- it is especially important to help people notice a very promising demo and thus give feedback to its author.
- if unstarred reviews are good enough to help an audience find good demos, then they are good enough to find good games too and stars are entirely irrelevant, complete game or not.
I admit that the second point can be mitigated a bit since people willing to give feedback are more likely to make the effort of at least browsing the reviews. But the problem remains that the mechanisms that make high-rated games easy to find would also be useful to find demos with a lot of potential.
I have reverted my post above and added the details here.
I would say mainly two things:
- it is especially important to help people notice a very promising demo and thus give feedback to its author.
- if unstarred reviews are good enough to help an audience find good demos, then they are good enough to find good games too and stars are entirely irrelevant, complete game or not.
I admit that the second point can be mitigated a bit since people willing to give feedback are more likely to make the effort of at least browsing the reviews. But the problem remains that the mechanisms that make high-rated games easy to find would also be useful to find demos with a lot of potential.
