OPINION: STOP RATING DEMOS, IT'S UNFAIR...TO COMPLETED GAMES

Completed Games Have More Opportunity To Make Mistakes

So, today I had a minor revelation in the shower, as I often do. If I play a one hour long demo, and it's really good, I am likely to review it with a high rating based on the content I've seen.

If on the other hand, I play an eight hour long completed game, that game has seven more hours of gameplay in which to screw up. Seven more hours of "rope" with which to hang itself. Seven more hours of content with which to find fault with the gameplay, story, characters, or graphics. Seven more hours to make a design decision that will piss me off. And as a result of its expanded content and additional opportunity to make mistakes, it will very probably get a lower score, every time, than an equivalent demo.

And that is not as it should be.

Surely, by now it is long-codified conventional wisdom that a game must put its best foot forward. Games in this community are typically frontloaded with their best and most polished content, because of the truism that if your game doesn't hook players in the first fifteen minutes, it won't get a chance to at all. The inverse of this is logically obvious. Maintaining a high level of quality and polish during a feature-length game is much, much harder than maintaining it for a short demo.

How does one counteract this obvious inequity? Obviously, you could add one full star for completed games, or subtract one full star for game demos to try and offset this imbalance. But doing so would clearly skew the entire numerical component of the review rubric.

A better solution is not to allow starred reviews for incomplete games (demos) at all.

Nota Bene: Obviously, until this point, I have been a part of this problem. Because I have attached starred reviews to numerous demos. I have no defense for this except to say that I had not come to this conclusion presented here--which I do believe is the correct conclusion--until just now.

But Max, a reasonable person interrupts, creators of games need feedback on their demos. LOTS of feedback!. And of course, being a creator of games myself, I 100% agree. And here we arrive at an opportunity to restate the crux of the issue:

Game Reviews on this site uneasily serve two masters. They exist simultaneously for two separate purposes:

1) To publicly evaluate the merits of a game and recommend it (or not) to an audience of potential players. This function is served primarily by the review's rating, and then by it's text.

2) To provide feedback, critique, evaluation and suggestions to the game's developer. This function is served primarily by the review's text, and then by it's rating.


Demos require the second function of a review. I would argue that they do NOT require the first function of reviews (evaluating a game for the benefit of potential players). And therefore, they do not require STARRED reviews.

The reason I would argue this is that even without starred reviews, potential players have ample means to evaluate a demo they are considering playing. Consider:

  • Demos are shorter than full games, and are less of a time investment. Evaluation is therefore de facto less of a concern than for a complete game.
  • The quality of a demo can be evaluated by its screenshots and its game description.
  • The quality of a demo can be further evaluated by its user comments.
  • The demo itself is only a tool for evaluating/advertisement for the completed game, not a product that requires evaluation.
  • Finally, "Not Rated" reviews provide both an avenue of direct feedback to the game developer and yet another option for potential players to use in evaluating a demo before deciding to play it.


In conclusion, going forward, reviews should be allowed for game demos, but should not be allowed to have official star ratings that are aggregated and tracked on the site. Official star ratings should be reserved for completed games.

Reviewers should ABSOLUTELY be allowed to informally include star ratings for game demos in the text of their review if they desire. The stars can even be inserted as an image in the body of the text. But the review score should not be officially aggregated or tracked by the site's mechanics for reviews of incomplete games.

Reviews--with or without informal star ratings--of incomplete games (demos) should be called and handled as Critiques.

The Problem Of Implementation

Toggling off the option to officially append a starred rating for future reviews of game demos going forward should be easy. But what about the thousands of starred reviews of demos already on the site? Those present a greater challenge. I am not technically skilled and I have no real understanding of the site's codebase. But it seems to me it should be possible to use a script to find every review with a star rating attached to a game with a status other than Complete, and change all of those ratings to Not Rated. And the text of the reviews would not need to be altered at all, which is a good thing, because that could only be done manually. After all, including an informal rating or score in the body of a critique is fine, as long as the score is not tracked on the site.

The sitewide change that would happen if this were implemented is that the official score of every single game without a status of complete would change to Not Rated. Which makes actually perfect sense. IGN, Kotaku, GiantBomb, GameInformer and so on and so on and so on don't rate games before they are done, so this seems perfectly logical.

One more change would need to be made to the site's infrastructure to accommodate this, and that is in the Search Filter on the Games page tab. The Minimum Rating dropdown would have to be hidden unless Completed was selected from the Status dropdown.

Now ... sadly, I do not expect that there's terribly high chances of this positive change actually being implemented. That goes with the territory of writing an op-ed piece in any environment. Your opinion does not become law just because you've put it in writing. And besides the various and sundry objections to this suggestion I can't yet anticipate, the mere idea of making any major change to the site at all can be a difficult one to gain traction for. Change is inherently inconvenient and difficult and sometimes even scary. I am the guy who freaked out over signatures being removed six years ago, so I know.

But I look forward to the discussion this will generate, and I embrace the possibility of being pleasantly surprised by the way this idea is received.

Posts

Pages: first 123 next last
Corfaisus
"It's frustrating because - as much as Corf is otherwise an irredeemable person - his 2k/3 mapping is on point." ~ psy_wombats
7874
But what about cancelled games?
While I understand the concern behind this article, I disagree with the point of view.

If a game is 50% completed and has a download, that means that the second half of the game is vaporware. There is no guarantee that the game will actually be completed and that a new download will be released.
Therefore I must consider the available download as being 100% of the game so far. It is 100% of what's available, what exists.
This is still true if you release a tech demo that features 5 minutes of gameplay and nothing else is ever added to it. That tech demo is a finished, completed project.

Now if both a demo and the completed game are released and available, I believe that the demo should not receive any review or rating because it's an incomplete product at this point.

I must also add that I am not fully in favor of ratings because of how subjective they are to both the reviewer and the reader. The review itself, what you write in it, has infinitely more informative and objective potential than the rating.
I would rather encourage better reviews for any kind of download.
CashmereCat
Self-proclaimed Puzzle Snob
11638
Does this mean that shorter games will get better reviews than longer ones? Maybe it's been like that in your case, but often a shorter game does not automatically get better reviews. I don't know, but I'm willing to overlook a few flaws if the overall package is good. I don't rate a game based on a benchmark score minus its flaws, I rather rate it on how good I thought it was. Often I'll ignore flaws in a long game when scoring games, just because the overall experience was good. In fact someone who has created a longer game shows me that they've put more effort into making it good, usually.
Marrend
Guardian of the Description Thread
21781
My beliefs on this subject are, quite frankly, this.
Honestly, there are quite a few game demos who are 5, 10 times longer than, well, quite a few completed games!
I absolutely second everything that Avee and Cash said, as well. But I don't dismiss what you said, it's actually something important to think about. But just proper judgement from the reviewers is usually more than enough.
But being a little more restrictive about what reviews are accepted and what are not could be interesting. Emphasis on the could.
So does this mean now that you’re going to go back to all your previous demo reviews, Maxy, and change them all to the “not rated” status after writing all this? Hmm…

Anyways, opinion noted. I guess it does seem a bit silly to some people on why some reviewers, like yours truly, gives scores to demo versions of games in sort of the same vein as actual completed games, but I only do it because, one, that person gets at least some makerscore for their cause, even if they end up never completing the game or cancelling it in the end, and, B, because I still believe personally that giving a score to demos is still a viable rating scale based on whether or not I’d be stoked to play the full version of that game sometime in the future if it ever gets completed. Every reviewer does it a bit of a different way and what not when it comes to this.

And in any sense, I usually try to stay away from reviewing demos and focus more on completed versions of games unless that person wants any feedback, is for a contest, or hasn’t had a major update in years and has only that one demo build to speak of, or it got cancelled.
Max McGee
with sorrow down past the fence
9159
So does this mean now that you’re going to go back to all your previous demo reviews, Maxy, and change them all to the “not rated” status after writing all this?


It honestly depends how much traction this idea gets.

At present, I'm not planning on it.
LouisCyphre
can't make a bad game if you don't finish any games
4523
author=Max McGee
It honestly depends how much traction this idea gets.

At present, I'm not planning on it.
Oops, I hope I haven't started a landslide.

As far as revising previous reviews. I'd only do so if you go back and find that a given game has a "more definitive" version than what you played. If the version you reviewed is still the version being downloaded, no edits are necessary. You're explaining whether or not that demo is worth the player's time.

I think my irritation here had less to do with the principle of reviewing demos than it did with the idea that I hadn't updated the download fast enough to excise the faults mentioned before it was reviewed.
Max McGee
with sorrow down past the fence
9159
It wasn't Essence Enforcer specifically. It's actually not ANYTHING specifically. I really was just going back and thinking about all of the demos and completed games I have ever played and reviewed in the entire like eternity I've been around.
Isn't it possible to create a review without giving any stars? (I've never done a review myself, so I'm not certain, but...) I think I've read reviews that didn't award stars, without bringing down the game's rating. Just seemed to "skip" that option. So if it's not already in existence, maybe it would be fair to allow reviewers to "skip" the star rating.
Max McGee
with sorrow down past the fence
9159
Yes, that is presently possible. To clarify, I am currently suggesting it be mandatory for incomplete games.
Good to know, thanks. By the way, I agree with you 100%.
You know, thinking about it some more, maybe we should all put this to a community vote in the future and see whether or not to decide to change the rules or keep them currently in place. (Maybe you can talk to Kentona about possibly doing that?) I really don’t mind either way; I just don’t want to suddenly start changing demo scores to games that I’ve already reviewed in the past and have those said people suddenly come out of the woodwork and complain about “how come my makerscore is suddenly so low” type of thing. I mean, I don’t want to be THAT evil.
Max McGee
with sorrow down past the fence
9159
You know, thinking about it some more, maybe we should all put this to a community vote in the future and see whether or not to decide to change the rules or keep them currently in place.


This seems like the logical result of this topic, if this topic gets enough traction to seem like the vote would have any chance of passing. Which is a big if. We're definitely not there yet.

(As kentona can totally see this topic, I don't think I specifically need to call it to his attention.)
Idea: Keep it the 0.5/1.0/1.5/2.0 etc out of 5 stars but only the creator of the game and maybe co-creators or testers can actually see the score. Everyone else gets Not Rated/5 on their screen. The game still gets full makerscore and no one loses makerscore as a result.

Would that work?
One question: how am I supposed to proceed if I want to find a good demo to play, then?

I am kind of glad that, say, A Blurred Line is brought to the general attention through its high rating, even though it isn't complete and never will be.

I think if this is ever implemented as a rule, it should be left as an option for the author of the game. But more generally, any rating system that is supposed to hold over very heterogeneous items (different genres, wildly different lengths and team sizes) is going to be unfair and there is very little we can do about it.

(I have to say though that your reasoning is especially original on one point: so far, most of the people I've seen ask not to get ratings on their demos did so because they expect the complete game would have a higher rating. Someone once gave this excuse to remove a game from this site entirely after their demo got reviewed.)
Max McGee
with sorrow down past the fence
9159
One question: how am I supposed to proceed if I want to find a good demo to play, then?


I did actually invest literally hundreds of words into answering this question in the original article. Short version: use thine eyes.
Well not really, no. On the contrary, you argue that incomplete games do not require your first function of reviews, which is to be recommended to people, and I am unconvinced, since I may want to find good games and play them even though they are not complete.

Taking this to a not-too-serious extreme, no one should ever give a rating to the Chronicles of Amber because Zelazny died before he finished the series :P

(De-edited)
Max McGee
with sorrow down past the fence
9159
On the contrary, you argue that incomplete games do not require your first function of reviews, which is to be recommended to people, and I am unconvinced, since I may want to find good games and play them even though they are not complete.


But doesn't looking at screenshots and reading game descriptions and reading comments and reading reviews (SANS SCORES) help you find (incomplete) games you might be interested in?

To push this to a not-roo-serious extreme, no one should ever give a rating to the Chronicles of Amber because Zelazny died before he finished the series.


That's only true if you use like an insanely narrow and almost impossible to achieve definition of 'finished'. Of course the Chronicles of Amber should be reviewed. It's comprised of ten finished novels!

I agree with you that A Blurred Line is an interesting edge case. And honestly it sort of spills into a larger issue which while a semantic issue is nonetheless really important...we use 'demo' in a completely different meaning than the entire greater gaming industry. The 3-5 hour long (IIRC) latest release of ABL really would not be described as a "demo" anywhere outside the microcosm of this community.
Disclaimer: Sorry, I have the nasty habit of editing my post right after posting it, because I tend to write things too fast for my own good.

I have reverted my post above and added the details here.

I would say mainly two things:

- it is especially important to help people notice a very promising demo and thus give feedback to its author.

- if unstarred reviews are good enough to help an audience find good demos, then they are good enough to find good games too and stars are entirely irrelevant, complete game or not.

I admit that the second point can be mitigated a bit since people willing to give feedback are more likely to make the effort of at least browsing the reviews. But the problem remains that the mechanisms that make high-rated games easy to find would also be useful to find demos with a lot of potential.
Pages: first 123 next last