GATHERING DATA DURING GAMEPLAY. YES OR NO?

I've had a really crazy idea lately (or maybe not so crazy, we'll get to that) concerning gathering data during test plays. More and more, I've come to understand that people will not play the game the same way I did, and more often then not they will not even play it the other ways I intended for it to be played. They will, without a doubt, play it however the hell they damn well want to. That isn't a problem, in fact that is kind of the goal here. It isn't that I don't trust people to give me honest feedback, that hasn't been the problem so far... I just think that there are some things I may need to know that the player might not even be aware of, or might not think important enough to mention.

To solve that, I'm considering putting some data gathering events into my next download. I would have it take readings such as how many healing items were used, how many turns it took the player to defeat certain enemies, whether or not they were using the best equipment at a given time, how long it took them to resolve "mystery", "fetch" quests as opposed to how long they solved dexterity/spacial relationship puzzles, or even just how long it took them to advance the story in certain points, as well as how many optional tasks they take on, and how many npcs they talk to, etc...

Using pointers, I can chew up a ton of variables for this sub-system without actually effecting the data used for the game itself. I could write a second "game" that would just strip my data back out of the >5000 range and crunch the numbers for me.

Of course, I would have to ask for people to send me their clear file/last save file in order to analyze their data. Since I'm almost certain some people would object to having a bot reading over their shoulder and judging them for every decision they make, I would most likely provide the option to turn the Data Collection module off at the start of a new game.

Provided the above all goes off without a hitch, the next question would be "What do I do with all of this data?"

Clearly, I can't go around telling people how to play the game. "You wasted twelve medkits on a fight you could have won in three turns if you used the right skills, what the hell is wrong with you man!?"

Instead, I could use the data to figure out what parts of the game people just aren't getting, so that I could explain them better within the game, or tailor the experience in such a way as to gently persuade people to experiment with things they might have otherwise ignored.

I feel that as a game designer, you should never really use phrases like "No, you're playing my game wrong," or "Stick with it, it gets better later."

I know that "real" game developers keep tabs on how their games are played: They track player tendencies through achievements, and so forth. That gives them automatic feedback about how their games are being played, but only to a point. When someone KNOWS they get points for maxing their achievements out, its use as a fair metric is diminished. "Well, everyone saves the pilot, but they also let her die so they can complete no kill runs, which is weird because they also went for the X kills with a Y weapon trophies. Oh well. The world is schizophrenic, but the game is perfect, lets move on..."

Now we come to the ugly end of the spectrum. How would I get people to be a part of my data gathering plan for world domination(or at the very least minor game-suck removal services)? Do I go with a carrot or a stick?

Dumb question deserves a dumb answer: I need scissors, spoonman!

Of course I'd go with the carrot approach, but what kind of carrot will it have to be to get people to take a bite?

I'm sure there are some people who would be happy to send the files over to me for analysis without any other motivation than just to help me make a better experience for them and everyone else, but that isn't quite good enough for me. I don't want a handful of people, that is going to happen anyway during testing (hopefully, or at least in an ideal setup), I want every single person who plays the game, within reason of course.

Would a simple leader board page on the game's profile be enough to garner people's consent to data gathering? If so, how would I structure it to keep people from just maxing their stats out and ruining my data? Well, for starters, instead of things like "x kills with y weapon" or "you pushed start, congratulations!" the achievements could be for "Most Efficient Fighter."

That would be something I could derive and track pretty easily. "Best Sleuth" could rank players on how long it took them to track things down in the game. Would internet bragging rights alone be enough? Or would I have to start embedding cheat codes so I can release them to whoever is at the top of a given leader board. All hubris aside, I'm honestly not sure anyone would actually give a damn about achievements in indie games like this, let alone be invested enough to replay the damn thing with some little addition or secret character unlock - to say nothing of how much adding in all this superfluous content would cut into time that could be spent working on the game's base experience. (I remember posting a screen shot with my clear score back when Hima released Hello Panda on GamingW, and feeling a little jolt of pride that I actually managed to surprise the game's creator with my high score) There are a lot of possibilities here, but also the same problem that "real" game developers face.

Let me introduce you to my friend, Mr.Heisenberg. No, I'm not talking about Walter White. I'm talking about the fact that by measuring (and by extension, rewarding) people's various tracks through my game, I could effect the way the game was being played, potentially ruining the integrity of my whole data gathering experiment.

Transparency is the enemy, but as I stated before, I don't want to make my data grabs surreptitiously. I did come up with a method of doing so that I could use to track things in eventual reviews, should people feel so inclined as to write them, but I didn't implement it... again, it isn't anything I'd appreciate having done to me, so there you go.

Well, I've rambled enough here. The quick recap is: I have an idea for gathering data about how my games are played, I have an idea for how to get people interested in participating, and I even have ideas about how to be evil during the whole process.

What do you think?

author=Sauce
Sounds like a pretty cool idea. Also sounds really hard parsing those numbers for usable info. You got that figured out? Could be a lot of work for what could amount to something like "lower hi-potion from 100hp to 75hp."

I have some ideas, but it all really depends on the data I get. If I see that people are taking a long time to locate one objective, I know that I might need to add in some extra hints, via dialog and/or map design. If I see that people are hoarding their healing items instead of using them, that would be an indication to me to make them more common, and also possibly decrease the effect of some of the 'better' items to increase the chance that they get used instead of stockpiled for a rainy day.

I'd have to compare data and try to establish the values for a base line experience first. From there I could start to analyze trends, deviations, tendencies, etc.

Of course none of this would occur in a vacuum. Naturally, I would still want testers to give me their feedback. The difference is that I could see exactly (well, for the elements I'd be capturing data on) how they played the game so I could evaluate what they told me in the context of how they played. "X boss was next to impossible." + <Data shows that the player took a party in for the fight that didn't have a good mix of skills the boss was weak against/Player took a team of strikers in, but no healers, buffers, or debilitators.>

Also, I'm working on a way to capture data that will give me an idea of how long the game is played between saves, along with which area the player is navigating. That way I might be able to tell if an area is frustrating or boring if I see that it took the player six sessions to actually make it through.

I'm actually considering doing away with "game overs", just from the standpoint that capturing data during a loss is of equal (or perhaps greater) importance than capturing for a victory. After losing a battle, the player gets the option to just continue from their last auto-save so that the data from the loss is preserved.

Spoiler alert - I'm also toying with how to implement auto-saves via pointers & the >5000 variable range. They would occur just before each battle (that way I'd have less to have to track & reset), and if the player loses, they get the option to recall straight to the menu/character selection (with a slight xp penalty) or jump right back into the battle like it never happened.