Oh, forgot to mention this: different party members and enemies gain the favor of the crowd at different rates. Characters like Cid and Parker, known to be more personable, identify with the crowd easier than others, for example. They'll gain more favor with each attack than, say, a character like Beclem, or Reeve.
Some enemies you'll face in the Coliseum are "regulars" and will gain favor much, much faster than others. Typhon comes to mind.