Colors aren't elemental, they're stat-based. Green is HP, Blue is SP, Attack is Red, Defense Yellow.
Skills "take over" their opposite colors, so if you used "Fire" most of the blue gems would become green.
You use the gems to increase your stats (HP, SP, ATK, DEF), you think of them like Potions, Ethers, and other buff items. The purple increase your purple bar which, once full gives you one of your skills (based on class and a basic set.)