Gems are used for Powering-up Necklaces or Anklets. They can be bought in the shop under the Academy tab for 1000 GP each. They can also be used to feed pets.

Another use for them are for unlocking certain characters with their cash mission or other quests.