Did you know that there are only a few gemstones that are classified as organic gemstones?
These gems are organic materials used as gems. So, basically, these became precious because we looked at them and said "Pretty! These must be valuable." And now, items like ivory in particular are precious, because of the history of how ivory was obtained.
But there are other kinds of gems that are technically "organic" - these mainly are rocks... But that is for another post. 
