I always thought it was some kind of glass, concidering how easy they are to smash and how they explode in tiny little pieces. The reason I always thought this was because of the shooting range in OoT with the slingshot and bow. However, it would be very logical if that dude used fake rupees, otherwise his business would go bankrupt after a few visits, considering it only costs you 20 rupees to play
Since the LoZ universe doesn't actually exist. I always have believed that rupees are made of a material that could not exist or has not been discovered yet. But we can determine that the material that rupees are made out of is very breakable. But we can't necessarily call it a crystal or glass. Since the color of a rupee seems opaque, meaning the color of a rupee may be dyed or painted.