I actually never noticed that, but it is true. The medallion does resemble yin and yang.
I think the temple signifies childhood and adulthood. At the beginning of the temple you have to be a child in order to get the gauntlets because you need them to move the stone to get to the bosses...