Gorons are an extremely durable race. Take WW for example; they survived the Great Flood without a need to change physiologically even when both the Kokiri and Zora had to "evolve" to survive the new geography. They have appeared in virtually every land Link has arrived at, showing that they have a great tenacity to survive in a variety of locations. It is also suggested that Gorons are an ancient race due to their appearance in the inkblot opening animation of the game alongside the Goddess and the other races. Given their ability to survive without the need to adapt, it makes perfect sense to consider them one of those "fossil species" that endures for great periods of time without changing in appearance or anatomy. Like any race, their numbers probably changed depending on circumstance, so SS may have taken place during a low point in their numbers, or they may have lived in an area Link never entered.