He's not heading towards the village. He is heading to Snowhead Temple to stop the curse in place of Darmani.
If he was heading for the village he also wouldn't say this:
"My son misses me... Ulp! Forgive me, my child! Your father has work to do! Darmani... Be you a ghost or a figment of my...
He's inside the biggest one.
When you leave the Goron village area you must make a right off the first island .The snowball that Tatl shows as green will be the one that Goron Elder is. You'll quickly be able to tell if you look hard enough. Also he never changes where he is. He is always at...