Also, he pretty much implies that he had met Link before, also at the beggining of the game, when Epona knocks Link down, and he falls, Skull Kid says this to Tatl and Tael: "That guy...?Hm... Should be no problem at all".
It is pretty much implied that it is the very same skull kid from OoT...