I don't think so. I'm pretty sure it was when Link fell down the giant abyss while chasing the Skull Kid, and that's when he entered Termina. Termina is supposed to be some sort of different dimension, and I think it makes the most sense that it was the abyss that brought him there.
Edit...