I've always thought this too. Part of why I like that Link was sent back to reclaim his childhood is that he doesn't really do it. It seems like most of his childhood is spent re-saving the world of Termina. Navi, his one actual friend who isn't dead, and a representation of his childhood...