I believe that the forest that brought Link and the Skull Kid together in Termina was a portal to an alternate world. At the end of MM, the Skull Kid smells Link and tells Link that he smells like the kid who taught him 'that song' in the woods. If you recall, in OOT you teach one of the Skull Kids Saria's Song in OOT's Lost Woods.
I think that destiny brought the Mask Salesman, Link, and the Skull Kid together... They are the same people from OOT, but they got into the other world because they were all wandering through the forest that pitted them in Termina.
The reason the Mask Salesman doesn't directly mention to Link that he remembers him from Hyrule was because of the kind of mysterious personality he has. And I'm sure Link didn't say anything either because, well, we all know he doesn't talk too much :p