The stars might be billions of years old but the constellations change over time. Not over billions of years but thousands.
This shows the transformation of the big dipper. How it would have looked 50,000 years ago, today and how it will look in 50,000 years time. If Hyrule is on earth it is...
If it were to happen I would prefer a Hyrule Warriors style situation where portals are opened into different realities and the Zelda characters come through into our own world rather than them being part of it originally.
I don't subscribe to the idea that Hyrule is in our world in the past...