The chamber of sages is situated within the temple of light which is situated within the sacred realm. Which is kind of another universe. Rauru built the Temple of Time to connect to this other place which is not accessible otherwise.
The chamber is the last stronghold within the sacred realm...