If you take the shrine quest Eye of the Sandstorm from Kara Kara Bazaar, finding that shrine will put you practically right next to the tower. You just gotta climb up to the west from there and it’s in a canyon. It’s surrounded by cliffs so it’s easy to miss.