If there wasn't a timer even replaying floors wouldn't be so horrible. But that you did have to keep repeating floors over-and-over, especially if you've not afforded yourself enough time in later floors (once you have the save point) just made the whole process tedious.
EDT:... I recognise that comes across as just hating on it for the one element; time. This wasn't, however, my only beef with it, and there are other elements that make it more annoying.
I mean don't get me wrong, a race is fine to run. Bettering one's time for a great reward isn't a bad thing. There are timed races in this and the next game that I had no issues with. It's fulfilling when you can charge through and find you've done better than last, and gotten more for repeating it.
The problem is in the difference. This race wasn't optional. And you were running it more than once for your 'reward'. You're also never told it's a race, so you're just thrown at it with the stick instead of the carrot ("Do this or you'll be poisoned and die!").
Much as I know this'll net me some negative points among many, I will admit that I did 'turn off' the timer. Led to more time to explore to find things you wouldn't on earlier runs, and meant getting through the parts of the dungeon much faster than when I was told I'd die if I wasn't fast! That very timer slowed me down immensely, and its removal sped me up. It did mean I could get through the puzzles you had to do over and over, to get me to the less annoying siren enemies and not-always-accurate drawing puzzles. But it wasn't as grating as the timer forced me to be. I get quite agitated at games as it is, and extra reasons just put me off.
Long story short, the timer just exacerbated other annoying issues within the temple and could have been left out, and still made for a decent gaming experience. Not like there weren't other irritating elements in that game that would have made up for that loss (ship battles).