Right, he has Zelda to chase after. That's exactly why he shouldn't play with Link; merely get down to business and swiftly return to his task at hand. But he didn't, and that's what got him beaten in the end (though he gets props for carrying his plan out regardless)
- He specifically mentions Zelda's presence being gone during Skyview, so he's got time.
- Impa decided to be a hoe in the Earth Temple.
- Zelda escapes during the Temple of Time cutscene.
At this point in the story, Zelda is god-knows-where to Ghirahim, so it's more "Busy, gotta think this through" than "Time is of the essence".
- Couldn't give two damns about Link during the Ancient Cistern, he just poofs after introducing Koloktos.
- Tries to strike a deal with Link at the Fire Sanctuary.
All his acts are pretty justified. At this point, he does a bit of backtracking because it was quite obvious the Gate of Time was in Faron Woods, and lo' and behold, manages to curbstomp Link at the precise moment. Plus, correct me if I'm wrong, but he says he has a guilty pleasure/fetish for ruining Link whenever possible, which could also explain things.
