I didn't take on Ganon until I had all four Divine beasts beaten and had collected all memories. Still hadn't found all the shrines and stuff, but had gotten fairly far before I attempted it. Have only done that particular fight once so far, though I have started other games since.