I don't think Ghirahim really took Link all that seriously until the little Temple of Time scene. After that he's usually just too busy for Link or he actually tries to kill him. Please get over your blind hatred for everything Skyward Sword for a total of 10 seconds and perceive the facts.
I totally understand that Ghirahim didn't take Link seriously until ToT, and later Fire Sanctuary. Here:
Do you remember when I told you that the next time we met, I'd make your ears bleed from the sound of your own screams?
That comes from the second fight. But why is it a "next time"? Why didn't he just beat the **** out of Link the first time, huh? It wouldn't be TOO bad of him to do so, especially considering that Link *dares* to meddle in the Demon Lord's plans.