Just z focus on him and hold your shield and when he spit the ball towards you it will hit your shield and directly bounces back and hit him instead. Then go to him and he will kinda appologize and leave. After this you can go through the door.
I always use the Megaton Hammer at the second part. Never had a problem. Are you sure you have hit him at the right spot? Sometimes it's a bit more tricky and you think you hit him at his tail but accidentally could have missed the right spot. I don't say you did but it could be possible, right...