When it comes up do not hold it he will just automatically push you away. Push it then wait for the repeating A button and just hit it for awhile until you push HIM back.
I was always under the impression that the ‘change’ mode was completely random. After reading some of these guys’ posts however there seems to be a strategy to it. I didn’t know that.