Here's what I think happened: he votes me as a joke and as a way of adding speed to my wagon (subconsciously having the discord joke in mind), then he realizes that my reaction to this kinda stuff is usually seen as AI and could make him look bad, and decides to go back on it. I call him out and now he doesn't have proper explanations from it.
I'm not sure what the relation between you and him is but I'm mildly confident at least one of you is scum.
I don't like you using me reacting to his vote/pressing him for explanations as a way of throwing shade at me, only to also say you can see the other way around and that he can be scum. Seems like setting up for either jumping on a mislynch wagon or even bussing him if you're scummates.